Performance of the CMS Silicon Tracker Front-End Driver

LECC2004: Performance of the CMS Silicon Tracker FED: Greg Iles13 September 2004 1

Performance of the CMS Silicon Tracker Performance of the CMS Silicon Tracker Front-End Driver Front-End Driver

10th Workshop on Electronics for LHC 10th Workshop on Electronics for LHC Experiments and Future ExperimentsExperiments and Future Experiments

R. Bainbridge, E. Corrin, C.Foudas, J. Fulcher, G. Hall, G. Iles, J. Leaver, M. Noy, M. Raymond, O. Zorba

Imperial College

D. Ballard, I. Church, J.A.Coughlan, C.P.Day, E.J.Freeman, W.J.F.Gannon,R.N.J. Halsall, M. Pearson, G. Rogers, J. Salisbury, S. Taghavi, I.R.Tomalin

CCLRC Rutherford Appleton Laboratory

I. ReidBrunel University

Presented by Greg Iles: [email protected]


Microstrip Tracker readout chainMicrostrip Tracker readout chain

Off Detector (counting room)Off Detector (counting room)

– Optical links transmit equivalent to 1.3 TB/s @ 100kHz trigger rate

– 440 Front End Drivers (FEDs)

On Detector:On Detector:

– 9M silicon strips

– 73k APV25 readout chips

– Analogue readout via 43k optical readout links

APV

MUX

2:1

APV readout chip

128:1

PLL

DCU

APV readout chip

128:1

Front End ModuleDetector

FEDADC

ADC

x12

FP

GA

ADC

ADC

x12

FP

GA

x8

FP

GA

RA

M

Transition card

S-link card

DAQ

FMMThrottle signals

AOH

9U VME back plane

x96


Front End Driver (FED)Front End Driver (FED)

96 optical fibres inputs, each a multiplexed pair of APVs

8 front end blocks each driven by a 12 way optical ribbon cable

Raw input data rate (all 96 fibres)

= 3.4GB/s.

Output rate down slink

= 50MB/s /% occupancy

VME FPGA

Front-End data processing FPGA

Power

S-Link

Back End “System” FPGA

FE Unit


Front End (FE) Unit on FEDFront End (FE) Unit on FED

Opto-RX, 12 way

12 x Buffers

3 x Delay FPGA(ADC clk timing)

6 x Dual 40MHz, 10bit ADCs

Virtex II, 2M gate FPGA performs signal processing

Optical ribbon cable input

Analogue circuitry duplicated on secondary side

Sig

nal

mag

nit

ud

e

Digital header

128 analogue values (one for each microstrip)

MIP

De-multiplexed fibre channel = APV Data FrameDe-multiplexed fibre channel = APV Data FrameTo extract hit need to perform:

- Common mode subtraction

- Pedestal subtraction - Cluster finding - Sync checking

Opto-to-electrical conversion Digitise & sync data Find hit clusters

Time


FED statusFED status

At LECC ’03 in AmsterdamAt LECC ’03 in Amsterdam– Start ‘03: The first two FEDv1 boards were manufactured– June ‘03: After testing showed there were no major faults a further 3 FEDv1 boards were produced

Progress in the last yearProgress in the last year– Sept. ‘03: Further 6 boards had serious problems– Start ‘04: A further batch of 6 boards manufactured and assembled at

different company– Spring ‘04: FEDs distributed to CERN, Pisa, Lyon– June ‘04: Beam test at CERN– Sept. ‘04: FEDv2 should return from manufacture.– End ‘04: Manufacture a further 20 FEDv2 assuming no surprises.

Software and DAQ for the CMS Silicon Tracker Front End Driver Software and DAQ for the CMS Silicon Tracker Front End Driver Poster by Jon FulcherPoster by Jon Fulcher

The Manufacture of the CMS Tracker Front-End DriverThe Manufacture of the CMS Tracker Front-End Driver

Poster by John CoughlanPoster by John Coughlan


Beam Test June ‘04Beam Test June ‘04

CERN X5 AreaCERN X5 Area– 4 FEDs– 252 fibres– 65,000 strips– 2 FEDs slink

– Provided excellent opportunity for system integration

– Large complex system useful for finding system weak points.

Beam


Testing in the labTesting in the lab

FEDFED– Tested with FED

Tester Ensemble

FED Tester EnsembleFED Tester Ensemble– Drives all 96 FED

channels with data similar to that expected in CMS.

– 100kHz Poisson L1A.

– Also provides clock, L1A and throttling from built in Trigger Control System (TCS)

– An ensemble is made up of 4 FED Testers.

Master FED Tester: Provides clock and control to additional 3 FED Testers

FED under test sandwiched between 2 additional FEDs and crate closed to simulate airflow & temperatures in fully populated crate.

VME access via SBS 620 PCI-VME bridge


The FED TesterThe FED Tester

DACs

System FPGA

VME FPGA

AOHs

x-point switches and buffers on back side

Power

Master & slave I/O

Optical outputs

Fibre spools

Data for 3 channels loaded into FPGA.

Converted to analogue form by 3 DACs.

Cross-point switch controls distribution of the 3 unique channels to the 24 channels.

8 three channel TOB type AOHs convert the electrical signal to optical signals.

Temp of AOHs controlled to +/-1ºC

Provides 24 optical channels & Trigger Control System (TCS)Provides 24 optical channels & Trigger Control System (TCS)


Laboratory test set-upLaboratory test set-up

Slink Rx

Generic PCI Card

VME crate PC

Slink PC (PCI-X slots)

Access VME with SBS620 PCI-VME link

LVDS cableFED

Simulate Local Trigger Control System

FT (master)

FT (slave)

FT (slave)

FT (slave)

Slink-Tx

Clock & L1As from FT to FEDThrottle signals from FED to FT

Merg

e 9

6 fi

bre

s in

to 8

rib

bon

s o

f 12

fib

res

J0 Connector

J1 Connector

VME crate

J2 Connector


Slink transition cardSlink transition card

Carrier for slink transmitterCarrier for slink transmitter– Buffers control and data

signals– Buffers throttle signals – FED v1 and V2 compatible

StatusStatus– Returned from

manufacture in August.

– S-link verified– Throttle signals still

under test

DAQ – Slink Transmitter

FED

Slink data and control signals

Throttle signals to Fast Merge Module (FMM)

VM

E B

ackp

lan

e


Slink verificationSlink verification

CheckCheck– Verify data transmitted by the FED via slink is not corrupted.

Set-UpSet-Up– FED sending test patterns (scope mode, sample size 6)– 100 kHz repetitive trigger

• CPU can only verify data transmitted at ~28 Mbytes/sec• Hence FED asserts “BUSY” and trigger rate falls to ~17.1 kHz

ResultsResults– Verified 1 TB of data in 10.8 hrs

• No errors– No errors observed so far, however it would take 146 days to

guarantee no more than 1 error per week in CMS.

However....However....– In rate test with transition card borrowed from

another sub detector (no signal buffering) we did observe occasional errors.

– More tests needed. Add CRC check.


700

600

500

400

300

200

100

0

Slin

k dat

a ra

te (

MB/s

)

6543210

Event size (kB)

Measured rate

Rate if L1As not throttled

Slink max data throughputSlink max data throughput

FED configurationFED configuration– Scope mode, repetitive

triggers at 100kHz.– Varied scope sample length to

vary event size.

Results & ConclusionsResults & Conclusions– Maximum transfer rate =

469MB/s– Observed S-link receiver

exerting back pressure for events > 4.88kB

– PC rather than FED setting upper limit.

– Switched to random triggers at 100kHz

• OK in scope mode & zero suppressed mode

Requirements for CMSRequirements for CMS– 200MB/s Average– 400MB/s Peak


Temperature MonitoringTemperature Monitoring

Fan direction

Air deflector

Simple air deflector in centre of card lowers temperature in hottest region by ~10°C

OptoRx #7

OptoRx #0

Max temp of OptoRx = 70°C

Measurements from on-board sensors and thermocouples

70

60

50

40

30

20

Tem

pera

ture

ºC

Fan speed = 2460 RPM

, No air deflector

, Air defelector

70

60

50

40

30

20

Tem

pera

ture

ºC

6420OptoRx

Fan speed = 3480 RPM


Where are Tracker buffers ?Where are Tracker buffers ?

Buf 1

Buf 12

Front-End FPGA 1

Buf 1

Buf 12

Chan 1

Chan 12

Head Buf

Back-End FPGA

Mux

BE B

uf

Bus 1

Bus 8

80MB/s

200 MB/s Avg

400 MB/s Peak

640 MB/s Unlimited

APV

40 MS/s @ 10bit

Chan 1

Chan 12

Laser Driver

40 MS/s @ 10bit

APV Buffers, ~10 in decon mode

controlled by APVE

FE Buffers, 2kB, ~125 ZS eventsBE Buffers, 2MB, ~1000 ZS events

Front-End FPGA 8

Header Buffer

Quad Data Rate SRAM. Handles 640MB/s in/out simultaneously.


Buffer testsBuffer tests

Check FED buffers can handle high rates & occupanciesCheck FED buffers can handle high rates & occupancies– Want to test “unconstrained” FED

• Don’t want slink back pressure limiting results• Slink throttle disabled.• Data sent to slink oblivion

– Ideally would like all FED buffers to assert “Busy” or “Warn” when becoming full.

• If buffer overflow inevitable -> Detect the event, but ignore data• Set flag to indicate data loss and record the number of these

events

– Not possible yet, although BE buffer can assert throttle signals• Use different approach. Not perfect, but still informative.• No throttling used• Count number of triggers before FED hangs


Buffer test resultsBuffer test results

Conditions:Conditions:Data taken in Zero Suppression mode

Single strip clusters used to create largest event size possible and thus worst case.

APV frame occupancy kept constant. - e.g. 6 strips = 4.7% occupancy - Easier to understand results. - Large buffers -> Valid approx

Performed 5000 "tests“. Each comprised 100k triggers, 100kHz Poisson distributed

Results:Results:FED handles single strip occupancy up to 6.25% (8 single strip clusters)

100

101

102

103

104

105

106

Mea

n lif

etim

e [n

umber

of

trig

gers

]

1612840

Occupancy [%]

Poisson-distributedtriggers (100 kHz mean)


Deadtime Versus OccupancyDeadtime Versus Occupancy

FED Tester EnsembleFED Tester Ensemble– Sending isolated hits– (i.e single strip

clusters)– 2x106 Poisson L1As @

100kHz

FEDFED– Currently transmits

48MB/s of debug information in zero suppressed mode in addition to 32MB/s of status information.

SlinkSlink– Limited to average

rate of 248 MB/s

CMSCMS– Maximum = 3%

occupancy, but large cluster width > 1

60

50

40

30

20

10

0

Dea

dti

me

(Tri

gger

s V

etoe

d %

)

543210

Occupancy based on single strip clusters (%)

3.53.02.52.01.51.00.5

Measured average event size minus debug inf o (kB)

Slink operating at 248MB/ s which is equivalentto 200MB/ s without FED debug information

Worst case because FED uses data f ormat, which is ineffi cient f or single strip clusters. Need more realistic "f ake" data to simulate CMS high ocupancy regions.


Cluster width extrapolationCluster width extrapolation

Cluster algorithmCluster algorithm– Following is simplification,

however:• 1 byte pos• 1 byte width• 1 byte for every hit strip

Calculate average event sizeCalculate average event size– Make some simplifications

• Ignore data padding for byte aligned data

• Ignore 2 dead channels

Beam test June 2004Beam test June 2004– Average cluster width = 1.29 strips– Average cluster event size = 3.29 bytes

100

80

60

40

20

0

Dea

dti

me

(Tri

gger

s V

etoe

d %

)

43210

Measured average event size minus debug inf o (kB)

Measured deadtime

Average event size based on 3% occupancy

Measured: single strip clusters Theory: single strip clusters Theory: beam test clusters Theory: fi xed cluster width of 2


SummarySummary

FED v1FED v1– Commissioning tests at RAL, Imperial & CERN continuing well.– Baseline firmware and software operational.

FED v2 and S-link Transition cardFED v2 and S-link Transition card– Both back from manufacture in August ’04– Only preliminary results so far, but everything seems OK

FutureFuture– System integration

• Calibration of Tracker, Databases, Large system issues (e.g. error handling)– Production issues

• Tender, Industrial testing– Continued testing

• Beam test in October, Continued testing with the Fed Tester.– Commissioning

• Start to plan


More information...More information...

Software and DAQ for the CMS Silicon Tracker Front End Driver Software and DAQ for the CMS Silicon Tracker Front End Driver Poster by Jon FulcherPoster by Jon Fulcher

The Manufacture of the CMS Tracker Front-End DriverThe Manufacture of the CMS Tracker Front-End Driver

Poster by John CoughlanPoster by John Coughlan

Documents

Performance of the CMS Silicon Tracker Front-End Driver