Start-up & synchronization sequence for Front-End
LHCb Electronics Upgrade Meeting13 February 2014
F. Alessio, CERN
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 2
Scope & Outline
Aim at describing the way in which the system would globally:
- Synchronize the readout of events at the beginning of a run («Start-of-run fast sequence»)
o to ensure TELL40 code is able to align to incoming data stream
o to ensure FE is aligned to correct BXID
- Resynchronization mechanisms• After a desynchronization• Preventive resynch to avoid loss of data• Usage of FE Reset and Synch command
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 3
Synch and alignment of TFC links
TELL40sTELL40s
Front-EndsFront-Ends
SOL40S-ODIN TELL40s
Front-Ends
LHC Clocks
= Receiver
= Transmitter
GBT for TFC+ECS GBT for
DATA
FE ASIC
FE ASIC
FE ASIC
1. Synch and align the TFC links: send Synch pattern in 8b/10b
BXID(12 bits)
Synch Pattern (h bits) ...00000...
630
Firmware will check synchronization of buffer and alignment of content of frame When done, set synch bit via ECS.
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 4
TELL40sTELL40s
Front-EndsFront-Ends
SOL40S-ODIN TELL40s
Front-Ends
LHC Clocks
= Receiver
= Transmitter
GBT for TFC+ECS GBT for
DATA
FE ASIC
FE ASIC
FE ASIC
2. Synch and align TFC-TELL40 links: send Synch pattern in 8b/10b
BXID(12 bits)
Synch Pattern (h bits) ...00000...
630
Synch and alignment of TFC links
Firmware will check synchronization of buffer and alignment of content of frame When done, set synch bit via ECS.
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 5
TELL40sTELL40s
Front-EndsFront-Ends
SOL40S-ODIN TELL40s
Front-Ends
LHC Clocks
= Receiver
= Transmitter
GBT for TFC+ECS GBT for
DATA
FE ASIC
FE ASIC
FE ASIC
3. Synch and align TFC-FE links: send TFC commands in GBT according to each FE’s mapping
SOL40 must be configured before FE and clock transmission must be up (minimal SODIN configuration)
Synch and alignment of TFC links
GBT does the job. When synch’d, can start configuring FE.
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 6
TELL40sTELL40s
Front-EndsFront-Ends
SOL40S-ODIN TELL40s
Front-Ends
LHC Clocks
= Receiver
= Transmitter
GBT for TFC+ECS GBT for
DATA
FE ASIC
FE ASIC
FE ASIC
3bis. Special mode: relay TFC commands back to SOL40 measure latency
Synch and alignment of TFC links
SOL40 will measure transmission delay by comparing BXID Interested only in
BXID latency, not fine phase
However if at the limit might see some desynch and adjust...
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN
BXID(12 bits)
Synch Pattern (h bits) ...00000...
79 or 1110
7
TELL40sTELL40s
Front-EndsFront-Ends
SOL40S-ODIN TELL40s
Front-Ends
LHC Clocks
= Receiver
= Transmitter
GBT for TFC+ECS GBT for
DATA
FE ASIC
FE ASIC
FE ASIC
4. Synch and align FE-TELL40 links: send Synch Pattern in GBT
Synch and alignment of TFC links
TELL40 LLI (GBT decoding+ GX buffer) will synchronize the links Special Synch Pattern is used to align the data stream to TELL40 processing logic
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 8
TELL40sTELL40s
Front-EndsFront-Ends
SOL40S-ODIN TELL40s
Front-Ends
LHC Clocks
= Receiver
= Transmitter
GBT for TFC+ECS GBT for
DATA
FE ASIC
FE ASIC
FE ASIC
4bis. Special mode: relay TFC commands back to TELL40 measure latency
Synch and alignment of TFC links
TELL40 will measure transmission delay by comparing BXID received by SOL40 and FE Interested only in
BXID latency, not fine phase
However if at the limit might see some desynch and adjust...
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 9
ECS start-of-run sequence
Just like now, current LHCb experiment
CONFIGU
RE
• SOL40s get configured first, then FE gets configured, via SOL40.• Everything goes “READY” (if ok…) (very quickly…)• Links are synch’d across the whole system
READY
• FE is free running in this state (idle state at FE), but ready to react on TFC commands. On output link send random frames to keep GBT link up
• ECS issues “RUN”. Some configuration might go via SOL40 to FE (if)
RUNNIN
G
• Everything is ready to take data, SODIN is still in ACTIVE. ECS issues “GO”• Fast TFC start-of-run sequence starts. Then, data taking!
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 10
TFC start-of-run fast sequence
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 11
TFC start-of-run sequence
FE Reset is the first thing issued via control system: asynchronous!
Issued when TFC receives command «GO»
Ensure that FE got the right BXID value before synch-ing to TELL40
This sequence is the same every time a FE Reset is issued. If FE Reset is ussed, change Run #. For «in-Run» resynch, use Synch
command.
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 12
What to do on SYNCH?
Synch command is meant to be sure that (whole) system is synchronized… in a synchronous way!
FEs send Synch Patter for the same BXIDs everywhere• FE frees its memory : delete its content, read and write pointers back to empty• FE sends Synch Pattern for as many as Synch commands are set by TFC• Next event, starts at bit 0 (LSB) – might be header only• TELL40 will align to corresponding frame and BXID
TELL40 closes all events before and sends them out truncated TELL40 does not know what BXID to expect. Synch pattern says it.
Note: Synch command can be used on lab tests without TFC. For few links, system can align independently of TFC (just program your FE to send it and TELL40 to receive it). TELL40 would be in pass-through (i.e. accept all).
As soon as you need synchronicity across all links, then you need TFC. TFC will ensure that synch command is sent out for same BXID everywhere automatically aligning TELL40 readout.
Send something like this! BXID
(12 bits)Synch Pattern (h bits) ...00000...
79 or 1110
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 13
What to do on SYNCH?
Double usage (in AND or in OR):
1. Periodically: i.e., SYNCH command sent every n Hz a la FE Reset, but without resetting a bit inefficient to clear the FE buffer (could be like this only at the
beginning and when very few bunches – lots of empty-empty crossing)
2. Asynchronously: i.e. when a desynch is detected, like TELL40 detects wrong frames, wrong packing
needs fast diagnostics in TELL40 codes makes sense to clear the FE buffer in this case
could be sent only for a local sub-detector from SOL40 • (slow) triggered by ECS (TELL40 or FE set a desynch bit, ECS sends
command to SOL40) • or by TELL40 via SOL40 transmitting an info field regarding this w/
throttle protocol (we have plenty of BW)
Resynchronization sequence
Send something like this! BXID
(12 bits)Synch Pattern (h bits) ...00000...
79 or 1110
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 14
Resynchronzation sequence
Synch command was requested From ECS or fast via TELL40
Note: Synch command is entirely programmable in frequency, length and location around the orbit. Header Only is entirely programmable in length.
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 15
Conclusion
TFC documents updated, please review and comment.
LHCb-PUB-2012-001 (TFC specs)LHCb-PUB-2012-017 (FE and BE specs)
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 16
Backup
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 17
System and functional requirements
1. Bidirectional communication network
2. Clock jitter, and phase and latency control
At the FE, but also at TELL40 and between S-TFC boards
3. Partitioning to allow running with any ensemble and parallel partitions
4. LHC interfaces
5. Events rate control
6. Low-Level-Trigger input
7. Support for old TTC-based distribution system
8. Destination control for the event packets
9. Sub-detectors calibration triggers
10.S-ODIN data bank
Infomation about transmitted events
11.Test-bench support
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 18
The S-TFC system at a glance
DATA
DATA
S-ODIN responsible for controlling upgraded readout system • Distributing timing and synchronous commands• Manages the dispatching of events to the EFF• Rate regulates the system• Support old TTC system: hybrid system!
SOL40 responsible for interfacing FE+TELL40 slice to S-ODIN• Fan-out TFC information to TELL40• Fan-in THROTTLE information from
TELL40• Distributes TFC information to FE • Distributes ECS configuration data to FE• Receives ECS monitoring data from FE
STORAGEReadout
Supervisor S-ODIN
Interface boardsSOL40
Readout BoardsTELL40s
Front-Ends
LHC Interfaces
TFC
TFC + ECS
TFC
FARMECS (FE)
LLT
THROTTLEECS
40 Gb/s
3.2 Gb/s
4.8 Gb/s
3.2 Gb/s
1 Gb/s
2.4 Gb/s 40 MHz clock
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN
S-TFC concept reminder
19
LHC Interface
Clock Fanout
CLK
ECS
PH
Y
Programmable Switch layer (Partitioning)
Built-in GX Transceivers layer
PH
Y FARM
n
n
#links = #LHCb sub-systems
S-ODIN
TFC+ECS Interface
TF
C,
Th
rott
le
FAN-OUT/FAN-IN Logic
+ECS to FE
encoding Logic+
optional S-ODIN Logic
Switch Logic
Master FPGA (STRATIX IV/V GX)
S-ODIN Logic Instantiations (x6)
Master FPGA (STRATIX IV/V GX)
LLT
FE electronics
TELL40s
GB
T
FPGA-to-FPGA transceivers
S-ODIN Logic
PH
Y
ECS to FE
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN
Readout Crate
TE
LL
40s
FEs FEs FEs...FEs FEs FEs...FEs FEs FEs...
SO
L40
TFC on backplane
ECS
TFC Crate
S-O
DIN
, LL
T, L
HC
LHC clock, LHC interfaces
TR
IG40
s
FARM
DATATFC+ECS
TE
LL
40s
TE
LL
40s
TE
LL
40s
TE
LL
40s
TE
LL
40s
TE
LL
40s
TE
LL
40s
The upgraded physical readout slice - ATCA
Common electronics board for upgraded readout system: AMC40 cards fitted in an ATCA motherboard
• S-ODIN & SOL40 AMC cards• LLT & TRIG40 AMC cards• TELL40s AMC card• LHC Interfaces specific AMC card
20
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN
The upgraded physical readout slice – PCIe
Common electronics board for upgraded readout system: PCIe-Gen3 card fitted on a host PC • SODIN & SOL40 PCIe card• LLT PCIe ard• TELL40 PCIe card• LHC Interfaces specific PCIe card
21
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN
TFC protocol to TELL40
22
«Extended» TFC word to TELL40 via SOL40: 64 bits sent every 40 MHz = 2.56 Gb/s packed with 8b/10b protocol (i.e. total of 80 bits) no dedicated GBT buffer, use ALTERA GX simple 8b/10b
encoder/decoder
THROTTLE information from each TELL40 to SOL40: • 1 bit for each AMC board + BXID for which the throttle was set
merged and aligned in SOL40 same GX buffer as before (same bidirectional transceiver)
Constant latency after BXID
BXID(11..0) MEP Dest(31..0) Trigger Type(3..0) Calibration Type(3..0)
Trigger BX Veto NZS Mode
Header Only
BE Reset
BXID Reset
FE Reset
EID Reset
Synch Snapshot
Reserve
0123456789
13 .. 1017 .. 1449 .. 18505163 .. 52MEP
Accept
MEP accept command when MEP ready: Take MEP address and pack to FARM No need for special address, dynamic
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN
TFC protocol to FE
23
TFC word on downlink to FE via SOL40 embedded in GBT word: 24 bits in each GBT frame every 40 MHz = 0.98 Gb/s all commands associated to BXID in TFC word
Put local configurable delays for each TFC command • GBT does not support individual delays for each line• Need for «local» pipelining: detector delays+cables+operational logic (i.e. laser
pulse?) DATA SHOULD BE TAGGED WITH THE CROSSING TO WHICH IT BELONGS!
TFC word will arrive before the actual event takes place• To allow use of commands/resets for particular BXID• Accounting of delays in SODIN: for now, 16 clock cycles earlier + time to receive• Aligned to the furthest FE (simulation, then in situ calibration!)
TFC protocol to FE has implications on GBT configuration and ECS to/from FE• see specs document!
23 .. 12 11 10 9 8 .. 5 4 3 2 1 0
BXID(11..0) Synch NZS Mode
Header Only
BXID Reset
FE ResetCalibration Type(3..0) BX VetoSnapshotReserve
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 24
SODIN firmware v1r0 – block diagram
OR
BIT
BXID cnt BXIDBCLK
Internal Triggers
RndTrg Gen
PerTrg Gen
PerTrg Gen
CalibTrg Gen
CalibTrg Gen
CalibTrg Gen
CalibTrg Gen
BXID
RndTrgA
LumiTrgs
RndTrgC
RndTrgD
PerTrg01
PerTrg02
CalibTrgA
CalibTrgB
CalibTrgC
CalibTrgD
LLT INPUT LLT Trg
LLT Type
AUX INPUTAUX Trg
Trg Manager
FE reset handler
BE reset handler
FE RESET (from ECS)
BE RESET (from ECS)
Header Only TRG
BX Veto TRG
Trigger TRG
Trigger Type TRG
LLT Trg Enb
LLT Beamgas Enb
AUX Trg Enb
Calib Type TRG
Trigger BXID TRG
NZS/TAE handler
LLT BXID
BXID
NZ
S e
nb
NZ
S
con
secu
tive
NZ
S w
ait
EID RESET (from ECS)
EID reset handler
TA
E e
nb
FE Reset TRG (1 clock cycle)
BE Reset TRG (1 clock cycle)
Header Only TAE
BX Veto TAE
Calib Type TAE
Trigger BXID TAE
NZS mode TAE
Synch(from ECS)
Synch cmd
handler
Syn
ch Cm
d
MEP handler
FARM
ME
P w
ind
ow
MEP destination
TFC WORD (64bits in 8b/10b
encoder) ~2.5Gb/s
ME
P
des
tinat
ion
enb
Throttle handler
Throttle
Throttle BXID
BXID
S-ODIN Master Logic
SODIN data bank creator SODIN data bank to FARM
(12 x 32b Ethernet)~16Gb/s
ME
P d
estinatio
n
Trg BX Type TRG Trg BX Type TAE
UTC start time(from ECS)
UTC time++
OD
IN d
ata
ban
k e
nb
Throttle enb
BX Type Handler BX Type
WriteRead
Fill
ing
S
che
me
(f
rom
EC
S)
Filling Scheme RAM (224 x 32b)
EID cntEID accept
Run number (from ECS)
EID Reset TRG (1 clock cycle)
Synch Cmd TRG (2 or more clock cycles)
TCK (from ECS)
Snapshot(from ECS)
Snapshot cmd
handler
Synch length(from ECS)
Snapshot interval
(from ECS)
BX
ID R
ese
t
FE Reset TAE
BE Reset TAE
EID Reset TAE
Synch Cmd TAE
TA
E la
tenc
y co
mp
ens
atio
n
Trigger Type TAE
Trigger TAE
FE
Re
set wa
term
ark
FE
Re
set
BE
Re
set w
ate
rma
rk
BE
Re
set
EID
Re
set
Th
rottle
NZ
S+
TA
E
wa
term
ark
TA
E w
ind
ow
EID
Re
set
Sn
apsh
ot C
md
BXID reset handlerBXID
PA
US
E (fro
m E
CS
)
ME
P d
yna
mic
d
estin
atio
n en
b
MEP throttle
EID
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN
Timing distribution
25
From TFC point of view, ensured constant: LATENCY: Alignment with BXID FINE PHASE: Alignment with best sampling point
TELL40sTELL40s
Front-EndsFront-Ends
TFC+ECSInterfaceS-ODIN TELL40s
Front-Ends
LHC Clocks
= Receiver
= Transmitter
GBT for TFC+ECS GBT for
DATA
FE ASIC
FE ASIC
FE ASIC
Some resynchronization mechanisms envisaged: Within TFC boards With GBT
No impact on FE itself
Loopback mechanism: re-transmit TFC word
back allows for latency
measurement + monitoring of TFC commands and synchronization
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN
DCS objectDCS
object
TELL40TELL40TELL40
GBTX SCA SCA
DCS object
GBTX GBTX GBTX GBTX GBTX GBTX
FE ASIC
FE ASIC
FE ASIC
FE ASIC
FE ASIC
24
DATA
TFC commands
SOL40
SCA
TFC+ECS GBT
DATA GBT
SCA SCA SCA Configuration data / monitoring data
I2C, JTAG…
Clock
TFC+ECS
Generic FE electronics architecture26
How to decode TFC in FE chips?
Usage of TFC+ECS GBTs in FE is 100% common to everybody!! dashed lines indicate the detector
specific interface parts please pay particular care in the
clock transmission: the TFC clock must be used by FE to transmit data, i.e. low jitter!
Kapton cable, crate, copper between FE ASICs and GBTX
FE electronics block
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN
FEModule
FEModule
Phase – Aligners + Ser/Des for E – Ports
FEModule
E – PortE – Port
E – Port
GBT – SCA
E – Port
Phase - Shifter
E – PortE – Port
E – PortE – Port
CDR
DEC/D
SCR
SER
SCR/ENC
I2C MasterI2C Slave
Control Logic Configuration(e-Fuses + reg-Bank)
Clock[7:0]
CLK Manager
CLK Reference/xPLL
External clock reference
clockscontroldata
one 80 Mb/s port
I2C port
I2C (light)
JTAG
80, 160 and 320 Mb/s ports
GBTIA
GBLD
GBTXe-Link
clock
data-up
data-down
ePLLTxePLLR
x
JTAG port
27
The TFC+ECS Master GBT
These clocks should be the main clocks for the FE• 8 programmable phases • 4 programmable
frequencies (40,80,160,320 MHz)
Used to:• sample TFC bits • drive Data GBTs • drive FE
processes
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 28
The TFC+ECS GBT protocol to FE
D0D1D2D3D4ECICHEADER FEC
GBT word: 120 bits
2x16 bits16bits2x2bits4bits
24 e-links @ 80 Mb/sfor TFC
Idle: 0110Data: 0101
1 e-link @ 80 Mb/s for
GBT internal use only 1 e-link @
80 Mb/s to GBT-SCA
SCA
16 e-link @ 80 Mb/s to 16 GBT-SCAs
for ECS
16bits16bits16bits16bits
SCAs SCAs
TFC protocol has direct implications in the way in which GBT should be used everywhere• 24 e-links @ 80 Mb/s dedicated to TFC word as a baseline (effectively 48 bits)
use 80 MHz phase shifter clock to sample TFC parallel word• TFC bits are packed in GBT frame so that they all come out on the same clock edge
Modification are possible in order to satisfy sub-detector’s FE requirements
Leftover e-links dedicated to GBT-SCAs for ECS configuring and monitoring (see later)
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 29
Words come out from GBT at 80 Mb/s
D0[14..2,0]D1D2D3D4EC[0]IC[0]H[2,0] FEC[30..6,4,2,0]
D0[15..3,1]D1D2D3D4EC[1]IC[1]H[3,1] FEC[31..7,5,3,1]
24bits x TFC à grouped e-link + 1
clock line
Other purposes?
17bits x ECSà 17 GBT-SCA
17bits x ECS msb first, odd bits
lsb second, even bits
In simple words:• Odd bits of GBT protocol on rising edge of 40 MHz clock (first, msb), • Even bits of GBT protocol on falling edge of 40 MHz clock (second,
lsb)
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 30
TFC decoding at FE after GBT
D0[1] D0[0] D0[1] D0[0]
D0[3] D0[2] D0[3] D0[2]
...... ......
D1[1] D1[0] D1[1] D1[0]
D0[7] D0[6] D0[7] D0[6]
...... ......
D2[1] D2[0] D2[1] D2[0]
D1[7] D1[6] D1[7] D1[6]
...... ......
D2[7] D2[6] D2[7] D2[6]
TFC[23..0] TFC[23..0]unused unused
24 2424 24
80 MHz
40 MHz
E-link 1, group 1
E-link 2, group 1
E-link 8, group 1
E-link 1, group 2
E-link 8, group 2
E-link 1, group 3
E-link 8, group 3
This is crucial!!
we can already specify where each TFC bit will come out on the GBT chip
this is the only way in which FE designers still have minimal freedom with GBT chip
if TFC info was packed to come out on only 12 e-links (first odd then even), then decoding in FE ASIC would be mandatory!
which would mean that the GBT bus would have to go to each FE ASIC for decoding of TFC command
there is also the idea to repeat the TFC bits on even and odd bits in TFC protocol
would that help? FE could tie logical blocks directly on GBT pins… Or to select a minimal set of TFC commands and
repeat them to profit from fan-out possibilities
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 31
Now, what about the ECS part?
Each pair of bit from ECS field inside GBT can go to a GBT-SCA • One GBT-SCA is needed to configure the Data GBTs (EC one for example?)• The rest can go to either FE ASICs or DCS objects (temperature, pressure) via other
GBT-SCAs GBT-SCA chip has already everything for us: interfaces, e-links ports ..
No reason to go for something different! However, «silicon for SCA will come later than silicon for GBTX»…
We need something while we wait for it! FPGA emulator (working on it)
D0D1D2D3D4ECICHEADER FEC
GBT word: 120 bits
2x16 bits16bits2x2bits4bits
24 e-links @ 80 Mb/sfor TFC
Idle: 0110Data: 0101
1 e-link @ 80 Mb/s for
GBT internal use only 1 e-link @
80 Mb/s to GBT-SCA
SCA
16 e-link @ 80 Mb/s to 16 GBT-SCAs
for ECS
16bits16bits16bits16bits
SCAs SCAs
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 32
Protocol drivers build GBT-SCA packets with addressing scheme and bus type for associated GBT-SCA user busses to selected FE chip Basically each block will build one of
the GBT-SCA supported protocols
Memory Map with internal addressing scheme for GBT-SCA chips + FE chips addressing, e-link addressing and bus type: content of memory loaded from ECS
SOL40 firmware
E-LinkProtcolDrivers
E-LinkProtcolDrivers
PCIe Memory
Map
TFC Relay &
Alignment
TFC decoder
Synchronous TFC Info to
TELL40s (64 bits)
S-ODIN synchronousTFC Info fan-out
ECS
FE
E-LinkProtcolDrivers
CCPCPCIe Slave
24 bits
34 bits
GBT Tx
EC
ST
FC
LHCb Electronics Upgrade Meeting, 26/07/12 F. Alessio, R. Jacobsson
Usual considerations …
33
TFC+ECSInterface has the ECS load of an entire FE cluster for configurating and monitoring 34bits @ 40 MHz = 1.36Gb/s on single GBT link
• ~180 Gb/s for full SOL40 (132 links)• Single CCPC might become bottleneck… Clara & us, December 2011
How long to configure FE cluster? how many bits / FE? how many FEs/ GBT link? how many FEs / TFC+ECSInterface?
Numbers to be pinned down soon + GBT-SCA interfaces and protocols.
Readout CrateT
ELL
40
TE
LL40
TE
LL40
TE
LL40
TE
LL40
TE
LL40
TE
LL40
TE
LL40
FEs FEs FEsFEs...FEs FEs FEsFEs...FEs FEs FEsFEs...
SO
L40
ECS
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN
Old TTC system support andrunning two systems in parallel
34
We already suggested the idea of a hybrid system:reminder: L0 electronics relying on TTC protocol part of the system runs with old TTC system part of the system runs with the new architecture
How?
1. Need connection between S-ODIN and ODIN (bidirectional) use dedicated RTM board on S-ODIN ATCA card
2. In an early commissioning phase ODIN is the master, S-ODIN is the slave S-ODIN task would be to distribute new commands to new FE, to new
TELL40s, and run processes in parallel to ODIN ODIN tasks are the ones today + S-ODIN controls the upgraded part
In this configuration, upgraded slice will run at 40 MHz, but positive triggers will come only at maximum 1.1MHz…
• Great testbench for development + tests + apprenticeship…• Bi-product: improve LHCb physics programme in 2015-2018…
3. In the final system, S-ODIN is the master, ODIN is the slave ODIN task is only to interface the L0 electronics path to S-ODIN
and toprovide clock resets on old TTC protocol
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN
SODIN on Marseille’s ATCA board
35
ATCA motherboard
FPGA
RJ45
CCPC GbE
Serial crossbar
PCIe
Clock
Int. Comm.
S-ODIN
AMC
AMC
LLT Trigger
AMC
LHC Interfaces
AMC
FPGA
FPGA
FPGA
FPGA
Bac
kpla
ne c
onne
ctor
Clock
TFC+ Throttles
LHC Interfaces
LLT sub-trigger
FARM
SOL40
ATCA RTM
RT
M
conn
ecto
r
Ext
erna
l co
nnec
tors
à
OD
IN
Clock crossbar
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN
TFC+ECSInterface on Marseille’s ATCA board
36
SOL40
RJ45
CCPC GbE
FPGA
Serial crossbar
PCIe
Clock
TFC + Throttles
FE
AMC
FE
AMC
FE
AMC
S-ODIN + FE
AMC
FPGA
FPGA
FPGA
MasterFPGA
Bac
kpla
ne c
onne
ctor
Clock
TFC+ Throttles
S-ODIN
FE (GBT, opt.)
FE (GBT)
FE (GBT)
FE (GBT)
Clock crossbar
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 37
Reminder: your (generic) FE
For details, see LHCb-INT-2011-011
Compress (zero-suppress) data already at the FE• reduce # of links from ~80000 to ~12500 (~20 MCHF to ~3.1 MCHF)• data driven readout (asynchronous) + variable latencies!
Efficiently use data link bandwidth• pack data on data link continuously with elastic buffer• extensive use of GBT (robust FEC vs WideBus mode)
evaluate choices based on complexity vs robustness
NO TRIGGER to FE! Only commands, clock
and slow control
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 38
Fast & Slow Control to FE
Separate links between controls and data
• A lot of data to collect
• Controls can be fanned-out (especially fast control)
Compact links merging Timing, Fast and Clock (TFC) and Slow Control (ECS).
• Extensive use of GBT as Master GBT to drive Data GBT (especially for clock)
• Extensive use of GBT-SCA for FE configuration and monitoring
On detector
Off detector
4.8 Gb/s
4.8 Gb/s
TFC
ECS
Data
TFC
ECS
Data
4.8 Gb/s
Off detector
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 39
Reminder: generic FE data flow scheme
Compression/suppression logic can
have dynamic or static latency
Applies changes to data
FE buffer for data
Tag data with TFC commands and pipe them across
compresson/suppression logic block
Modify data according to TFC commands + BufferFull then pack continuously onto
GBT
Data available needed only if compression / suppression is
dynamic
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 40
The code: FE data generator
DERANDOMIZING BUFFERProgrammable Depth
Width: (number of channels * channel size) + header size
CHANNEL DATA GENERATOR
DATA OCCUPANCY GENERATOR
(POISSON)
LHC MACHINE (FILLING SCHEME)
DATA WORD(number of channels * channel size)
EVENT HEADER( 12 bits BCLK identifier
+ 4 status bits + 8 bits for data size )
ELSE
IF DATA_SIZE = 0 (no hit)or BX_VETO = 1
or HEADER_ONLY = 1
DO NOT WRITE TO DERANDOMIZER WHILE PACKING JUST HEADERS!à RECUPERATE BUFFER SPACE
à FORCE DATA SIZE TO 0 IN HEADER FOR TELL40 DECODINGà MAX # OF HEADERS IN WORD =
ROUND_TO_INT(GBT WORD SIZE / HEADER SIZE)
GBT PACKING LOGIC
GBT FRAME – EXAMPLE WITH GBT SIZE 84 BITS WITH FEC CORRECTION
IF NZS_MODE = 1 occupancy 100%
à put all channels in word
WRITE TO DERANDOMIZER
Padded to 0s... EVENT HEADER
EVENT HEADER
EVENT HEADER
...
0x5 NB: GBT HEADER = X”5” if data or X”6" if IDLEEV 01 HEADERDATA WORD of EV 01
0x5EV 02 HEADER REST OF DATA WORD of EV 01 DW of EV 02
0x5REST OF DATA WORD of EV 02
0x5REST OF EV 04
HEADER
EX: EV02 is NZS
0x5REST OF DATA WORD of EV 02
0x5REST OF DATA WORD of EV 02
FOLLOWING 3 EVENTS HAVE HEADER ONLY = 1 FROM TFC
EV 05 HEADEREV 06 HEADERDW OF EV 06
0x5REST OF DATA WORD of EV 06 NEXT EVENT IS IN LINE
0x5EV 07 HEADER REST OF DATA WORD of EV 06 DW of EV 07
0x5EV 08 HEADER REST OF DATA WORD of EV 07 DW of EV 08
0x5REST OF DATA WORD of EV 08
0x5
EV 09 HEADER
DATA WORD of EV 09
.
.
.GBT ENCODER
IF FE_RESET = 1 or SYNCH = 1
à reset derandomizer buffer
0x5EV XX HEADER00000MANUAL ALIGNMENT FRAME
(programmable via ECS)
.
.
.
This part should be replaced by sub-detector’s specific data generated from MonteCarlo simulation
NB: data word width can include other informationà e.g. width would be (number_of_channels *
channel_size) + hit_pattern_size (== number_of_channels)
0x600000DERANDOMIZER IS EMPTY, SEND IDLE
FRAME OVER GBT FRAME
0x5... TWO EMPTY EVENTS
DERANDOMIZER ALMOST FULL
à leaving some buffer space for safety margin
SYNCH will always follow a FE RESET in order to achieve link synchronizationà during SYNCH command asserted, FE should exchange data field with MANUAL
ALIGNMENT FRAME for links synchronization
0x5EV XX+1 HEADER
00000MANUAL ALIGNMENT FRAME
(programmable via ECS)
READ FROM DERANDOMIZER
FIXED OR DYNAMIC LATENCY,
Not implemented and sub-detector specific
SUPPRESSION / COMPRESSION
MECHANISM
DO NOT WRITE TO DERANDOMIZER WHILE PACKING JUST HEADERS!à RECUPERATE BUFFER SPACE
à SET TRUNCATED BIT = 1à MAX # OF HEADERS IN WORD =
ROUND_TO_INT(GBT WORD SIZE / HEADER SIZE)
Padded to 0s... EVENT HEADER
EVENT HEADER
EVENT HEADER
...
EV 03 HEADEREV 04
HEADER
GBT ENCODER
ELSE
EV 10 HEADEREV 11
HEADER
REST of EV 11 HEADER
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 41
The code: FE buffer manager
DERANDOMIZING BUFFERProgrammable Depth
Width: (number of channels * channel size) + header size
CHANNEL DATA GENERATOR
DATA OCCUPANCY GENERATOR
(POISSON)
LHC MACHINE (FILLING SCHEME)
DATA WORD(number of channels * channel size)
EVENT HEADER( 12 bits BCLK identifier
+ 4 status bits + 8 bits for data size )
ELSE
IF DATA_SIZE = 0 (no hit)or BX_VETO = 1
or HEADER_ONLY = 1
DO NOT WRITE TO DERANDOMIZER WHILE PACKING JUST HEADERS!à RECUPERATE BUFFER SPACE
à FORCE DATA SIZE TO 0 IN HEADER FOR TELL40 DECODINGà MAX # OF HEADERS IN WORD =
ROUND_TO_INT(GBT WORD SIZE / HEADER SIZE)
GBT PACKING LOGIC
GBT FRAME – EXAMPLE WITH GBT SIZE 84 BITS WITH FEC CORRECTION
IF NZS_MODE = 1 occupancy 100%
à put all channels in word
WRITE TO DERANDOMIZER
Padded to 0s... EVENT HEADER
EVENT HEADER
EVENT HEADER
...
0x5 NB: GBT HEADER = X”5” if data or X”6" if IDLEEV 01 HEADERDATA WORD of EV 01
0x5EV 02 HEADER REST OF DATA WORD of EV 01 DW of EV 02
0x5REST OF DATA WORD of EV 02
0x5REST OF EV 04
HEADER
EX: EV02 is NZS
0x5REST OF DATA WORD of EV 02
0x5REST OF DATA WORD of EV 02
FOLLOWING 3 EVENTS HAVE HEADER ONLY = 1 FROM TFC
EV 05 HEADEREV 06 HEADERDW OF EV 06
0x5REST OF DATA WORD of EV 06 NEXT EVENT IS IN LINE
0x5EV 07 HEADER REST OF DATA WORD of EV 06 DW of EV 07
0x5EV 08 HEADER REST OF DATA WORD of EV 07 DW of EV 08
0x5REST OF DATA WORD of EV 08
0x5
EV 09 HEADER
DATA WORD of EV 09
.
.
.GBT ENCODER
IF FE_RESET = 1 or SYNCH = 1
à reset derandomizer buffer
0x5EV XX HEADER00000MANUAL ALIGNMENT FRAME
(programmable via ECS)
.
.
.
This part should be replaced by sub-detector’s specific data generated from MonteCarlo simulation
NB: data word width can include other informationà e.g. width would be (number_of_channels *
channel_size) + hit_pattern_size (== number_of_channels)
0x600000 DERANDOMIZER IS EMPTY, SEND IDLE FRAME OVER GBT FRAME
0x5... TWO EMPTY EVENTS
DERANDOMIZER ALMOST FULL
à leaving some buffer space for safety margin
SYNCH will always follow a FE RESET in order to achieve link synchronizationà during SYNCH command asserted, FE should exchange data field with MANUAL
ALIGNMENT FRAME for links synchronization
0x5EV XX+1 HEADER
00000MANUAL ALIGNMENT FRAME
(programmable via ECS)
READ FROM DERANDOMIZER
FIXED OR DYNAMIC LATENCY,
Not implemented and sub-detector specific
SUPPRESSION / COMPRESSION
MECHANISM
DO NOT WRITE TO DERANDOMIZER WHILE PACKING JUST HEADERS!à RECUPERATE BUFFER SPACE
à SET TRUNCATED BIT = 1à MAX # OF HEADERS IN WORD =
ROUND_TO_INT(GBT WORD SIZE / HEADER SIZE)
Padded to 0s... EVENT HEADER
EVENT HEADER
EVENT HEADER
...
EV 03 HEADEREV 04
HEADER
GBT ENCODER
ELSE
EV 10 HEADEREV 11
HEADER
REST of EV 11 HEADER
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 42
The code: GBT dynamic packing DERANDOMIZING BUFFERProgrammable Depth
Width: (number of channels * channel size) + header size
CHANNEL DATA GENERATOR
DATA OCCUPANCY GENERATOR (POISSON)
LHC MACHINE (FILLING SCHEME)
DATA WORD(number of channels * channel size)
EVENT HEADER( 12 bits BCLK identifier
+ 4 status bits + 8 bits for data size )
ELSE
IF DATA_SIZE = 0 (no hit)or BX_VETO = 1
or HEADER_ONLY = 1
DO NOT WRITE TO DERANDOMIZER WHILE PACKING JUST HEADERS!à RECUPERATE BUFFER SPACE
à FORCE DATA SIZE TO 0 IN HEADER FOR TELL40 DECODINGà MAX # OF HEADERS IN WORD =
ROUND_TO_INT(GBT WORD SIZE / HEADER SIZE)
GBT PACKING LOGIC
GBT FRAME – EXAMPLE WITH GBT SIZE 84 BITS WITH FEC CORRECTION
IF NZS_MODE = 1 occupancy 100%
à put all channels in word
WRITE TO DERANDOMIZER
Padded to 0s... EVENT HEADER
EVENT HEADER
EVENT HEADER
...
0x5 NB: GBT HEADER = X”5” if data or X”6" if IDLEEV 01 HEADERDATA WORD of EV 01
0x5EV 02 HEADER REST OF DATA WORD of EV 01 DW of EV 02
0x5REST OF DATA WORD of EV 02
0x5REST OF EV 04
HEADER
EX: EV02 is NZS
0x5REST OF DATA WORD of EV 02
0x5REST OF DATA WORD of EV 02
FOLLOWING 3 EVENTS HAVE HEADER ONLY = 1 FROM TFC
EV 05 HEADEREV 06 HEADERDW OF EV 06
0x5REST OF DATA WORD of EV 06 NEXT EVENT IS IN LINE
0x5EV 07 HEADER REST OF DATA WORD of EV 06 DW of EV 07
0x5EV 08 HEADER REST OF DATA WORD of EV 07 DW of EV 08
0x5REST OF DATA WORD of EV 08
0x5
EV 09 HEADER
DATA WORD of EV 09
.
.
.GBT ENCODER
IF FE_RESET = 1 or SYNCH = 1
à reset derandomizer buffer
0x5EV XX HEADER00000MANUAL ALIGNMENT FRAME
(programmable via ECS)
.
.
.
This part should be replaced by sub-detector’s specific data generated from MonteCarlo simulation
NB: data word width can include other informationà e.g. width would be (number_of_channels *
channel_size) + hit_pattern_size (== number_of_channels)
0x600000 DERANDOMIZER IS EMPTY, SEND IDLE FRAME OVER GBT FRAME
0x5... TWO EMPTY EVENTS
DERANDOMIZER ALMOST FULL
à leaving some buffer space for safety margin
SYNCH will always follow a FE RESET in order to achieve link synchronizationà during SYNCH command asserted, FE should exchange data field with MANUAL
ALIGNMENT FRAME for links synchronization
0x5EV XX+1 HEADER
00000MANUAL ALIGNMENT FRAME
(programmable via ECS)
READ FROM DERANDOMIZER
FIXED OR DYNAMIC LATENCY,
Not implemented and sub-detector specific
SUPPRESSION / COMPRESSION
MECHANISM
DO NOT WRITE TO DERANDOMIZER WHILE PACKING JUST HEADERS!à RECUPERATE BUFFER SPACE
à SET TRUNCATED BIT = 1à MAX # OF HEADERS IN WORD =
ROUND_TO_INT(GBT WORD SIZE / HEADER SIZE)
Padded to 0s... EVENT HEADER
EVENT HEADER
EVENT HEADER
...
EV 03 HEADEREV 04
HEADER
GBT ENCODER
ELSE
EV 10 HEADEREV 11
HEADER
REST of EV 11 HEADER
Very important to analyze simulation
output bit-by-bit and clock-by-clock!
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 43
Reminder: dynamic packing algorithm
01234
Average event size =
link bandwidth
Buff
er d
epth
Average event size
01234
Link bandwidth
01234
BX0BX1BX2BX3BX4
BX0BX1
BX2 BX3 BX4
Header is the unique identifier for each event in frame Compulsory (tag for each crossing), partly programmable (must contain length of
frame+BXID) Difficult buffer management, but almost no truncation. Flexible against occupancy problem (what if your estimate is wrong?). Maximum exploitation of bandwidth. Readout Board uses Header to decode and separate frames lots of resources.
+ =
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 44
Reminder: fixed packing algorithm
01234
Average event size /=link bandwidth Truncation!
Buff
er d
epth
Average event size
01234
Link bandwidth
01234
BX0BX1BX2BX3BX4
BX0BX1BX2BX3BX4
This is different: one clock cycle one event one GBT frame Header more flexible: you can add addresses, hitmaps… Very simple buffer management, but truncation has to happen eventually. Not flexible against occupancy problem (again, what if your estimate is wrong?). Loses a bit of bandwidth as empty spaces must be padded to be sent out. Readout Board uses Header to decode and separate frames much fewer resources
+ =
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 45
Reminder: fixed vs variable length header
in dynamic packingDynamic packing with fixed length header.
Dynamic packing with dynamic length header (fully flexible!)
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 46
Few comments to start with:- BX Veto and Header Only commands are identical from FE point of view
ORed- TFC commands are synchronous wrt to BXID Reset
once we align BXID Reset with beam, TFC commands come ALWAYS at the same latency
• (wrt to BXID Reset, hence BXID)! Compression/suppression logic should act accordingly to TFC command
• (why would you want to compress/suppress if that crossing is rejected a priori? Especially if your pre-processing is dynamic…)
- Data is filtered according to TFC commands and the FE buffer status- Data is packed onto the GBT link in a continuous fashion
FE flow control scheme
BX Veto with a calib trigger in between
Clock
FE Reset
Header Only
Other cmds
BXID Reset
BX Veto
Calib type[0]
Length of BX veto depends exclusively on filling scheme
Fixed distance after alignment
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 47
Data Valid
GBT can accept DATA or IDLE frame: Send IDLE frame whenever a GBT frame
is not ready to be sent! IDLE frame can contain whatever your
sub-detector wants to send.
See TELL40 fw specs, coming soon…
Data Valid signal to distinguish between DATA and IDLE frame:
Clock 80 MHz
Data Valid
Data
Data Valid Usage
Clock 40 MHz
IDLE DATA DATA DATA DATA DATAIDLE
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 48
Data Valid
Be careful to rise synchronize the Data Valid signal to the right rising edge when using the 80 MHz clock (or 160 or 320…)
Clock 80 MHz
Data Valid
Data
Data Valid Wrong Usage
Clock 40 MHz
IDLE DATA DATA DATA DATA DATAIDLE
GBTX would split the frame in this case!!Synchronize your DV signal to the beginning of the GBT frame!
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 49
Data Valid
Some sub-detectors will connect more FEs to the same GBT transmitter:- Each FE with its own memory- Can happen that one can send DATA, the other cannot! (IDLE vs DATA in the same
packet)
Keep DV always high!
Clock 80 MHz
Data Valid
Data
Clock 40 MHz
IDLE DATA DATA DATA DATA DATAIDLE
Data Valid Many Channels
DATA DATA DATA DATA DATAIDLE
You HAVE to indicate whether the packet was DATA or IDLE, by sacrificing one bit your DATA/IDLE frame
LHCb Electronics Upgrade Meeting, 13/02/14 F. Alessio, CERN 50
Data Valid
Some sub-detectors will connect more FEs to the same GBT transmitter:- Each FE with its own memory- Can happen that one can send DATA, the other cannot! (IDLE vs DATA in the same
packet)
Keep DV always high!
Clock 80 MHz
Data Valid
Data
Clock 40 MHz
IDLE DATA DATA DATA DATA DATAIDLE
Data Valid Many Channels
DATA DATA DATA DATA DATAIDLE
You HAVE to indicate whether the packet was DATA or IDLE, by sacrificing one bit your DATA/IDLE frame