FE cards (L1FE1)• 4 outputs(DAQ)+1(Throttle)• 2 types of traffic

•HLT(1link)•Level 1 (up to 3)

•Ex: VELO_L1FE00_00_00

.

Multiplexing layer:2 types of switch

SubFarm •SFC(94-175)(SFC1), Ex: SFC_02• Switch(SFC_Switch1): 1I/16 O, Ex: SFC_Switch_02

•CPU (2000 in total)(SFC_Node1)Ex:SFC_Node_02_13

Sorter

TRM (1 module)

TFC system(readout supervisor)

HLT traffic

Level 1 traffic

mixed traffic

ethernet traffic

HLT possible link

Level 1 possible link

Storage System

1 component type names : they will be used through this presentation

ReadOutNetwork Data Flow

Readout Network Slot(RNS_Slot1)

RN

S_S

lot_

00

RN

S_S

lot_

15

Level 1(16I/2O) (L1MS)

Ex:L1MS_05

HLT (16 I /2 O)(HLTMS)

Ex:HLTMS_10

ReadOutNetwork :6 use cases (1/4)

1. Intrinsic parameters for a switch: A switch has a type (L1MS for example) and a name

(L1MS_05)and a hostname.Each switch type is characterized by a number of input

and output ports.

2. Intrinsic parameters for a port: A port is identified by a physical address. It is part of a

switch.It has a number which can be input or output. It has a

status(broken or not). It has a physical implementation link

(phy) that is to say it can optical or copper...It has a receiver and a transmitter flow control

(respec.RxFCand TxRC) which can be enabled or disabled. It has a set of

speedand you can force a specific one. A port can be a port management or not and has a wire

sense : itcan be switch-switch, switch-computer or autosense.

ReadOutNetwork :6 use cases (2/4)

3. Find the neighboring connections (up or down) of a switch by

specifying the direction of the data flow and the type of link

Example: I want to know to which port(s) of which switch aL1MS_05are connected, carrying L1 traffic in the downdirection.→ So it cames out that a link has :

- a start identifed by (switch name from, port number from) - an end identified by (switch name to, port number to) - a type (HLT, L1 or mixed) - a status: broken or not which is linked to the status of the involved ports.

ReadOutNetwork :6 use cases (3/4)

4. Find all the connection path between 2 components given a switch name,a switch type,the direction of the data flow and a type of the link.

Example: I want to know all the possible connexion paths from VELO_L1FE00_00_00 to RNS_Slot which carrie HLT Traffic. → So it cames out that a path is a sequence of links:

Sw

itch fro

mV

EL

O_

L1

FE

00

_0

0_

00

Sw

itch to

RN

S_

Slo

t

L1MS_05

100

15

HLTMS_10

2

3

4

8

5

20

0

HLT trafficLevel 1 traffic

path 1: [(VELO_L1FE00_00_00, 2), (HLTMS_10, 4), HLT] [(HLTMS_10,0),(RNS_Slot_03, 2), HLT]→ Selected

path 2 : [(VELO_L1FE00_00_00, 2), (HLTMS_10, 4), HLT] [(HLTMS_10,0),(RNS_Slot_03, 2), HLT]→Selected

path 3: [(VELO_L1FE00_00_00, 1), (L1MS_05, 5),L1] [L1MS_05,0),(RNS_Slot_11, 5), L1]→ Not Selected

path 4:[(VELO_L1FE00_00_00, 0), (L1MS_05, 1),L1] [L1MS_05,1),(RNS_Slot_11, 5), L1]→Not Selected

RNS_Slot_03

RNS_Slot_11

first link second link

ReadOutNetwork :6 use cases (4/4)

5. Check that each L1FE can be reached from each SFC_Node that is to say check all the L1FEs and all the SFC Nodes are connected through the RNS_Slot.In other words, a path must exist between each L1FE and aRNS_Slot and a path between each SFC_Node and a RNS_Slot.6. In the readout network, some of the links are grouped together to form a set of links which is also called a band. A band is found between 2 switches directly connected that is to say there is only one hop. So given a band link number, retrieve all the links belonging to it and vice versa, given a link, find the band link number.Example: I want to know all the links belonging to band link number 5. NB. Not all the links belong to a band link. → it comes out that a band is a collection of links.

Entity relationship modelSwitch +Switch Name+Switch type+Host name

Link+LinkID+Switch Name From+Port Number From+Switch Name To+Port Number To+Type of link

Path+PathID+Link

1

n

Switch Type+Switch type+nbr of input ports+nbr of output ports

Port+ physical address+ switch name+ nbr + input or output+ RxFC (Enable or not)+ TxFC (Enable or not)+ status (Broken or not)+ speed+ Port Management(yes/no)+ Wire sense (Switch/Switch,Switch/computer, Autosense)+ Phy: T, Sx, SL+ Forced Speed (Yes/No)

has

1 n

Associaton : a link has 2 ports

will determine the status of link

1 2Band link +BandID+LinkID

1 n

Table Design

Switch Connectivity+Switch_LinkID (pk)+Switch_From+Port_nbr_from+Switch_to+Port_nbr_to+link_type (fk) ref Link Type(link nbr)+ bidirectional_used+ Unique(Switch_From, port_nbr_from)+ Unique(Switch_to, port_nbr_to)+ Switch_From, Port_nbr_from (fk) ref Port(switch name,nbr,in_or_out=‘out’)+ Switch_to, Port_nbr_to (fk) ref Port(switch name,nbr,in_or_out=‘in’)

SwitchType+Switch type (pk)+nbr of input port+nbr of output port

Switch List+Switch Name (pk)+Switch type (fk) ref SwitchType(Switch_type)+host name

Band LinkLink_id (fk) ref SwitchConnectivity(Switch_LinkID)link_band_nbr (pk)Unique(Link_id)

Link Type+Link type+Link nbr (pk)

READOUTNET_PATH_L1FE

+L1FE link_readNet+MS link_readNet+RNS_Slot link_readNet

READOUTNET_PATH_SFC+RNS_Slot link_readNet+SFC link_readNet+SFC_Switch link_readNet+SFC_Node link_readNet

Link_readNet (object)+switch_from

(fk) ref Switch List(Switch_Name) +switch_to

(fk) ref Switch List(Switch_Name) +port_nbrfrom +port_nbrto +typeOfLink

1 n

Port+ physical @ (pk)+ switch name (fk)

ref Switch List(Switch_Name) + nbr+ in_or_out+ RxFC (Enable or not)+ TxFC (Enable or not)+ status (Broken or not)+ speed:(10,100)/(10,100,1000)/1000/10.000 Mbit/s+ Port Management(yes/no)+ Wire sense (Switch/Switch,+ Switch/computer, Autosense)+ Phy: T, Sx, SL+ Forced Speed (Yes/No)Unique(switch name,nbr,in_or_out)

11

1

nn

2

1n<11

bijection

THOR_00 (TFC)16*16

TTCtx_Xtransmitter(16*14)

TTCocX_YOpt. couplers(1*32)

TTCrxX_Y_Wreceiver (1*32)

SubDet_L0FEX_Y_Z(?*1)

SubDet_L1FEX_Y_Z(?*5)

Munin0_X32*2

DAQ (readout net)

ODIN_XRead out s.(?*16)

MuninL0_0016*16

MuninL1_0016*16

Throttle set of linksTFC set of linksDAQ set of linksTFC set of links sometimes possible

X ≤16 used 1

X≤16

Y≤14

W≤32

X≤16

TTCrxX_Yreceiver (1*32)

SubDet can be (VELO, RICH, L0MUON,...)

Z<16

TFC SYSTEM

TFC_System : use cases (1/3)

1. Intrinsic parameters for a switch: A device has a type (TTCtx for example) and a name (TTCtx_05).Each device type is characterized by a number of input and output ports.

2. Intrinsic parameters for a port: A port is part of a device. It has a number which can be input or

output. An output port can be disabled or not. It has a status (broken or not) and

aphysical location. Each output port has a delay setting which is downloaded once.Each input port of the THOR (switch) is associated with an address. Thisaddress consists of the address in hexa of the input port and a output

portnumber. It’s like a routine table.Ex: Input port 1 is associated with (0*1004 , 4) register@ output number

NB: Here the device looks like a switch but it is less smart. We use thisword to make the distinction between a “true” switch like in the readoutnetwork system.

TFC System : use cases (2/3)

3. Find the neighboring connections (up or down) of a switch by

specifying the direction of the data flow and the type of link

Example: I want to know to which port(s) of which switch aTTCtx_05are connected, carrying TFC traffic in the downdirection.→ So it cames out that a link has :

- a start identifed by (switch name from, port number from) - an end identified by (switch name to, port number to) - a type (TFC, throttle or DAQ) - a status: broken or not which is linked to the status of the involved ports.

TFC System : use cases (3/3)4. Find all the connection path between 2 components given a

switch name,a switch type,the direction of the data flow and a type of the link.

Example: I want to know all the possible connexion paths from TTCtx to VELO_L0FE00_00_00 which carrie TFC Traffic. → So it cames out that a path is a sequence of links:

Sw

itch

fro

mV

EL

O_

L0

FE

00

_0

0_

00

TT

Crx

00

_0

0_

00

path 1: [(VELO_L0FE00_00_00, 0), (TTCrx00_00_00, 0), TFC] [(TTCrx00_00_00, 0),(TTCoc00_00, 0), TFC] [(TTCoc00_00,0) , (TTCtx_00,0), TFC]->Selected

path 2: [(VELO_L0FE00_00_00, 1), (TTCrx00_00_11, 0), TFC] [(TTCrx00_00_11, 0),(TTCoc00_00, 11), TFC] [(TTCoc00_00,0) , (TTCtx_00,0), TFC]->Selected

TT

Crx

00

_0

0_

11

0

0

01

T

TC

oc0

0_

00

0

0

0

11 T

TC

tx_

00

0

0

first link

second linkthird link

Entity relationship modelDevice +Switch Name+Switch type

Link+LinkID+Switch Name From+Port Number From+Switch Name To+Port Number To+Type of link

Path+PathID+Link

1

n

Device Type+Device type+nbr of input ports+nbr of output ports Port

+ physical location+ switch name+ nbr + input or output+ status (Broken or not)+ Output_Disabled+ Delay_setting

has

1 n

Associaton : a link has 2 ports

will determine the status of link

1 2

Table Design

TFC Connectivity+Switch_LinkID (pk)+Switch_From+Port_nbr_from+Switch_to+Port_nbr_to+link_type (fk) ref Link Type(link nbr)+ bidirectional_used+ Unique(Switch_From, port_nbr_from)+ Unique(Switch_to, port_nbr_to)+ Switch_From, Port_nbr_from (fk) ref Port(switch name,nbr,in_or_out=‘out’)+ Switch_to, Port_nbr_to (fk) ref Port(switch name,nbr,in_or_out=‘in’)

DeviceType+device type (pk)+nbr of input port+nbr of output port

Device+Switch Name (pk)+Switch type (fk) ref SwitchType(Switch_type)

Link Type+Link type+Link nbr (pk)

TFC_PATH_ODIN00

+tfc_pathid+THOR+TTCtx+TTCoc+TTCrx+L0FE+L1FE+Munin0+MuninL

READOUTNET_PATH_SFC+tfc_pathid+THOR+TTCtx+TTCoc+TTCrx+L0FE+L1FE+Munin0+MuninL+ODIN

Link_readNet (object)+switch_from

(fk) ref Switch List(Switch_Name) +switch_to

(fk) ref Switch List(Switch_Name) +port_nbrfrom +port_nbrto +typeOfLink

1 n

Port+ physical location (pk)+ switch name (fk)

ref Switch List(Switch_Name) + nbr+ in_or_out+ status (Broken or not)+ Output_Disabled(on/off)+ Delay_setting number.Unique(switch name,nbr,in_or_out)

11

1

nn

2

bijection

view

Use Case triggers

• From the CfDB to PVSSDB– insert and update values– makes translation between values

• From the PVSSDB to CfDB– if a device is required by an user, download the parameters

such nb of input, output into the different tables– if a user makes new changes, make the updates or insert in

the cfDB

Use Case for partioning

1) Characteristics of a partition• A partition is a set of devices which can be run

independtly. • A device belongs to one active partition.• A partition in the TFC system will be characterized by

an ODIN number and (one or several subdetector(s) OR one or several TTCtx(s)): thus there is a starting and a final point.

• In the DAQ system, a partition will stop at the level of SFC:SFC_Switch and SFC_Node are not included.

• A partition is associated with a running mode, a version number and an owner. It will be active or not.

• A partition is either predefined(linked to one subdetector) or customized.

• A customized partition will be created from predefined one(s). Hence the need to write scripts to download them.

• Check the compatibility between the different running partitions, and between different created partition

-> In fact a partition is a projection (a view) of the TFC_Path:

Use Case for partioning

2) Characteristics of a running mode (or “ activity”)• A running mode is defined by the user: some are already

predefined(test, debugging,physic running mode). It has a priority.

• Ex of running mode:– calibration & alignment– cosmics – physics (the highest priority)

3) The identity card of a device.• A device is fully identified by a word of 128 bits which

consists of:– a system ID– a board type– a revision number– a serial number

Implementation requirements for the user interface

• From a user interface (for ex. within PVSS), the user should:– load a selected predefined partition from the Conf.DB– load a previous created customized partition from the

Conf.DB– create a customized partition and store it in the

Conf.DB– run any partition (from PVSS) provided that it satisfies all

requirements.– configure the THOR routine table and store it in the

Conf.DB

THOR routine table• Associated with a Partition number • Identified by a number• The way it is stored in the database (cf schema below):• input : 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0• output : 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0• like words of 16 bits: 1 indicates the port is busy.• possible because output ports are driven only by one input port.

THOR switch

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

connection

busy

Implementation requirements for the user interface

• Implement scripts to check that :– the user has the rights to create and/or run partitions:

(for ex. a guy from the VELO can create only partitions relating to VELO FE cards).

– the selected ODIN for a running a partition is still free.– the routine THOR table is coherent with the

partition(same ODIN and TTCtx names used in both cases for ex.)

– this partition can be performed along with the running ones.

– if it is a physic running mode(and the first one), an alert should be sent to other running partitions to warn them and also to stop all the running partitions.

– the ended and started time are empty when the partition stops running, in the Conf.DB.

• Create datapoint type and datapoints (up to 12 : maximum nb of independent partition, for ex.)which can be to fit partitions structure (for PVSS)

Thor Routine

RoutineIdPartitionIdDevice nameOutput statusInput status

Predefined_Partition

PartitionId_Version NumberOwnerODIN_nameStarted pointFinal pointRunning ModeActive

Running Mode

Mode_nameAuthorDate of creationPriority

Customized_Partition

PartitionIdVersion NumberOwnerODIN_nameStarted pointFinal pointRunning ModeActive

Entity relationship model

HAS

1 1

1 1

HAS

HAS

HAS

1 1

1 1

Thor RoutineRoutineId number(pk)

THOR_name(fk) references device_list(devname)Output _port _nb number

Output _port _@ varchar

Input _port_nb numberunique(THOR_name,output_port_nb)

Running Mode

RunMode_name varchar (pk)

Author varchar

Date of creation date

Priority number

TFC_Partition

PartitionId_VersionNb varchar

Owner varcharODIN_name (fk) references device_list(devname)Start_End link_readnetActive numberPartitionId_VersionNb, Start_End (pk)

Partit_ActPartitionId_ VersionNb varchar (fk) references TFC_Partition (PartitionId_ VersionNb)

RunMode_name varchar (fk) references Running Mode(RunMode_name)unique(partitionid_VersionNb,RunMode_name)

Partit_Thor_routinePartitionId_ VersionNb varchar (fk) references TFC_Partition (PartitionId_ VersionNb)

RoutineId varchar (fk) references Thor Routine(RoutineId)unique(partitionid_VersionNb,RoutineId)

Link_readnetswitch_from:starting pointswitch_to: final pointport_nbrfrom:port nbr starting pointport_nbrto:port nbr final pointlink_type:0 for predefined part.,1 for cust. exclusion,2 for cust. inclusionLinkId:tfc_path_id

SFC_Node

SFC_Node

name: varchar2 IP @: control /data varchar2 Ethernet @ : varcharPhysical @ : varcharType :varcharKernel Image (location) : varcharInitrd image (location): varchar

Control PC

name: varcharIP @: varchar1 Ethernet @ : varcharPhysical @ : varcharType :varcharKernel Image (location) : varcharInitrd image (location): varcharSFC 1 supervisedSFC 2 supervised