September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Shore station brainstorm 1

September 11-12, 2013 KM3NeT, CLBv2 Workshop Valencia

Peter JansweijerNikhefAmsterdamElectronics- Technology

Shore station brainstorm

1



1. Switch Routing Table (software?) needs to be adjusted.2. PTP timestamps t1 for all DOMs are equal and reside in outgoing port (needs firmware- or software-

change or both)3. MAC Control-Level multicast MAC addresses; such as “pause frames” for flow control need to be

handled correctly. Example:◦ “DOM-B” request “Pause” = Okay (request over point to point link), but…◦ “Shore station” request “Pause” may be problematic (request over broadcast link, all ports are stalled).

Address single DOM?

4. Other surprises?

Shore Station Broadcast brainstorm

DOMA

DOMB

DOMC

DOMD

BufferBufferBufferBuffer

Port-2SFP

Broadcast

OpticalNetwork

Start

Tx

t4 Stop1

ReferenceClock

PTP

Time Stamp

t1

Time Stamp

t4

t4 Stop2Time

Stamp t4

t4 Stop3Time

Stamp t4

t4 Stop4Time

Stamp t4Shore Station interface

2

Rxj: DDMTD

Rxj: DDMTD

Rxj: DDMTD

Rxj: DDMTD

Port-1SFP

Main Electrical Optical Cable

Port-3SFP

Port-4SFP

Port-5SFP



Other surprises:PTP t1,t4 timestamps over multiple switches

3

PTP t1

Broadcast

PTP t4

DOM inputs

PTP t4

DOM inputs

PTP t4

DOM inputs

1. Redistribution of PTP t1 not only within one switch…

2. Routing PTP frames is an issue (layer-2; no IPv4!)… Note that t4 needs to be routed to the appropriate DOM via the broadcast channel.

…over multiple switches (depending on detector broadcast network segmentation)

This needs brainpower!

One t1

Many t4



Brainstorm-1

4

PTP t1

Broadcast

PTP t4

DOM inputs

PTP t4

DOM inputs

PTP t4

DOM inputs

1. When layer-2 is not routable then distribute the timestamps using your custom protocol.

2. … kind of awkward! And it may consume switch ports

1 Gbps WR

10Gbps COTS

1 Gbps WR 1 Gbps WR



Brainstorm-2

5

PTP t1

Broadcast

PTP t4

DOM inputs

PTP t4

DOM inputs

PTP t4

DOM inputs

1. Layer-2 is not routable, then…

2. Wrap PTP t1 in a layer-3 message and route this to the rest of the switches.

3. Each timing input calculates timing servo action and passes this to the broadcasts port.

1 Gbps WR

10Gbps COTS

1 Gbps WR 1 Gbps WR



Tempting thoughts(that shouldn’t be thought at this stage in the design

cycle…)

6



Current major troubles in the design

7

Optical network with 80 channels @ 50 GHz is very tempting… (will it be possible?)

Broadcast networking is difficult to implement.

Broadcast network increases “asymmetry” =>this complicates timing calibration.

Each DOM has a different wavelength laser. This is not disastrous but requires more careful tracking of DOMs.



Tempting idea: a WR-Switch in the DU(Paul Kooijman)

8

1 Gbps WR

Shore

1 Gbps WR 1 Gbps WR

1 Gbps WR, 10Gbps data

Sub-Sea

A tempting idea to put a WR switch in each DU!

But it solves lots of problems!



Tempting idea: a WR-Switch in the DU(Paul Kooijman)

9

One WR switch per DU (1x16 or 2x8 DOMs, later redesign 1x18

or 2x9 DOMs). + Better bandwidth utilization per wavelength => + Less wavelengths needed => possibility for

bidirectional network + Even with bidirectional network the optical network

easier! + cost effective! Inexpensive SFPs in the string and a

limited amount of expensive long haul SFPs (no needed EDFA’s anymore)

+ No broadcast anymore! All point to point. Timing is distributed as it is meant to be => “Normal network”

+ All DOMs can be the same (colorless) - More complexity sub-sea. - Shore station switch must handle 10 Gbps (WR only

available on 1 Gbps) - SFP reliability not proven!

6 x plus!

Only 2 minus

1 general worry!



Tempting idea: distributed WR-Switch(es) in the CLB (Henk Peek)

10



11

+ Redundancy + Better bandwidth utilization per wavelength => + Less wavelengths needed => possibility for bidirectional

network + Even with bidirectional network the optical network easier! + cost effective! Inexpensive SFPs in the string and a limited

amount of expensive long haul SFPs (no needed EDFA’s anymore)

+ No broadcast anymore! All point to point. Timing is distributed as it is meant to be => “Normal network”

+ All DOMs can be the same (colorless) - More complexity sub-sea. - Needs further CLB development (implement simple static

switch functionality) - Shore station switch must handle 10 Gbps (WR only

available on 1 Gbps) - SFP reliability not proven!

7 x plus!

3 x minus

1 general worry!

Tempting idea: distributed WR-Switch(es) in the CLB (Henk Peek)



Conclusion on tempting thoughts

12

These thoughts shouldn’t be thought at this stage in the design cycle…

But please let us re-think before proceeding! We are designing an awkward system right now.

If we take a step back now we may have an operational KM3NeT detector faster!

I know… This involves all disciplines:◦ Hardware◦ Software (+ Networking)◦ Mechanics

◦ Firmware◦ Optical

network◦ Power



Backup slides

13



7500 Hz hitrate/ PMT48 bit/hit=> 7500 * 48= 360.000 bps/PMT31 PMTs / DOM360.000 * 31 ~ 12 Mbps / DOM18 DOMs / string12 Mbps * 18 = 216 Mbps / string

= 21% of a 1 Gbps channel

Bandwidth

14



WR I2C buses

15

Soft PLL FMC

SFP

EPROM

SCL

SDA

SFP+MOD_DEF1

SFP_MOD_DEF2

C10

C18

SCL

SDA

SCL

SDA

C30

C31

KC705

FMC

-J2

2

FPGA

H30

B28

K21

L21

U49PCA954

8(I2C mux)

IIC_SDA_MAIN

IIC_SCL_MAIN

clb_xwr_core

xwc_periph

SCL

SDA

SFP_SCL

SFP_SDA

SY

SC

ON

Reg

s

GPIO Set/Readback Register

Bit 2: SCLBit 3: SDABit 8: SFP_SCLBit 9: SFP_SDA

Ch 1

Documents

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Shore station brainstorm 1