25
Orsay 16 November.2011 uTCA fo 21cm 1 Using xtca crate for 21cm correlator Thanks to Jean-Pierre Cachemiche uTCA Global architecture Development status ATCA architecture ADC board Conclusion

uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

  • Upload
    others

  • View
    3

  • Download
    0

Embed Size (px)

Citation preview

Page 1: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 1

Using xtca crate for 21cm correlator

Thanks to Jean-Pierre Cachemiche

uTCA Global architecture

Development status

ATCA architecture

ADC board

Conclusion

Page 2: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 2

Motivations

Why uTCA?

Dedicated backplane for high-speed data communication.Scalable system : between 1 to 10 communication board

(AMC)No backplane to design : Standard product were, clock

distribution, system control, power supply sequencing, are take into account.

Little board 72mm x 160 mmEvolutionary system.1 on the shelf crate controller (MCH).Up to 360 10Gbit optical linkLess expensive than ATCA standard.

Page 3: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 3

Mechanic

Page 4: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 4

Connectivité backplane :Full Mesh ou Dual Star

Page 5: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 5

Connectivity between board

4 bidirectional pairs

2 bidirectional pairs

ECS

Throttles,Clocks,TFC information

Relies on powerful connectivity coming with the ATCA standard

Page 6: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 6

Links architecture

AMC In

AMC In

AMC In

AMC In

AMC In

AMC In

AMC In

AMC In

CrossbarMCH1

AMC out

CrossbarMCH2

AMC out

44

44

44

44

44

44

44

44

44

44

36 x10 Gbits/s 8 x 10 Gbits/s

8 x10Gbits/s

10 x10 Gbits/s

Page 7: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 7

Architecture

Stratix VFPGA

Crossbar144 x 144

max6.5 Gbits/s

Cyclone IIIFPGA

3 x 12 channels receivers

1 Gigabit Ethernet transceiver

Acquisition board (AMC)

NAT MCH (Tongues 1 and 2)

Supervision

NIOS

Standard backplane(dual star layout)

Stratix VFPGA

1X12 channelsemiters

Output board (AMC)

NIOS

(36 - 10 Gbits/s)

(36 x 10 Gbits/s)

10 boards

36

1

mezzanine board

mezzanine board

Switch board (Tongues 3 and 4)

GigabitEthernetSwitch

CPU

Ethernet link

SMAClock

distribution

Tongue 1

Tongue 2

Tongues 3 & 4

External clock

4 to MCH14 to MCH2

PHY

4 to MCH14 to MCH2

Page 8: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 8

Control system

Integer in a FPGAs

Base on Gigabit Ethernet

1 input at MCH board and distribution by backplane to all AMC board.

NIOSII cpu + serveur integer on each AMC board

SwitchGbit Ethernet

Nios +µCLinux

FPGA

Carte MCH

Carte AMC

Nios +µCLinux

FPGA

Carte AMC

Nios +µCLinux

FPGA

Carte AMC

Nios +µCLinux

FPGA

Carte AMC

Backckplane

Carte MCH

Page 9: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 9

MCH board

Cyclone IIIEPC3C25

Cyclone IIIEPC3C25

72 x 72crossbar

72 x 72crossbar

6 x 4

To

ngu

e 4

Backplaneconnector

NIOS core

To

ngu

e 3

PCB 4

6 x 4

6 x 4

6 x 4

Command bus

NAT MCHconnector

NAT MCHconnector

I2C, SPIGbit Ethernet

MarvellPHYDDR2

VS3441

VS3441

I2C bus

CCPM development

Page 10: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 10

Mécanique

17

mm

12

,9 m

m

1,6

mm

22

,9 m

m

23

,76

mm

Maximum component height

22,9 mm

3,9

mm

24

,46

mm1,6

mm

24

,56

mm

Maximum height

NAT mother board

Prototype control board

Tongue 1

Tongue 2

Tongue 3

Tongue 4

63.38 mm

Page 11: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 11

Example of MCH board

Page 12: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 12

Status

Page 13: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 13

AMC Board

• Now days in development at CCPM (J.P Cachemiche IN2P3 Marseilles).• Up to 36 10Gbits optical link.• Custom by re design optical mezzanine.• ~ 6000Euro Tbp

Stratix V GXStratix V GX

FLASHFLASH DDR3DDR3

@ @Data Data

MMCMMC

Clock out

1 serial links 4 serial links

2 PCIe links

1 GBe link

PCIe Clock

PLLPLLClock In

Optical InterfaceOptical Interface 36 serial links

Jittercleaner

Jittercleaner

ECS path

AMCconnector

X_FPGA path

TFC path

Throttle in, out

Clock In and out

Page 14: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 14

First version of AMC board

181 mm

73 m

m

Page 15: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 15

Mezzanine

76 mm

68 m

m

Page 16: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 16

Mesures a 8 Gbits/s

Serial link at 8 Gbits/s with GBTprotocol

18 hours of ruining without error: BER ~ 2 10-15

Measure Jitter à 10-12

Total : 77 ps (p to p)Random : 3.0 psDeterministic : 36 ps

UI= 125 ps

Page 17: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 17

Bridge architecture

Stratix VFPGA

Crossbar144 x 144

max6.5 Gbits/s

Cyclone IIIFPGA

3 x 12 channels receivers

1 Gigabit Ethernet transceiver

Acquisition board (AMC)

NAT MCH (Tongues 1 and 2)

Supervision

NIOS

Standard backplane(dual star layout)

Stratix VFPGA

1X12 channelsemiters

Output board (AMC)

NIOS

(12 - 10 Gbits/sEthernet

10 boards

24

1

mezzanine board

mezzanine board

Switch board (Tongues 3 and 4)

GigabitEthernetSwitch

CPU

Ethernet link

SMAClock

distribution

Tongue 1

Tongue 2

Tongues 3 & 4

External clock

4 to MCH14 to MCH2

PHY

4 to MCH14 to MCH2

(12 - 5 Gbits/sSerial LiteII

(12 - 5 Gbits/sSerial LiteII

(12 - 10 Gbits/sEthernet

(12 - 5 Gbits/sSerial LiteII

(12 - 5 Gbits/sSerial LiteII

Page 18: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 18

ATCA standard

Page 19: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 19

ATCA standard

ATCA Standard: 2 solutions1) “custom“ use.2) Full standard compatibility

Page 20: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 20

ATCA generic board

CCPCCCPC

J2J2

J1J1

J3J3

SerialCrossbar

SerialCrossbar

Fabric Channel 1

Fabric Channel 2

Fabric Channels3 to 14

Base Channels3 to 14

ShMC port

Base Channel 2

AMCAMC

AMCAMC

AMCAMC

AMCAMC

12

24

GbE 1

GbEswitch

GbEswitch

12

12

ClockCrossbar

ClockCrossbar

1

TFC

4

4

16

Clock

Throttle

4

SerialCrossbar

SerialCrossbar

1

122

1

TFC

4

Clock

Throttle

1

TFC

4

Clock

Throttle

1

TFC

4

Clock

Throttle

4

1

PCIe

Input/output:144 links

Page 21: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 21

ATCA standard

Page 22: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 22

ADC board

Page 23: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 23

LAL ADC board

➢ 4 input channels (digitized at 500 MHz max, 8bits)

➢ Input band-with 1.7GHhz.

➢ Time resolution ~10ps

➢ FFT 8K on each channel.

➢ Input clock + control (start/stop…) ports

➢ USB, and VME communications port (for control)

➢ 2 high speed (~ 5 Gbit/s) optical transceiver (mono mod).

➢ ~ 4500Euro

Page 24: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 24

On the shelf ADC board

• ICS1650A-001• PCI Express 250MHz/ch Digital Receiver (analog input), • 4 channels, embeded FPGAVirtex-5• V5SX95T, 12-bit resolution, 16MB mémory• 8400Euro.

Page 25: uTCA Global architecture Development status ATCA ......Architecture Stratix V FPGA Crossbar 144 x 144 max 6.5 Gbits/s Cyclone III FPGA 3 x 12 channels receivers 1 Gigabit Ethernet

Orsay 16 November.2011 uTCA fo 21cm 25

Conclusion

uTCA plus CCPM AMC board interesting solution for link bridge and data massaging.

Beam forming and visibility capability.

Easily scalable for future evolution.

Limited hardware development (optical mezzanine)

Rugged uTCA.