Upload
others
View
11
Download
0
Embed Size (px)
Citation preview
September 2011 Altera Corporation
AN-570-1.2 Application Note
© 2011 Altera Corporation. All rights reserved. ALTERA, ARRIA, CYCLONE, HARDCOPY, MAX, MEGACORE, NIOS, QUARTUS and STRATIX words and logos are trademarks of Altera Corporation and registered in the U.S. Patent and Trademark Office and in other countries. All other words and logos identified as trademarks or service marks are the property of their respective holders as described at www.altera.com/common/legal.html. Altera warrants performance of its semiconductor products to current specifications in accordance with Altera's standard warranty, but reserves the right to make changes to any products and services at any time without notice. Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.
101 Innovation DriveSan Jose, CA 95134www.altera.com
Subscribe
ISO 9001:2008 Registered
AN 570: Implementing the 40G/100GEthernet Protocol in Stratix IV Devices
IntroductionAltera’s 40 nm Stratix® IV GX and Stratix IV GT devices provide a complete solution for developing 40 Gbps and 100 Gbps network line card applications.
This application note describes how to implement a 40 Gbps/100 Gbps (40G/100G) Ethernet line-side interface using Stratix IV GT devices. It focuses on the following two implementation aspects:
■ Transceiver configuration and clocking recommended for 40G/100G Ethernet
■ Interfacing 40G/100G Ethernet Soft IP to the transceivers
Figure 1 shows an example 40G/100G network line card application with Stratix IV GT and Stratix IV GX devices.
The three-chip 40G/100G line card solution shown in Figure 1 consists of:
Figure 1. Altera’s 40G/100G Network Line Card Application (1)
Note to Figure 1:(1) Figure 1 shows the 40G/100G MAC, PCS, and PMA designed in a Stratix IV GT device with integrated 10 Gbps serial transceivers. You can
implement this application in Altera’s Stratix IV GX and HardCopy® IV GX devices with external RXAUI to XFI multiplexer chips between the Altera device and the optical module. For more information, refer to Using 10-Gbps Transceivers in 40G/100G Applications.
Opticalmodule
Look-uptables
Packetbuffer
TM linklist
Stratix IV GX Stratix IV GX
40G/100G chip-to-backplaneInterlaken
SPAUI and DDR-XAUI
40G/100G chip-to-chipSystem Side Interface
Interlaken or SerialLite II
40G/100G chip-to-moduleLine Side Interface
XLAUI (4 lanes @ 10G)CAUI (10 lanes @ 10G)
Switch fabric/backplane
RLDRAM IIDDR3QDR II/III
40G/100GMAC
Trafficmanagement
Packetprocessing
Stratix IV GT
Page 2 Stratix IV GT Hard IP Blocks Used in 40G/100G Ethernet Implementation
September 2011 Altera Corporation AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
■ 40G/100G Ethernet MAC device
■ Targeted to Stratix IV GT EP4S40G2 or EP4S40G5 devices for 40G Ethernet and to EP4S100G2, EP4S100G3, EP4S100G4, or EP4S100G5 devices for 100G Ethernet
■ Line side interface to optical module comprising 4 or 10 channels, each configured at a serial data rate of 10.3125 Gbps for 40G or 100G Ethernet, respectively
The 40G Ethernet line side interface targets compliance to XLAUI electrical specifications
The 100G Ethernet line side interface targets compliance to CAUI electrical specifications
■ System side interface to the packet processing Stratix IV GX FPGA consisting of a multi-channel Interlaken or proprietary SerialLite II link with each channel configured at data rates up to 6.375 Gbps
■ Packet Processing and Traffic Management devices
■ Targeted to Stratix IV GX devices with transceivers configured to implement Interlaken or SerialLite II links
Stratix IV GT Hard IP Blocks Used in 40G/100G Ethernet ImplementationThe 40G Ethernet MAC implementation uses four full-duplex Stratix IV GT transceiver channels at 10.3125 Gbps, as shown in Figure 2.
Figure 2. 40G MAC Implementation in a Stratix IV GT Device
40G MAC
Interlakenor
SerialLite IISoft IP
40G PCS
VL2
VL3
Ch0 Ch0
Ch2 Ch2
Ch1 Ch1
Ch3 Ch3 10.3125G
10.3125G
10.3125G
10.3125G
VL0
VL1
40
40
40
40
CustomLogic
XLAUI
40
40
40
40
Gearbox
Gearbox
Gearbox
Gearbox
66
66
Line Side InterfaceSystem Side Interface
66
66
66
66
Soft IP Hard IP
66
66
FourTransceiverChannels @10.3125G
Interlakenor
SerialLite IITransceivers
40G PMD(Optical
modules)
Stratix IV GT Soft IP Blocks Used in 40G/100G Ethernet Implementation Page 3
September 2011 Altera CorporationAN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
The 100G Ethernet MAC implementation uses 10 full-duplex Stratix IV GT transceiver channels at 10.3125 Gbps, as shown in Figure 3.
Stratix IV GT Soft IP Blocks Used in 40G/100G Ethernet ImplementationStratix IV GT devices do not have dedicated hard IP blocks that implement 40G/100G Ethernet Media Access Control (MAC), Physical Coding Sublayer (PCS), and gearbox logic. These blocks must be implemented with a soft IP core in the FPGA fabric.
1 40G/100G Ethernet soft IP cores targeted to Stratix IV GT devices are available from Altera’s IP partners MorethanIP (www.morethanip.com).
Physical Coding Sublayer (PCS) Block In the egress direction, the PCS block is responsible for functions such as 64B/66B encoding, scrambling using an X58 + X39 + 1 polynomial, and striping the 66-bit data blocks into virtual lanes (VLs). In the ingress direction, it is responsible for functions such as 66B/64B decoding, descrambling using an X58 + X39 + 1 polynomial, and destriping the VLs into 66-bit data blocks. It also performs status monitoring and decoding functions as specified in the IEEE802.3ba specification.
Media Access Control (MAC) LayerThe MAC layer is responsible for functions such as Ethernet frame encapsulation and decapsulation, Frame Check Sequence (FCS) generation and checking, flow control, and reconciliation sublayer functions as required by the IEEE802.3ba specification.
Figure 3. 100G MAC Implementation in a Stratix IV GT Device
100G MAC 100G PCS
VL18
VL19
Ch0Ch0
Ch9 Ch9 10.3125G
10.3125G
VL0
VL1
20
20
20
20
CAUI
20
20
20
20
40
40
66
6666
66
Mux/Demux
Mux/Demux
66
66
Soft IP Hard IP
66
66
Ten TransceiverChannels @10.3125G
Ch9
Ch0
100G PMD(Optical
modules)
40
40
Interlakenor
SerialLite IISoft IP
CustomLogic
System Side Interface Line Side Interface
Interlakenor
SerialLite IITransceivers
Gearbox
Gearbox
Gearbox
Gearbox
Page 4 Stratix IV GT Transceiver Configuration for 40G/100G Ethernet Implementation
September 2011 Altera Corporation AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
Gearbox LogicThe 40G/100G Ethernet data is 64B/66B encoded in the egress direction and 66B/64B decoded in the ingress direction. Each VL from or into the PCS block is 66 bits wide. When configured at 10.3125 Gbps for 40G/100G Ethernet, the Stratix IV GT transceiver channels support a parallel data width of 40 bits.
In the egress direction, the gearbox logic is responsible for translating each 66-bit wide VL from the PCS into 40-bit wide data to the Stratix IV GT transmitter channel. In the ingress direction, the gearbox logic is responsible for translating the 40-bit wide data from each Stratix IV GT receiver channel into 66-bit wide data to the PCS.
Stratix IV GT Transceiver Configuration for 40G/100G Ethernet Implementation
The following sections discuss the required Stratix IV GT transceiver configuration and the recommended clocking for 40G/100G Ethernet implementation:
■ “Required Transceiver Channel Placement”
■ “Transceiver Channel Datapath Configuration” on page 8
■ “40G Ethernet Transceiver Channel Datapath Clocking” on page 10
■ “100G Ethernet Transceiver Channel Datapath Clocking” on page 12
Required Transceiver Channel Placement Stratix IV GT EP4S40G2 and EP4S40G5 devices support up to six channels on each side of the device. Each channel can operate at 10.3125 Gbps. Altera recommends using these devices to implement a 40G Ethernet link.
Stratix IV GT Transceiver Configuration for 40G/100G Ethernet Implementation Page 5
September 2011 Altera CorporationAN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
Figure 4 shows the transceiver channel locations in EP4S40G2F40 and EP4S40G5H40 devices that support 10.3125 Gbps data rate to implement a 40G Ethernet link.
Stratix IV GT EP4S100G2 and EP4S100G5 devices support up to 12 channels on each side of the device. Each channel can operate at 10.3125 Gbps. Altera recommends using these devices to implement a 100G Ethernet link.
Figure 4. Transceiver Channel Locations for 40G Ethernet Implementation
8G Transceiver Channel
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0/1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
10G ATX PLL
6G ATX PLL 6G ATX PLL
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
EP4S40G2F40EP4S40G5H40
GXBR1GXBL1
GXBL0 GXBR0
GXBL2 GXBR2
Four transceiverchannels at
10.3125 Gbps oneach side for 40G Ethernet
implementation
2.488-10.3125 GbpsTransceiver Channels
2.488-8.5 GbpsTransceiver Channels
2.488-6.5 GbpsCMU Channels (PMA-Only)
9.9 Gbps-11.3 GbpsATX PLL
4.8 Gbps-6.5 GbpsATX PLL
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel 8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
10G XCVR Channel
Page 6 Stratix IV GT Transceiver Configuration for 40G/100G Ethernet Implementation
September 2011 Altera Corporation AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
Figure 5 shows the transceiver channel locations in EP4S100G2F40 and EP4S100G5H40 devices that support 10.3125 Gbps data rate to implement a 100G Ethernet link. Figure 6 shows the transceiver channel locations in the EP4S100G3F45/EP4S100G4F45/EP4S100G5F45 devices that support 10.3125 Gbps data rate to implement a 100G Ethernet link.
Figure 5. Transceiver Channel Locations for 100G Ethernet Implementation in EP4S100G2F40 and EP4S100G5H40 Devices
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
CMU0/1 Channel
10G ATX PLL
6G ATX PLL
2.488-10.3125 GbpsTransceiver Channels
2.488 Gbps-6.5 GbpsCMU Channels (PMA-Only)
9.9 Gbps-11.3 GbpsATX PLL
4.8 Gbps-6.5 GbpsATX PLL
GXBL3
GXBL2
GXBL1
GXBR3
GXBR2
GXBR1
Ten transceiverchannels at
10.3125 Gbps oneach side for 100G Ethernetimplementation
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel 10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
EP4S100G2F40EP4S100G5H40
Stratix IV GT Transceiver Configuration for 40G/100G Ethernet Implementation Page 7
September 2011 Altera CorporationAN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
Figure 6. Transceiver Channel Locations for 100G Ethernet Implementation in an EP4S100G5F45 Device
Note to Figure 6:(1) EP4S100G3F45 and EP4S100G4F45 devices only support up to 12 10G channels on each side of the device.
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
EP4S100G3F45EP4S100G4F45EP4S100G5F45
CMU0/1 Channel
10G ATX PLL
6G ATX PLL
2.488-10.3125 GbpsTransceiver Channels
2.488 Gbps-6.5 GbpsCMU Channels
(PMA-Only)
9.9 Gbps-11.3 GbpsATX PLL
4.8 Gbps-6.5 GbpsATX PLL
GXBL3
GXBL2
GXBL1
GXBR3
GXBR2
GXBR1
Ten transceiverchannels at
10.3125 Gbps oneach side for 100G Ethernetimplementation
CMU1 Channel
CMU0 Channel
6G ATX PLL
CMU1 Channel
CMU0 Channel
6G ATX PLL
GXBL0 GXBR0
10G XCVR Channel 10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel 10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
(1)
(1)
Page 8 Stratix IV GT Transceiver Configuration for 40G/100G Ethernet Implementation
September 2011 Altera Corporation AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
Transceiver Channel Datapath Configuration In a 40G or 100G Ethernet link, each of the 4 or 10 transceiver channels operates at 10.3125 Gbps. The Stratix IV GT transceiver channel must be configured in Basic Double-Width mode with a 40-bit FPGA fabric-transceiver interface with a low latency PCS datapath, as shown in Figure 7.
f For more information about the low latency PCS datapath, refer to “Low Latency PCS Datapath” in the Stratix IV Transceiver Architecture chapter in volume 2 of the Stratix IV Handbook.
Figure 7. Low Latency PCS Transceiver Configuration Required for Implementing 40G/100G Ethernet LinkR
X P
hase
C
ompe
nsat
ion
FIF
O
Byt
e O
rder
ing
Byt
e D
eser
ializ
er
8B10
B D
ecod
er
Rat
e M
atch
FIF
O
Receiver Channel PCS Receiver ChannelPMA
Des
kew
FIF
O
Wor
d A
ligne
r
rx_datain@
10.3125 Gbps
Des
eria
lizer
CD
R
TX Phase Compensation
FIFO
ByteSerializer
8B10B Encoder
Transmitter Channel PCS
Low Latency PCS Datapath
Transceiver Channel PMA
tx_dataout@
10.3125 GbpsSerializer
FPGAFabric
40 20 20
20
20
40
40
Stratix IV GT Transceiver Configuration for 40G/100G Ethernet Implementation Page 9
September 2011 Altera CorporationAN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
Figure 8 shows the ALTGX MegaWizard™ Plug-In Manager setup for 40G/100G Ethernet implementation.
Figure 8. ALTGX Instance for 40G/100G Ethernet Implementation
40G Ethernet = 4 channels100G Ethernet = 10 channels
Page 10 Stratix IV GT Transceiver Configuration for 40G/100G Ethernet Implementation
September 2011 Altera Corporation AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
40G Ethernet Transceiver Channel Datapath ClockingFigure 9 shows the recommended transceiver channel datapath clocking for implementing a 40G Ethernet link in Stratix IV GT devices.
Figure 9. Transceiver Channel Datapath Clocking in 40G Ethernet Implementation
RX
Pha
se
Com
pens
atio
n F
IFO
TX Phase Compensation
FIFO
Byt
e O
rder
ing
Byt
e D
eser
ializ
er
Byte Serializer
8B/1
0 D
ecod
er
8B10B Encoder
Rat
e M
atch
FIF
O
Receiver Channel PCS Receiver Ch0 PMAReceiver Ch1 PMA
Receiver Ch2 PMAReceiver Ch3 PMA
Channel1Channel0
Channel1Channel0
Receiver Channel PCS
Des
kew
FIF
O
Wor
d A
ligne
r
Des
eria
lizer
CD
R
Transmitter Channel PCS
Transceiver Block
Transmitter Channel PCSTransmitter Channel PCS
Transceiver Ch0 PMATransceiver Ch1 PMA
Transceiver Ch2 PMATransceiver Ch3 PMA
Ser
ializ
er
tx_d
atao
ut[3
:0] @
10.
3125
Gbp
s×
1 H
igh_
Spe
ed S
eria
l and
Low
-Spe
ed P
aral
lel C
lock
rx_d
atai
n[3:
0] @
10.
3125
Gbp
s
wrclk rdclk
/2
/2
FPGAFabric
Input Reference Clock at644.53 MHz/322.26 MHz
5156.25 MHz
515.625 MHz
515.625 MHz
CMU0_PLL
257.8125 MHztx_clkout[3:0]
257.8125 MHzrx_clkout[3:0]
CMU0_Channel
CMU0 Clock Divider
Channel2 Receiver Channel PCSReceiver Channel PCSChannel3
Channel2Transmitter Channel PCSChannel3
40 4020
20
20
20
20
40
8G Transceiver Channel
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
EP4S40G2F40EP4S40G5H40
GXBR1GXBL1
GXBL0 GXBR0
GXBL2 GXBR2
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel 8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
8G XCVR Channel
Stratix IV GT Transceiver Configuration for 40G/100G Ethernet Implementation Page 11
September 2011 Altera CorporationAN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
Transmitter Channel Datapath ClockingThe CMU0 PLL located in the transceiver block containing four channels is used to generate the high-speed serial and low-speed parallel transmitter clocks. Table 1 lists the allowed input reference clock frequencies and recommended input reference clock sources for optimum jitter performance.
The high-speed serial clock at 5156.25 MHz and the low-speed parallel clocks at 515.625 MHz generated by the CMU0 PLL are distributed to the four transmitter channels through the ×1 clock line. The PMA and PCS blocks within all four channels use the high-speed serial and low-speed parallel clock for their operation.
f For more details about the ×1 clock network, refer to the Stratix IV Transceiver Clocking chapter in volume 2 of the Stratix IV Handbook.
The low-speed parallel clock of 515.625 MHz to each transmitter channel gets further divided-by-two at the byte serializer. This divided-by-two clock at 257.8125 MHz is forwarded as the FPGA fabric-transceiver interface clock on each channel’s tx_clkout port. You can use the tx_clkout port from one of the four channels to clock the 66-bit to 40-bit gearbox logic, as well as the PCS layer in the FPGA. Refer to “Gearbox Logic” on page 4 for more details.
Receiver Channel Datapath ClockingAs shown in Figure 9, the clock and data recovery (CDR) in each of the four receiver channels forwards the high-speed serial (5156.25 MHz) and low-speed parallel (515.625 MHz) recovered clocks to the receiver PMA and PCS blocks for their operation. The low-speed parallel recovered clock of 515.625 MHz gets further divided-by-two at the byte deserializer. This divided-by-two clock at 257.8125 MHz is forwarded as the FPGA fabric-transceiver interface clock on each channel’s rx_clkout port. You can use the rx_clkout port from one of the four channels to clock the 40-bit to 66-bit gearbox logic, as well as the PCS layer in the FPGA. Refer to “Gearbox Logic” on page 4 for more details.
Table 1. Input Reference Clock Recommendation for 40G Ethernet Link
Input Reference Clock Frequency Recommended Input Reference Clock Source for Optimum Output Jitter
644.53125 MHz Dedicated refclk0 or refclk1 pin of the transceiver block containing the four 10.3125G channels322.265625 MHz
Page 12 Stratix IV GT Transceiver Configuration for 40G/100G Ethernet Implementation
September 2011 Altera Corporation AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
100G Ethernet Transceiver Channel Datapath ClockingFigure 10 shows the recommended transceiver channel datapath clocking for implementing a 100G Ethernet link in Stratix IV GT devices.
Figure 10. Transceiver Channel Datapath Clocking in 100G Ethernet ImplementationR
X P
hase
C
ompe
nsat
ion
FIF
O
TX Phase Compensation
FIFO
Byt
e O
rder
ing
Byt
e D
eser
ializ
er
Byte Serializer
8B/1
0 D
ecod
er
8B10B Encoder
Rat
e M
atch
FIF
OReceiver Channel PCS Receiver Ch0 PMA
Receiver Ch1 PMAReceiver Ch2 PMA
Receiver Ch3 PMA
Channel1Channel0
Channel1Channel0
Receiver Channel PCS
Des
kew
FIF
O
Wor
d A
ligne
r
Des
eria
lizer
CD
R
Transmitter Channel PCS
Transceiver Block (Four 100G Ethernet Channels)
Transmitter Channel PCSTransmitter Channel PCS
Transceiver Ch0 PMATransceiver Ch1 PMA
Transceiver Ch2 PMATransceiver Ch3 PMA
Ser
ializ
er
tx_d
atao
ut[3
:0] @
10.
3125
Gbp
s×
1 H
igh_
Spe
ed S
eria
l and
Low
-Spe
ed P
aral
lel C
lock
rx_d
atai
n[3:
0] @
10.
3125
Gbp
s
wrclk rdclk
/2
/2
FPGAFabric
5156.25 MHz
515.625 MHz
515.625 MHz
CMU0_PLL
257.8125 MHztx_clkout[3:0]
257.8125 MHzrx_clkout[3:0]
CMU0_Channel
CMU0 Clock Divider
Channel2 Receiver Channel PCSReceiver Channel PCSChannel3
Channel2Transmitter Channel PCSChannel3
40 4020
20
20
20
20
40
RX
Pha
se
Com
pens
atio
n F
IFO
TX Phase Compensation
FIFO
Byt
e O
rder
ing
Byt
e D
eser
ializ
er
Byte Serializer
8B/1
0 D
ecod
er
8B10B Encoder
Rat
e M
atch
FIF
O
Receiver Channel PCS Receiver Ch0 PMAReceiver Ch1 PMA
Channel1Channel0
Channel1Channel0
Receiver Channel PCS
Des
kew
FIF
O
Wor
d A
ligne
r
Des
eria
lizer
CD
R
Transmitter Channel PCS
Transceiver Block (Two 100G Ethernet Channels)
Transmitter Channel PCSTransceiver Ch0 PMA
Transceiver Ch1 PMA
Ser
ializ
er
tx_d
atao
ut[3
:0] @
10.
3125
Gbp
s
×1
Hig
h_S
peed
Ser
ial a
nd L
ow-S
peed
Par
alle
l Clo
ck
rx_d
atai
n[3:
0] @
10.
3125
Gbp
s
wrclk rdclk
/2
/2
Input Reference Clock at644.53 MHz/322.26 MHz
5156.25 MHz
515.625 MHz
515.625 MHz
CMU0_PLL
257.8125 MHztx_clkout[3:0]
257.8125 MHzrx_clkout[3:0]
CMU0_Channel
CMU0 Clock Divider
40 4020
20
20
20
20
40
RX
Pha
se
Com
pens
atio
n F
IFO
TX Phase Compensation
FIFO
Byt
e O
rder
ing
Byt
e D
eser
ializ
er
Byte Serializer
8B/1
0 D
ecod
er
8B10B Encoder
Rat
e M
atch
FIF
O
Receiver Channel PCS Receiver Ch0 PMAReceiver Ch1 PMA
Receiver Ch2 PMAReceiver Ch3 PMA
Channel1Channel0
Channel1Channel0
Receiver Channel PCS
Des
kew
FIF
O
Wor
d A
ligne
r
Des
eria
lizer
CD
R
Transmitter Channel PCS
Transceiver Block (Four 100G Ethernet Channels)
Transmitter Channel PCSTransmitter Channel PCS
Transceiver Ch0 PMATransceiver Ch1 PMA
Transceiver Ch2 PMATransceiver Ch3 PMA
Ser
ializ
er
tx_d
atao
ut[3
:0] @
10.
3125
Gbp
s×
1 H
igh_
Spe
ed S
eria
l and
Low
-Spe
ed P
aral
lel C
lock
rx_d
atai
n[3:
0] @
10.
3125
Gbp
s
wrclk rdclk
/2
/2
5156.25 MHz
515.625 MHz
515.625 MHz
CMU0_PLL
257.8125 MHztx_clkout[3:0]
257.8125 MHzrx_clkout[3:0]
CMU0_Channel
CMU0 Clock Divider
Channel2 Receiver Channel PCSReceiver Channel PCSChannel3
Channel2Transmitter Channel PCSChannel3
40 4020
20
20
20
20
40
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
GXBL3
GXBL2
GXBL1
GXBR3
GXBR2
GXBR1
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel 10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
EP4S100G2F40EP4S100G5H40
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
EP4S100G3F45EP4S100G4F45EP4S100G5F45
GXBL3
GXBL2
GXBL1
GXBR3
GXBR2
GXBR1
CMU1 Channel
CMU0 Channel
6G ATX PLL
CMU1 Channel
CMU0 Channel
6G ATX PLL
GXBL0 GXBR0
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
EP4S100G3F45EP4S100G4F45EP4S100G5F45
GXBL3
GXBL2
GXBL1
GXBR3
GXBR2
GXBR1
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
CMU1 Channel
CMU0 Channel
CMU1 Channel
CMU0 Channel
10G ATX PLL
6G ATX PLL
GXBL3
GXBL2
GXBL1
GXBR3
GXBR2
GXBR1
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel 10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
EP4S100G2F40EP4S100G5H40
3R3RBXG
10G XCVR Channel 10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel 10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
10G XCVR Channel
Input Reference Clock at644.53 MHz/322.26 MHz
Input Reference Clock at644.53 MHz/322.26 MHz
Gearbox Requirements in 40G/100G Ethernet Implementation Page 13
September 2011 Altera CorporationAN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
Transmitter Channel Datapath ClockingThe 10 channels that form the 100G Ethernet link are located across three transceiver blocks. The CMU0 PLL in each transceiver block is used to generate the high-speed serial and low-speed parallel transmitter clocks for channels located in that transceiver block.
Table 2 lists the allowed input reference clock frequencies and recommended input reference clock sources for optimum jitter performance.
The high-speed serial clock at 5156.25 MHz and the low-speed parallel clocks at 515.625 MHz generated by each CMU0 PLL are distributed to the transmitter channels located in that transceiver block through the ×1 clock line. The PMA and PCS blocks within all four channels use the high-speed serial and low-speed parallel clock for their operation.
f For more details about the ×1 clock network, refer to the Stratix IV Transceiver Clocking chapter in volume 2 of the Stratix IV Handbook.
The low-speed parallel clock of 515.625 MHz to each transmitter channel gets further divided-by-two at the byte serializer. This divided-by-two clock at 257.8125 MHz is forwarded as the FPGA fabric-transceiver interface clock on each channel’s tx_clkout port. You can use the tx_clkout port from one of the four channels to clock the 66-bit to 40-bit gearbox logic, as well as the PCS layer in the FPGA. Refer to “Gearbox Logic” on page 4 for more details.
Receiver Channel Datapath ClockingAs shown in Figure 10, the CDR in each of the 10 receiver channels forwards the high-speed serial (5156.25 MHz) and low-speed parallel (515.625 MHz) recovered clocks to the receiver PMA and PCS blocks for their operation. The low-speed parallel recovered clock of 515.625 MHz gets further divided-by-two at the byte deserializer. This divided-by-two clock at 257.8125 MHz is forwarded as the FPGA fabric-transceiver interface clock on each channel’s rx_clkout port. You can use the rx_clkout port from one of the 10 channels to clock the 40-bit to 66-bit gearbox logic, as well as the PCS layer in the FPGA. Refer to “Gearbox Logic” on page 4 for more details.
Gearbox Requirements in 40G/100G Ethernet ImplementationThe 40G/100G Ethernet link uses 64-bit/66-bit encoding to achieve DC balance on the line. The Stratix IV GT FPGA fabric-transceiver interface is 40 bits wide, as shown in Figure 9 and Figure 10. Gearbox logic is required to compensate for the data-width mismatch between the PCS (66-bit wide) and the Stratix IV GT transceiver (40-bit side).
Table 2. Input Reference Clock Recommendation for 100G Ethernet Link
Input Reference Clock Frequency Recommended Input Reference Clock Source for Optimum Output Jitter
644.53125 MHz Dedicated refclk0 or refclk1 pin of each of the three transceiver blocks containing the 10 channels that form the 100G Ethernet link322.265625 MHz
Page 14 Gearbox Requirements in 40G/100G Ethernet Implementation
September 2011 Altera Corporation AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
Gearbox Requirements in the Egress DirectionIn the egress direction, the MAC forwards a continuous stream of 64-bit data blocks to the PCS in the FPGA fabric. The PCS encodes the 64-bit data blocks from the MAC into 66-bit data blocks with a 64-bit/66-bit encoder, scrambles the 66-bit data blocks, and forwards a continuous stream of scrambled 66-bit data blocks to the striping sublayer within the PCS.
In a 40G Ethernet link, the striping sublayer stripes the continuous stream of 66-bit data from the PCS across four VLs. The four 66-bit wide VLs from the striping sublayer within the PCS must be mapped to four 40-bit wide physical lanes into the transmitter channels. Gearbox logic must be implemented in the FPGA fabric to do this mapping.
Figure 11 shows the recommended gearbox scheme in the egress direction for 40G Ethernet implementation.
In a 100G Ethernet link, the striping sublayer stripes the continuous stream of 66-bit data from the PCS across 20 VLs. The twenty 66-bit wide VLs from the striping sublayer must be mapped to ten 40-bit wide physical lanes into the transmitter channels. Gearbox logic must be implemented in the FPGA fabric to do this mapping.
Figure 11. Recommended Gearbox Scheme in Egress Direction for 40G Ethernet Implementation
PCS
Transceiver FPGA Fabric
Data Valid
257.8125 MHz(FPGA Fabric-Transceiver Interface Clock)
156.25 MHz(System Clock)
FIFOGearbox6666
66
66
66
4040
Buffer
Data ValidFIFOGearbox
664040
Buffer
Data ValidFIFOGearbox
664040
Buffer
Data ValidFIFOGearbox
664040
Buffer
TXCh 0
TXCh 1
TXCh 2
TXCh 3
VL3
VL2
VL1
VL0
tx_clkout[0]
Gearbox Requirements in 40G/100G Ethernet Implementation Page 15
September 2011 Altera CorporationAN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
Figure 12 shows the recommended gearbox scheme in the egress direction for 100G Ethernet implementation.
As shown in Figure 11 and Figure 12, each 66-bit wide VL from the striping sublayer within the PCS is clocked by the 156.25 MHz system clock. The recommended gearbox scheme implements a FIFO for each VL that decouples the system clock at 156.25 MHz and the FPGA fabric-transceiver interface clock at 257.8125 MHz. The 66-bit wide data from the PCS is written into the FIFO at 156.25 MHz and read out of the FIFO at 257.8125 MHz. Because the read clock to the FIFO is faster than the write clock, the FIFO empties if data is continuously read from it. To avoid an empty FIFO, the FIFO logic must periodically stall data being read from the FIFO. The FIFO logic must also send a data_valid signal to the gearbox logic to indicate a valid or invalid data cycle.
1 There is no impact on the link throughput due to stalling of FIFO reads. The stalling is only required to maintain a 10.3125 Gbps throughput on each lane of the 40G/100G Ethernet link while compensating for the frequency difference between the system clock at 156.25 MHz and FPGA fabric-transceiver interface clock at 257.8125 MHz.
In a 40G Ethernet link, the gearbox logic converts each of the four 66-bit lanes from the FIFO into four 40-bit lanes to the Stratix IV GT transmitter channels. You can use the tx_clkout (FPGA fabric-transceiver interface) signal from any one of the four channels to clock the gearbox logic and the write side of the FIFO, as shown in Figure 11.
Figure 12. Recommended Gearbox Scheme in Egress Direction for 100G Ethernet Implementation
PCS
FPGA Fabric
Data Valid
257.8125 MHz(FPGA Fabric-Transceiver Interface Clock)
tx_clkout[0] 156.25 MHz(System Clock)
FIFOGearbox666620
20
20
20
Data ValidFIFOGearbox
MUX
MUX
6666
Data ValidFIFOGearbox
6666
Data ValidFIFOGearbox
6666
VL19
VL18
VL1
VL0
Transceiver
Buffer
40
40
Buffer
TXCh 0
TXCh 9
Page 16 Gearbox Requirements in 40G/100G Ethernet Implementation
September 2011 Altera Corporation AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
In a 100G Ethernet link, the gearbox logic converts each of the twenty 66-bit lanes from the FIFO into twenty 20-bit lanes to the Stratix IV GT transmitter channels. The multiplexer block converts these twenty 20-bit lanes from the gearbox logic into ten 40-bit lanes to the Stratix IV GT transmitter channels. You can use the tx_clkout (FPGA fabric-transceiver interface) signal from any of the 10 channels to clock the multiplexer block, the gearbox logic, and the write side of the FIFO, as shown in Figure 12.
1 The egress-side gearbox schemes shown in Figure 11 and Figure 12 assume that the 156.25 MHz system clock and the 257.8125 MHz FPGA fabric-transceiver interface clock are both derived from the same base frequency. This can be achieved by deriving the 156.25 MHz system clock from the input reference clock with either an on-chip general purpose PLL or an off-chip clock synthesizer.
Gearbox Requirements in the Ingress DirectionIn the ingress direction of a 40G Ethernet link, each of the four Stratix IV GT receiver channels forwards a 40-bit wide data bus to the Ethernet soft IP in the FPGA fabric. Data from each channel within the link goes to the PCS for destriping. Because each VL in the PCS is 66 bits wide, gearbox logic is required to convert the 40-bit data from each of the four channels to 66-bit data in the PCS.
Figure 13 shows the recommended gearbox scheme in the ingress direction for 40G Ethernet implementation.
In the ingress direction of a 100G Ethernet link, each of the 10 Stratix IV GT receiver channels forwards a 40-bit wide data bus to the Ethernet soft IP in the FPGA fabric. Data from each channel within the link goes to the PCS for destriping. Because each of the 20 VLs in the PCS is 66 bits wide, gearbox logic is required to convert the 40-bit data from each of the 10 channels to 66-bit data in the PCS.
Figure 13. Recommended Gearbox Scheme in Ingress Direction for 40G Ethernet Implementation
PCS
XCVR FPGA Fabric
Data Valid
257.8125 MHz(FPGA Fabric-Transceiver Interface Clock)
rx_clkout[0]
rx_coreclk[1]
rx_coreclk[2]
rx_coreclk[3]
156.25 MHz(System Clock)
FIFOGearbox6666
Data Valid
66
Data Valid
66
Data Valid
66
66
66
66
40
FIFOGearbox40
FIFOGearbox40
FIFOGearbox40
RXCh 0
RXCh 1
RXCh 2
RXCh 3
VL3
VL2
VL1
VL0
Gearbox Requirements in 40G/100G Ethernet Implementation Page 17
September 2011 Altera CorporationAN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
Figure 14 shows the recommended gearbox scheme in the ingress direction for 100G Ethernet implementation.
As shown in Figure 13 and Figure 14, each receiver channel in a 40G or 100G Ethernet link forwards a 40-bit wide data to the FPGA fabric.
Altera recommends enabling the rx_coreclk port (read clock for the receiver phase compensation FIFO) in the ALTGX MegaWizard Plug-In Manager instance used to instantiate the 40G/100G Ethernet link. Drive each channel’s rx_coreclk port with the recovered clock rx_clkout from any one of the 4/10 receiver channels in the 40G/100G Ethernet link. Use this rx_clkout signal to clock the gearbox logic for all receiver channels, as shown in Figure 13 and Figure 14.
f For more information about using rx_coreclk ports, refer to “User-Selected Receiver Phase Compensation FIFO Read Clock” in the Stratix IV Transceiver Clocking chapter in volume 2 of the Stratix IV Handbook.
Because the gearbox logic takes in 40-bit data and reads out 66-bit data at 257.8125 MHz, it could become empty. To avoid this, the gearbox logic must periodically stall reading out data. To indicate this periodic stalling of data to the FIFO, the gearbox logic sends a data_valid signal to the downstream FIFO. The FIFO must register the 66-bit data only when the data_valid signal from the gearbox logic is asserted. The FIFO reads out the 66-bit data to the PCS with the 156.25 MHz system clock, as shown in Figure 13 and Figure 14.
Figure 14. Recommended Gearbox Scheme in Ingress Direction for 100G Ethernet Implementation
XCVR FPGA Fabric
Data Valid
257.8125 MHz(FPGA Fabric-Transceiver Interface Clock)
rx_clkout[0]
rx_coreclk[9]
156.25 MHz(System Clock)
FIFOGearbox6666
Data Valid
66
Data Valid
66
Data Valid
66
66
66
66
20
40
40
FIFOGearbox20
FIFOGearbox20
FIFOGearbox20
RXCh 0
RXCh 9
Demux
Demux
PCS
VL19
VL18
VL1
VL0
Page 18 Summary
September 2011 Altera Corporation AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices
1 The recovered clock (rx_clkout[0]) at 257.8125 MHz and the system clock at 156.25 MHz can have a maximum of ± 100 PPM difference. You must design the FIFOs shown in Figure 13 and Figure 14 to be able to compensate for this PPM difference between the two clock domains.
SummaryStratix IV GT devices offer a single-chip solution for implementing 40G/100G Ethernet MAC devices in a 40G/100G line card. This application note provides guidance on the implementation of 40G/100G Ethernet links in a Stratix IV GT device.
Document Revision HistoryTable 3 shows the revision history for this document.
Table 3. Document Revision History
Date Version Changes
September 2011 1.2■ Updated reference to Altera’s IP partners.
■ Updated document layout.
December 2009 1.1 Updated figures to remove 8G channels.
June 2009 1.0 Initial release.