AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix … · AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices September 2011 Altera Corporation Figure

September 2011 Altera Corporation

AN-570-1.2 Application Note

© 2011 Altera Corporation. All rights reserved. ALTERA, ARRIA, CYCLONE, HARDCOPY, MAX, MEGACORE, NIOS, QUARTUS and STRATIX words and logos are trademarks of Altera Corporation and registered in the U.S. Patent and Trademark Office and in other countries. All other words and logos identified as trademarks or service marks are the property of their respective holders as described at www.altera.com/common/legal.html. Altera warrants performance of its semiconductor products to current specifications in accordance with Altera's standard warranty, but reserves the right to make changes to any products and services at any time without notice. Altera assumes no responsibility or liability arising out of the application or use of any information, product, or service described herein except as expressly agreed to in writing by Altera. Altera customers are advised to obtain the latest version of device specifications before relying on any published information and before placing orders for products or services.

101 Innovation DriveSan Jose, CA 95134www.altera.com

Subscribe

ISO 9001:2008 Registered

AN 570: Implementing the 40G/100GEthernet Protocol in Stratix IV Devices

IntroductionAltera’s 40 nm Stratix® IV GX and Stratix IV GT devices provide a complete solution for developing 40 Gbps and 100 Gbps network line card applications.

This application note describes how to implement a 40 Gbps/100 Gbps (40G/100G) Ethernet line-side interface using Stratix IV GT devices. It focuses on the following two implementation aspects:

■ Transceiver configuration and clocking recommended for 40G/100G Ethernet

■ Interfacing 40G/100G Ethernet Soft IP to the transceivers

Figure 1 shows an example 40G/100G network line card application with Stratix IV GT and Stratix IV GX devices.

The three-chip 40G/100G line card solution shown in Figure 1 consists of:

Figure 1. Altera’s 40G/100G Network Line Card Application (1)

Note to Figure 1:(1) Figure 1 shows the 40G/100G MAC, PCS, and PMA designed in a Stratix IV GT device with integrated 10 Gbps serial transceivers. You can

implement this application in Altera’s Stratix IV GX and HardCopy® IV GX devices with external RXAUI to XFI multiplexer chips between the Altera device and the optical module. For more information, refer to Using 10-Gbps Transceivers in 40G/100G Applications.

Opticalmodule

Look-uptables

Packetbuffer

TM linklist

Stratix IV GX Stratix IV GX

40G/100G chip-to-backplaneInterlaken

SPAUI and DDR-XAUI

40G/100G chip-to-chipSystem Side Interface

Interlaken or SerialLite II

40G/100G chip-to-moduleLine Side Interface

XLAUI (4 lanes @ 10G)CAUI (10 lanes @ 10G)

Switch fabric/backplane

RLDRAM IIDDR3QDR II/III

40G/100GMAC

Trafficmanagement

Packetprocessing

Stratix IV GT

http://www.altera.com/common/legal.html

http://www.altera.com

http://www.altera.com/literature/wp/wp-01080-stratix-iv-gt-40g-100g.pdf

http://www.altera.com/support/devices/reliability/certifications/rel-certifications.html

http://www.altera.com/support/devices/reliability/certifications/rel-certifications.html

https://www.altera.com/servlets/subscriptions/alert?id=AN-570

Page 2 Stratix IV GT Hard IP Blocks Used in 40G/100G Ethernet Implementation

September 2011 Altera Corporation AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices

■ 40G/100G Ethernet MAC device

■ Targeted to Stratix IV GT EP4S40G2 or EP4S40G5 devices for 40G Ethernet and to EP4S100G2, EP4S100G3, EP4S100G4, or EP4S100G5 devices for 100G Ethernet

■ Line side interface to optical module comprising 4 or 10 channels, each configured at a serial data rate of 10.3125 Gbps for 40G or 100G Ethernet, respectively

The 40G Ethernet line side interface targets compliance to XLAUI electrical specifications

The 100G Ethernet line side interface targets compliance to CAUI electrical specifications

■ System side interface to the packet processing Stratix IV GX FPGA consisting of a multi-channel Interlaken or proprietary SerialLite II link with each channel configured at data rates up to 6.375 Gbps

■ Packet Processing and Traffic Management devices

■ Targeted to Stratix IV GX devices with transceivers configured to implement Interlaken or SerialLite II links

Stratix IV GT Hard IP Blocks Used in 40G/100G Ethernet ImplementationThe 40G Ethernet MAC implementation uses four full-duplex Stratix IV GT transceiver channels at 10.3125 Gbps, as shown in Figure 2.

Figure 2. 40G MAC Implementation in a Stratix IV GT Device

40G MAC

Interlakenor

SerialLite IISoft IP

40G PCS

VL2

VL3

Ch0 Ch0

Ch2 Ch2

Ch1 Ch1

Ch3 Ch3 10.3125G

10.3125G

10.3125G

10.3125G

VL0

VL1

40

40

40

40

CustomLogic

XLAUI

40

40

40

40

Gearbox

Gearbox

Gearbox

Gearbox

66

66

Line Side InterfaceSystem Side Interface

66

66

66

66

Soft IP Hard IP

66

66

FourTransceiverChannels @10.3125G

Interlakenor

SerialLite IITransceivers

40G PMD(Optical

modules)

Stratix IV GT Soft IP Blocks Used in 40G/100G Ethernet Implementation Page 3

September 2011 Altera CorporationAN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices

The 100G Ethernet MAC implementation uses 10 full-duplex Stratix IV GT transceiver channels at 10.3125 Gbps, as shown in Figure 3.

Stratix IV GT Soft IP Blocks Used in 40G/100G Ethernet ImplementationStratix IV GT devices do not have dedicated hard IP blocks that implement 40G/100G Ethernet Media Access Control (MAC), Physical Coding Sublayer (PCS), and gearbox logic. These blocks must be implemented with a soft IP core in the FPGA fabric.

1 40G/100G Ethernet soft IP cores targeted to Stratix IV GT devices are available from Altera’s IP partners MorethanIP (www.morethanip.com).

Physical Coding Sublayer (PCS) Block In the egress direction, the PCS block is responsible for functions such as 64B/66B encoding, scrambling using an X58 + X39 + 1 polynomial, and striping the 66-bit data blocks into virtual lanes (VLs). In the ingress direction, it is responsible for functions such as 66B/64B decoding, descrambling using an X58 + X39 + 1 polynomial, and destriping the VLs into 66-bit data blocks. It also performs status monitoring and decoding functions as specified in the IEEE802.3ba specification.

Media Access Control (MAC) LayerThe MAC layer is responsible for functions such as Ethernet frame encapsulation and decapsulation, Frame Check Sequence (FCS) generation and checking, flow control, and reconciliation sublayer functions as required by the IEEE802.3ba specification.

Figure 3. 100G MAC Implementation in a Stratix IV GT Device

100G MAC 100G PCS

VL18

VL19

Ch0Ch0

Ch9 Ch9 10.3125G

10.3125G

VL0

VL1

20

20

20

20

CAUI

20

20

20

20

40

40

66

6666

66

Mux/Demux

Mux/Demux

66

66

Soft IP Hard IP

66

66

Ten TransceiverChannels @10.3125G

Ch9

Ch0

100G PMD(Optical

modules)

40

40

Interlakenor

SerialLite IISoft IP

CustomLogic

System Side Interface Line Side Interface

Interlakenor

SerialLite IITransceivers

Gearbox

Gearbox

Gearbox

Gearbox

http://www.morethanip.com/

Page 4 Stratix IV GT Transceiver Configuration for 40G/100G Ethernet Implementation


Gearbox LogicThe 40G/100G Ethernet data is 64B/66B encoded in the egress direction and 66B/64B decoded in the ingress direction. Each VL from or into the PCS block is 66 bits wide. When configured at 10.3125 Gbps for 40G/100G Ethernet, the Stratix IV GT transceiver channels support a parallel data width of 40 bits.

In the egress direction, the gearbox logic is responsible for translating each 66-bit wide VL from the PCS into 40-bit wide data to the Stratix IV GT transmitter channel. In the ingress direction, the gearbox logic is responsible for translating the 40-bit wide data from each Stratix IV GT receiver channel into 66-bit wide data to the PCS.

Stratix IV GT Transceiver Configuration for 40G/100G Ethernet Implementation

The following sections discuss the required Stratix IV GT transceiver configuration and the recommended clocking for 40G/100G Ethernet implementation:

■ “Required Transceiver Channel Placement”

■ “Transceiver Channel Datapath Configuration” on page 8

■ “40G Ethernet Transceiver Channel Datapath Clocking” on page 10

■ “100G Ethernet Transceiver Channel Datapath Clocking” on page 12

Required Transceiver Channel Placement Stratix IV GT EP4S40G2 and EP4S40G5 devices support up to six channels on each side of the device. Each channel can operate at 10.3125 Gbps. Altera recommends using these devices to implement a 40G Ethernet link.

Stratix IV GT Transceiver Configuration for 40G/100G Ethernet Implementation Page 5


Figure 4 shows the transceiver channel locations in EP4S40G2F40 and EP4S40G5H40 devices that support 10.3125 Gbps data rate to implement a 40G Ethernet link.

Stratix IV GT EP4S100G2 and EP4S100G5 devices support up to 12 channels on each side of the device. Each channel can operate at 10.3125 Gbps. Altera recommends using these devices to implement a 100G Ethernet link.

Figure 4. Transceiver Channel Locations for 40G Ethernet Implementation

8G Transceiver Channel

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0/1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

10G ATX PLL

6G ATX PLL 6G ATX PLL

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL

EP4S40G2F40EP4S40G5H40

GXBR1GXBL1

GXBL0 GXBR0

GXBL2 GXBR2

Four transceiverchannels at

10.3125 Gbps oneach side for 40G Ethernet

implementation

2.488-10.3125 GbpsTransceiver Channels


2.488-6.5 GbpsCMU Channels (PMA-Only)

9.9 Gbps-11.3 GbpsATX PLL


10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

8G XCVR Channel

8G XCVR Channel

8G XCVR Channel

8G XCVR Channel

8G XCVR Channel

8G XCVR Channel 8G XCVR Channel

8G XCVR Channel

8G XCVR Channel

8G XCVR Channel

8G XCVR Channel

8G XCVR Channel

8G XCVR Channel

10G XCVR Channel



Figure 5 shows the transceiver channel locations in EP4S100G2F40 and EP4S100G5H40 devices that support 10.3125 Gbps data rate to implement a 100G Ethernet link. Figure 6 shows the transceiver channel locations in the EP4S100G3F45/EP4S100G4F45/EP4S100G5F45 devices that support 10.3125 Gbps data rate to implement a 100G Ethernet link.

Figure 5. Transceiver Channel Locations for 100G Ethernet Implementation in EP4S100G2F40 and EP4S100G5H40 Devices

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL

CMU0/1 Channel

10G ATX PLL

6G ATX PLL


2.488 Gbps-6.5 GbpsCMU Channels (PMA-Only)



GXBL3

GXBL2

GXBL1

GXBR3

GXBR2

GXBR1

Ten transceiverchannels at

10.3125 Gbps oneach side for 100G Ethernetimplementation

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel


10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel




Figure 6. Transceiver Channel Locations for 100G Ethernet Implementation in an EP4S100G5F45 Device

Note to Figure 6:(1) EP4S100G3F45 and EP4S100G4F45 devices only support up to 12 10G channels on each side of the device.

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL

EP4S100G3F45EP4S100G4F45EP4S100G5F45

CMU0/1 Channel

10G ATX PLL

6G ATX PLL


2.488 Gbps-6.5 GbpsCMU Channels

(PMA-Only)



GXBL3

GXBL2

GXBL1

GXBR3

GXBR2

GXBR1

Ten transceiverchannels at

10.3125 Gbps oneach side for 100G Ethernetimplementation

CMU1 Channel

CMU0 Channel

6G ATX PLL

CMU1 Channel

CMU0 Channel

6G ATX PLL

GXBL0 GXBR0


10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel


10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

(1)

(1)



Transceiver Channel Datapath Configuration In a 40G or 100G Ethernet link, each of the 4 or 10 transceiver channels operates at 10.3125 Gbps. The Stratix IV GT transceiver channel must be configured in Basic Double-Width mode with a 40-bit FPGA fabric-transceiver interface with a low latency PCS datapath, as shown in Figure 7.

f For more information about the low latency PCS datapath, refer to “Low Latency PCS Datapath” in the Stratix IV Transceiver Architecture chapter in volume 2 of the Stratix IV Handbook.

Figure 7. Low Latency PCS Transceiver Configuration Required for Implementing 40G/100G Ethernet LinkR

X P

hase

C

ompe

nsat

ion

FIF

O

Byt

e O

rder

ing

Byt

e D

eser

ializ

er

8B10

B D

ecod

er

Rat

e M

atch

FIF

O

Receiver Channel PCS Receiver ChannelPMA

Des

kew

FIF

O

Wor

d A

ligne

r

rx_datain@

10.3125 Gbps

Des

eria

lizer

CD

R

TX Phase Compensation

FIFO

ByteSerializer

8B10B Encoder

Transmitter Channel PCS

Low Latency PCS Datapath

Transceiver Channel PMA

tx_dataout@

10.3125 GbpsSerializer

FPGAFabric

40 20 20

20

20

40

40

http://www.altera.com/literature/hb/stratix-iv/stx4_siv52001.pdf



Figure 8 shows the ALTGX MegaWizard™ Plug-In Manager setup for 40G/100G Ethernet implementation.

Figure 8. ALTGX Instance for 40G/100G Ethernet Implementation

40G Ethernet = 4 channels100G Ethernet = 10 channels



40G Ethernet Transceiver Channel Datapath ClockingFigure 9 shows the recommended transceiver channel datapath clocking for implementing a 40G Ethernet link in Stratix IV GT devices.

Figure 9. Transceiver Channel Datapath Clocking in 40G Ethernet Implementation

RX

Pha

se

Com

pens

atio

n F

IFO


FIFO

Byt

e O

rder

ing

Byt

e D

eser

ializ

er

Byte Serializer

8B/1

0 D

ecod

er

8B10B Encoder

Rat

e M

atch

FIF

O

Receiver Channel PCS Receiver Ch0 PMAReceiver Ch1 PMA

Receiver Ch2 PMAReceiver Ch3 PMA

Channel1Channel0

Channel1Channel0

Receiver Channel PCS

Des

kew

FIF

O

Wor

d A

ligne

r

Des

eria

lizer

CD

R


Transceiver Block

Transmitter Channel PCSTransmitter Channel PCS

Transceiver Ch0 PMATransceiver Ch1 PMA


Ser

ializ

er

tx_d

atao

ut[3

:0] @

10.

3125

Gbp

s×

1 H

igh_

Spe

ed S

eria

l and

Low

-Spe

ed P

aral

lel C

lock

rx_d

atai

n[3:

0] @

10.

3125

Gbp

s

wrclk rdclk

/2

/2

FPGAFabric

Input Reference Clock at644.53 MHz/322.26 MHz

5156.25 MHz

515.625 MHz

515.625 MHz

CMU0_PLL

257.8125 MHztx_clkout[3:0]

257.8125 MHzrx_clkout[3:0]

CMU0_Channel

CMU0 Clock Divider

Channel2 Receiver Channel PCSReceiver Channel PCSChannel3

Channel2Transmitter Channel PCSChannel3

40 4020

20

20

20

20

40

8G Transceiver Channel

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL


GXBR1GXBL1

GXBL0 GXBR0

GXBL2 GXBR2

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

8G XCVR Channel

8G XCVR Channel

8G XCVR Channel

8G XCVR Channel

8G XCVR Channel


8G XCVR Channel

8G XCVR Channel

8G XCVR Channel

8G XCVR Channel

8G XCVR Channel



Transmitter Channel Datapath ClockingThe CMU0 PLL located in the transceiver block containing four channels is used to generate the high-speed serial and low-speed parallel transmitter clocks. Table 1 lists the allowed input reference clock frequencies and recommended input reference clock sources for optimum jitter performance.

The high-speed serial clock at 5156.25 MHz and the low-speed parallel clocks at 515.625 MHz generated by the CMU0 PLL are distributed to the four transmitter channels through the ×1 clock line. The PMA and PCS blocks within all four channels use the high-speed serial and low-speed parallel clock for their operation.

f For more details about the ×1 clock network, refer to the Stratix IV Transceiver Clocking chapter in volume 2 of the Stratix IV Handbook.

The low-speed parallel clock of 515.625 MHz to each transmitter channel gets further divided-by-two at the byte serializer. This divided-by-two clock at 257.8125 MHz is forwarded as the FPGA fabric-transceiver interface clock on each channel’s tx_clkout port. You can use the tx_clkout port from one of the four channels to clock the 66-bit to 40-bit gearbox logic, as well as the PCS layer in the FPGA. Refer to “Gearbox Logic” on page 4 for more details.

Receiver Channel Datapath ClockingAs shown in Figure 9, the clock and data recovery (CDR) in each of the four receiver channels forwards the high-speed serial (5156.25 MHz) and low-speed parallel (515.625 MHz) recovered clocks to the receiver PMA and PCS blocks for their operation. The low-speed parallel recovered clock of 515.625 MHz gets further divided-by-two at the byte deserializer. This divided-by-two clock at 257.8125 MHz is forwarded as the FPGA fabric-transceiver interface clock on each channel’s rx_clkout port. You can use the rx_clkout port from one of the four channels to clock the 40-bit to 66-bit gearbox logic, as well as the PCS layer in the FPGA. Refer to “Gearbox Logic” on page 4 for more details.

Table 1. Input Reference Clock Recommendation for 40G Ethernet Link

Input Reference Clock Frequency Recommended Input Reference Clock Source for Optimum Output Jitter

644.53125 MHz Dedicated refclk0 or refclk1 pin of the transceiver block containing the four 10.3125G channels322.265625 MHz




100G Ethernet Transceiver Channel Datapath ClockingFigure 10 shows the recommended transceiver channel datapath clocking for implementing a 100G Ethernet link in Stratix IV GT devices.

Figure 10. Transceiver Channel Datapath Clocking in 100G Ethernet ImplementationR

X P

hase

C

ompe

nsat

ion

FIF

O


FIFO

Byt

e O

rder

ing

Byt

e D

eser

ializ

er

Byte Serializer

8B/1

0 D

ecod

er

8B10B Encoder

Rat

e M

atch

FIF

OReceiver Channel PCS Receiver Ch0 PMA


Receiver Ch3 PMA

Channel1Channel0

Channel1Channel0


Des

kew

FIF

O

Wor

d A

ligne

r

Des

eria

lizer

CD

R


Transceiver Block (Four 100G Ethernet Channels)




Ser

ializ

er

tx_d

atao

ut[3

:0] @

10.

3125

Gbp

s×

1 H

igh_

Spe

ed S

eria

l and

Low

-Spe

ed P

aral

lel C

lock

rx_d

atai

n[3:

0] @

10.

3125

Gbp

s

wrclk rdclk

/2

/2

FPGAFabric

5156.25 MHz

515.625 MHz

515.625 MHz

CMU0_PLL



CMU0_Channel

CMU0 Clock Divider



40 4020

20

20

20

20

40

RX

Pha

se

Com

pens

atio

n F

IFO


FIFO

Byt

e O

rder

ing

Byt

e D

eser

ializ

er

Byte Serializer

8B/1

0 D

ecod

er

8B10B Encoder

Rat

e M

atch

FIF

O


Channel1Channel0

Channel1Channel0


Des

kew

FIF

O

Wor

d A

ligne

r

Des

eria

lizer

CD

R


Transceiver Block (Two 100G Ethernet Channels)

Transmitter Channel PCSTransceiver Ch0 PMA

Transceiver Ch1 PMA

Ser

ializ

er

tx_d

atao

ut[3

:0] @

10.

3125

Gbp

s

×1

Hig

h_S

peed

Ser

ial a

nd L

ow-S

peed

Par

alle

l Clo

ck

rx_d

atai

n[3:

0] @

10.

3125

Gbp

s

wrclk rdclk

/2

/2


5156.25 MHz

515.625 MHz

515.625 MHz

CMU0_PLL



CMU0_Channel

CMU0 Clock Divider

40 4020

20

20

20

20

40

RX

Pha

se

Com

pens

atio

n F

IFO


FIFO

Byt

e O

rder

ing

Byt

e D

eser

ializ

er

Byte Serializer

8B/1

0 D

ecod

er

8B10B Encoder

Rat

e M

atch

FIF

O



Channel1Channel0

Channel1Channel0


Des

kew

FIF

O

Wor

d A

ligne

r

Des

eria

lizer

CD

R


Transceiver Block (Four 100G Ethernet Channels)




Ser

ializ

er

tx_d

atao

ut[3

:0] @

10.

3125

Gbp

s×

1 H

igh_

Spe

ed S

eria

l and

Low

-Spe

ed P

aral

lel C

lock

rx_d

atai

n[3:

0] @

10.

3125

Gbp

s

wrclk rdclk

/2

/2

5156.25 MHz

515.625 MHz

515.625 MHz

CMU0_PLL



CMU0_Channel

CMU0 Clock Divider



40 4020

20

20

20

20

40

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL

GXBL3

GXBL2

GXBL1

GXBR3

GXBR2

GXBR1

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel


10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel


CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL


GXBL3

GXBL2

GXBL1

GXBR3

GXBR2

GXBR1

CMU1 Channel

CMU0 Channel

6G ATX PLL

CMU1 Channel

CMU0 Channel

6G ATX PLL

GXBL0 GXBR0

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL


GXBL3

GXBL2

GXBL1

GXBR3

GXBR2

GXBR1

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL

CMU1 Channel

CMU0 Channel

CMU1 Channel

CMU0 Channel

10G ATX PLL

6G ATX PLL

GXBL3

GXBL2

GXBL1

GXBR3

GXBR2

GXBR1

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel


10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel


3R3RBXG


10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel

10G XCVR Channel


10G XCVR Channel

10G XCVR Channel

10G XCVR Channel



Gearbox Requirements in 40G/100G Ethernet Implementation Page 13


Transmitter Channel Datapath ClockingThe 10 channels that form the 100G Ethernet link are located across three transceiver blocks. The CMU0 PLL in each transceiver block is used to generate the high-speed serial and low-speed parallel transmitter clocks for channels located in that transceiver block.

Table 2 lists the allowed input reference clock frequencies and recommended input reference clock sources for optimum jitter performance.

The high-speed serial clock at 5156.25 MHz and the low-speed parallel clocks at 515.625 MHz generated by each CMU0 PLL are distributed to the transmitter channels located in that transceiver block through the ×1 clock line. The PMA and PCS blocks within all four channels use the high-speed serial and low-speed parallel clock for their operation.

f For more details about the ×1 clock network, refer to the Stratix IV Transceiver Clocking chapter in volume 2 of the Stratix IV Handbook.

The low-speed parallel clock of 515.625 MHz to each transmitter channel gets further divided-by-two at the byte serializer. This divided-by-two clock at 257.8125 MHz is forwarded as the FPGA fabric-transceiver interface clock on each channel’s tx_clkout port. You can use the tx_clkout port from one of the four channels to clock the 66-bit to 40-bit gearbox logic, as well as the PCS layer in the FPGA. Refer to “Gearbox Logic” on page 4 for more details.

Receiver Channel Datapath ClockingAs shown in Figure 10, the CDR in each of the 10 receiver channels forwards the high-speed serial (5156.25 MHz) and low-speed parallel (515.625 MHz) recovered clocks to the receiver PMA and PCS blocks for their operation. The low-speed parallel recovered clock of 515.625 MHz gets further divided-by-two at the byte deserializer. This divided-by-two clock at 257.8125 MHz is forwarded as the FPGA fabric-transceiver interface clock on each channel’s rx_clkout port. You can use the rx_clkout port from one of the 10 channels to clock the 40-bit to 66-bit gearbox logic, as well as the PCS layer in the FPGA. Refer to “Gearbox Logic” on page 4 for more details.

Gearbox Requirements in 40G/100G Ethernet ImplementationThe 40G/100G Ethernet link uses 64-bit/66-bit encoding to achieve DC balance on the line. The Stratix IV GT FPGA fabric-transceiver interface is 40 bits wide, as shown in Figure 9 and Figure 10. Gearbox logic is required to compensate for the data-width mismatch between the PCS (66-bit wide) and the Stratix IV GT transceiver (40-bit side).

Table 2. Input Reference Clock Recommendation for 100G Ethernet Link

Input Reference Clock Frequency Recommended Input Reference Clock Source for Optimum Output Jitter

644.53125 MHz Dedicated refclk0 or refclk1 pin of each of the three transceiver blocks containing the 10 channels that form the 100G Ethernet link322.265625 MHz


Page 14 Gearbox Requirements in 40G/100G Ethernet Implementation


Gearbox Requirements in the Egress DirectionIn the egress direction, the MAC forwards a continuous stream of 64-bit data blocks to the PCS in the FPGA fabric. The PCS encodes the 64-bit data blocks from the MAC into 66-bit data blocks with a 64-bit/66-bit encoder, scrambles the 66-bit data blocks, and forwards a continuous stream of scrambled 66-bit data blocks to the striping sublayer within the PCS.

In a 40G Ethernet link, the striping sublayer stripes the continuous stream of 66-bit data from the PCS across four VLs. The four 66-bit wide VLs from the striping sublayer within the PCS must be mapped to four 40-bit wide physical lanes into the transmitter channels. Gearbox logic must be implemented in the FPGA fabric to do this mapping.

Figure 11 shows the recommended gearbox scheme in the egress direction for 40G Ethernet implementation.

In a 100G Ethernet link, the striping sublayer stripes the continuous stream of 66-bit data from the PCS across 20 VLs. The twenty 66-bit wide VLs from the striping sublayer must be mapped to ten 40-bit wide physical lanes into the transmitter channels. Gearbox logic must be implemented in the FPGA fabric to do this mapping.

Figure 11. Recommended Gearbox Scheme in Egress Direction for 40G Ethernet Implementation

PCS

Transceiver FPGA Fabric

Data Valid

257.8125 MHz(FPGA Fabric-Transceiver Interface Clock)

156.25 MHz(System Clock)

FIFOGearbox6666

66

66

66

4040

Buffer

Data ValidFIFOGearbox

664040

Buffer


664040

Buffer


664040

Buffer

TXCh 0

TXCh 1

TXCh 2

TXCh 3

VL3

VL2

VL1

VL0

tx_clkout[0]



Figure 12 shows the recommended gearbox scheme in the egress direction for 100G Ethernet implementation.

As shown in Figure 11 and Figure 12, each 66-bit wide VL from the striping sublayer within the PCS is clocked by the 156.25 MHz system clock. The recommended gearbox scheme implements a FIFO for each VL that decouples the system clock at 156.25 MHz and the FPGA fabric-transceiver interface clock at 257.8125 MHz. The 66-bit wide data from the PCS is written into the FIFO at 156.25 MHz and read out of the FIFO at 257.8125 MHz. Because the read clock to the FIFO is faster than the write clock, the FIFO empties if data is continuously read from it. To avoid an empty FIFO, the FIFO logic must periodically stall data being read from the FIFO. The FIFO logic must also send a data_valid signal to the gearbox logic to indicate a valid or invalid data cycle.

1 There is no impact on the link throughput due to stalling of FIFO reads. The stalling is only required to maintain a 10.3125 Gbps throughput on each lane of the 40G/100G Ethernet link while compensating for the frequency difference between the system clock at 156.25 MHz and FPGA fabric-transceiver interface clock at 257.8125 MHz.

In a 40G Ethernet link, the gearbox logic converts each of the four 66-bit lanes from the FIFO into four 40-bit lanes to the Stratix IV GT transmitter channels. You can use the tx_clkout (FPGA fabric-transceiver interface) signal from any one of the four channels to clock the gearbox logic and the write side of the FIFO, as shown in Figure 11.

Figure 12. Recommended Gearbox Scheme in Egress Direction for 100G Ethernet Implementation

PCS

FPGA Fabric

Data Valid


tx_clkout[0] 156.25 MHz(System Clock)

FIFOGearbox666620

20

20

20


MUX

MUX

6666


6666


6666

VL19

VL18

VL1

VL0

Transceiver

Buffer

40

40

Buffer

TXCh 0

TXCh 9

Page 16 Gearbox Requirements in 40G/100G Ethernet Implementation


In a 100G Ethernet link, the gearbox logic converts each of the twenty 66-bit lanes from the FIFO into twenty 20-bit lanes to the Stratix IV GT transmitter channels. The multiplexer block converts these twenty 20-bit lanes from the gearbox logic into ten 40-bit lanes to the Stratix IV GT transmitter channels. You can use the tx_clkout (FPGA fabric-transceiver interface) signal from any of the 10 channels to clock the multiplexer block, the gearbox logic, and the write side of the FIFO, as shown in Figure 12.

1 The egress-side gearbox schemes shown in Figure 11 and Figure 12 assume that the 156.25 MHz system clock and the 257.8125 MHz FPGA fabric-transceiver interface clock are both derived from the same base frequency. This can be achieved by deriving the 156.25 MHz system clock from the input reference clock with either an on-chip general purpose PLL or an off-chip clock synthesizer.

Gearbox Requirements in the Ingress DirectionIn the ingress direction of a 40G Ethernet link, each of the four Stratix IV GT receiver channels forwards a 40-bit wide data bus to the Ethernet soft IP in the FPGA fabric. Data from each channel within the link goes to the PCS for destriping. Because each VL in the PCS is 66 bits wide, gearbox logic is required to convert the 40-bit data from each of the four channels to 66-bit data in the PCS.

Figure 13 shows the recommended gearbox scheme in the ingress direction for 40G Ethernet implementation.

In the ingress direction of a 100G Ethernet link, each of the 10 Stratix IV GT receiver channels forwards a 40-bit wide data bus to the Ethernet soft IP in the FPGA fabric. Data from each channel within the link goes to the PCS for destriping. Because each of the 20 VLs in the PCS is 66 bits wide, gearbox logic is required to convert the 40-bit data from each of the 10 channels to 66-bit data in the PCS.

Figure 13. Recommended Gearbox Scheme in Ingress Direction for 40G Ethernet Implementation

PCS

XCVR FPGA Fabric

Data Valid


rx_clkout[0]

rx_coreclk[1]

rx_coreclk[2]

rx_coreclk[3]


FIFOGearbox6666

Data Valid

66

Data Valid

66

Data Valid

66

66

66

66

40

FIFOGearbox40

FIFOGearbox40

FIFOGearbox40

RXCh 0

RXCh 1

RXCh 2

RXCh 3

VL3

VL2

VL1

VL0



Figure 14 shows the recommended gearbox scheme in the ingress direction for 100G Ethernet implementation.

As shown in Figure 13 and Figure 14, each receiver channel in a 40G or 100G Ethernet link forwards a 40-bit wide data to the FPGA fabric.

Altera recommends enabling the rx_coreclk port (read clock for the receiver phase compensation FIFO) in the ALTGX MegaWizard Plug-In Manager instance used to instantiate the 40G/100G Ethernet link. Drive each channel’s rx_coreclk port with the recovered clock rx_clkout from any one of the 4/10 receiver channels in the 40G/100G Ethernet link. Use this rx_clkout signal to clock the gearbox logic for all receiver channels, as shown in Figure 13 and Figure 14.

f For more information about using rx_coreclk ports, refer to “User-Selected Receiver Phase Compensation FIFO Read Clock” in the Stratix IV Transceiver Clocking chapter in volume 2 of the Stratix IV Handbook.

Because the gearbox logic takes in 40-bit data and reads out 66-bit data at 257.8125 MHz, it could become empty. To avoid this, the gearbox logic must periodically stall reading out data. To indicate this periodic stalling of data to the FIFO, the gearbox logic sends a data_valid signal to the downstream FIFO. The FIFO must register the 66-bit data only when the data_valid signal from the gearbox logic is asserted. The FIFO reads out the 66-bit data to the PCS with the 156.25 MHz system clock, as shown in Figure 13 and Figure 14.

Figure 14. Recommended Gearbox Scheme in Ingress Direction for 100G Ethernet Implementation

XCVR FPGA Fabric

Data Valid


rx_clkout[0]

rx_coreclk[9]


FIFOGearbox6666

Data Valid

66

Data Valid

66

Data Valid

66

66

66

66

20

40

40

FIFOGearbox20

FIFOGearbox20

FIFOGearbox20

RXCh 0

RXCh 9

Demux

Demux

PCS

VL19

VL18

VL1

VL0


Page 18 Summary


1 The recovered clock (rx_clkout[0]) at 257.8125 MHz and the system clock at 156.25 MHz can have a maximum of ± 100 PPM difference. You must design the FIFOs shown in Figure 13 and Figure 14 to be able to compensate for this PPM difference between the two clock domains.

SummaryStratix IV GT devices offer a single-chip solution for implementing 40G/100G Ethernet MAC devices in a 40G/100G line card. This application note provides guidance on the implementation of 40G/100G Ethernet links in a Stratix IV GT device.

Document Revision HistoryTable 3 shows the revision history for this document.

Table 3. Document Revision History

Date Version Changes

September 2011 1.2■ Updated reference to Altera’s IP partners.

■ Updated document layout.

December 2009 1.1 Updated figures to remove 8G channels.

June 2009 1.0 Initial release.

Documents

AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix … · AN 570: Implementing the 40G/100G Ethernet Protocol in Stratix IV Devices September 2011 Altera Corporation Figure