Ethernet Evolution: The Path to 100 Gigabit Ethernet · The Path to 100 Gigabit Ethernet John D’Ambrosia - Scientist, Components Technology Chair, IEEE 802.3 HSSG ... – Cable

Ethernet Evolution: The Path to 100 Gigabit Ethernet

John D’Ambrosia -Scientist, Components TechnologyChair, IEEE 802.3 HSSG

Version 2.0 16 Oct 2006

2

Per IEEE-SA Standards Board Operations Manual, January 2005

At lectures, symposia, seminars, or educational courses, an individual presenting information on IEEE standards shall make it clear that his or her views should be considered the personal views of that individual rather than the formal position, explanation, or interpretation of the IEEE.

3

Acronym Cheat SheetCFI – Call for InterestDWDM – Dense Wavelength Division MultiplexingEMI – Electro-magnetic InterferenceGbps – Gigabit per SecondHSSG – Higher Speed Study GroupITU – International Telecommunications UnionIETF – Internet Engineering Task ForceJEDEC - Joint Electron Device Engineering CouncilMAC – Media Access ControlMDI – Medium Dependent InterfaceMSA – Multi Source AgreementOIF – Optical Internetworking ForumPCS – Physical Coding SublayerPMA – Physical Medium AttachmentPMD – Physical Medium DependentPHY – Physical Layer DeviceSERDES – Serialize / De-serializeSMF / MMF – Single Mode Fiber / Multi Mode FiberTbps – Terabit per SecondWIS – WAN Interface SublayerXGMII – 10 Gigabit Media Indpendent Interface

4

Agenda

The Ethernet EcoSystemIEEE 802.3 Higher Speed Study GroupConsiderations for System Development

5

Agenda


6

The Ethernet Ecosystem

Research, Education and Government

Facilities

ResearchNetworks

Broadband Access

BroadbandAccess Networks

Data Centers and Enterprise

EnterpriseNetworks

Content Providers

ContentNetworks

Internet BackboneNetworks


IEEE 802.3 HSSG “Call-For-Interest”, 7/2006

Internet eXchange andInterconnection Points

7

What is Driving the Need for Higher Speed Ethernet?

Research network trends– 10 GbE WANs– Cluster and grid computing

Data center trends– Lots of GbE and 10 GbE servers

ISP / IX trends– Personalized content – Consumer broadband– 10 GbE peering

Aggregating 10 GbE links with LAG is an interim solution

Service ProviderService Provider

Data CenterData Center

Government/ResearchGovernment/Research

8

802.3ad Link Aggregation (LAG)

It is being usedTemporary fix for increased bandwidth demandIncreased complexity– Difficult to plan for capacity and traffic engineering– Harder to manage & troubleshoot multiple physical links based

on a single logical interface– Cable & link management

Uneven distribution of traffic– Limitations in the standard– Inefficient distribution of large flows – Load balancing requires packet inspection or other knowledge


9

Bandwidth and Growth Projections

Research, Educationand Government

Facilities

Consumer Broadband Access

Content Providers

Corporate Data Centers and Enterprise

ResearchNetworks

BroadbandAccess Networks

EnterpriseNetworks

ContentNetworks


Internet BackboneNetworksLevel 3: 8x10 GbE LAG today,

BW growth 15x in 5 years (~70%/year)

Internet Exchanges: Up to 8x10 GbE LAG today,

BW growth 50-75% per year for next 3 – 5 years

Cisco: 10GbE today, 40+ GbE (100 GbE

preferred) in 5 years

Cox: 10 GbE today, BW growth 50-75% per

year for next 3 – 5 years

Comcast: 4x10 GbELAG today, 3X BW

increase in 3 to 5 years

Yahoo!: 4x10 GbE LAG today,BW doubling in <12 months

LLNL: 4x10 GbE LAG and 500x10 GbE ports

today, 10x speed requirement in 5 years

on deployed ports

ESnet: 10 GbE today, 10 Gbps on 20+ links 5 years from now; 5-

10 locations will require more than 40

Gbps


10

Agenda


11

IEEE 802.3 July, 2006 Plenary

Higher Speed Study Group (HSSG) “Call-For-Interest”– http://grouper.ieee.org/groups/802/3/cfi/0706_1/CFI_01_0706.pdf– Led by John D’Ambrosia / Joel Goergen, Force10 Networks

Broad Industry Support– End Users– System Vendors– Component Vendors

“Higher Speed Study Group” formation approved

Sept 2006 - John D’Ambrosia, Force10 Networks, Confirmed as HSSG Chair

12

IEEE 802.3 HSSG Reflector and Web

To subscribe to the HSSG reflector, send an email to:[email protected]

with the following in the body of the message:subscribe stds-802-3-hssg <your first name> <your last name>end

HSSG web page URL:http://grouper.ieee.org/groups/802/3/hssg/index.html

13

Developing a Standard

Call for Interest

Study Group

Task Force

Working Group Ballot

Sponsor Ballot

Standards Board Approval

Publication

Feasibility and Research

Ideas From Industry

IEEE~4 Years

Industry Pioneering

1 Year

Higher Speed Ethernet is

Here

Ad Hoc Efforts

CFI July 18, 2006

14

Normal Study Group Output

Project Authorization Request5 Criteria– Broad Market Potential– Compatibility – Distinct Identity– Technical Feasibility– Economic Feasibility

Project Objectives

15

MDI

Objectives – Speed?

Higher Speed Ethernet– Means faster MAC data rate

Options Discussed– 40 Gbps– 80 Gbps– 100 Gbps– 120 Gbps– Scaleable

IEEE 802.3 HSSG adopted 100 Gb/s (Nov 2006)

64B/66B PCS

MAC Client

Higher Layers

MAC ControlMAC

Reconciliation Sublayer (RS)

WIS64B/66B PCS

PMA PMAPMD PMD

MDIMEDIUM MEDIUM

XGMII XGMII

10GBASE-R 10GBASE-W

MAC = Media Access ControlMDI = Medium Dependent InterfacePCS = Physical Coding SublayerPMA = Physical Medium AttachmentPMD = Physical Medium DependentWIS = WAN Interface SublayerXGMII = 10 Gigabit Media Independent Interface

MAC Rate = 10GPHY Rate < 10G

16

Objectives –PHY Types and Reach?

10 Gb Ethernet Historical Perspective

To date, IEEE 802.3 HSSG has adopted (Nov 2006)– Support at least 10km on SMF.– Support at least 100 meters on OM3 MMF

Serial4 lanes

Twisted Pair

4 lanes

SerialSerial

WDMSerialSerial

50 um / 62.5 um MMF220 m1310 nm10GBASE-LRM

1 m1 m

100 m15 m

40 km10 km

300 m 10 km

300 / 33 m

1550 nm1310 nm

1310 nm

850 nm

Improved FR-410GBASE-KX4Backplane

Copper

Fiber

Improved FR-410GBASE-KR

UTP10GBASE-TTwinaxial10GBASE-CX4

Single Mode Fiber10GBASE-ER / W Single Mode Fiber10GBASE-LR / W

50 um / 62.5 um MMFSingle Mode Fiber

10GBASE-LX4

50 um / 62.5 um MMF10GBASE-SR / W

17

Summary of Adopted Objectives

To date, IEEE 802.3 HSSG has adopted:– Support full-duplex operation only.– Preserve the 802.3/Ethernet frame format at the MAC Client

service interface.– Preserve minimum and maximum FrameSize of current

802.3 Std.– Support a speed of 100 Gb/s at the MAC/PLS interface.– Support at least 10km on SMF.– Support at least 100 meters on OM3 MMF.

All objectives adopted at IEEE 802 November 2006 Plenary

18

Existing Backbone Networks

ResearchNetworks

BroadbandAccess

Networks

EnterpriseNetworks

ContentNetworks



ResearchNetworksResearchNetworks

BroadbandAccess

Networks

EnterpriseNetworksEnterpriseNetworks

ContentNetworksContent

Networks



10 Gbps λ DWDM

40 Gbps λ DWDM

10 Gbps λ DWDM – wide deployment

40 Gbps λ DWDM– Being evaluated– Initial deployments– Return on Investment?– Interim solution?

Rate Choice– 100 Gbps maps into 10 Gb/s λ

DWDM – 120 Gbps maps into 40 Gb/s λ

DWDM“Ethernomics”– The Ecosystem is cost sensitive– Choice of 120 vs 100 Gbps

– Faster?– Wider?

IEEE 802.3 HSSG choose 100 Gbps

Ethernet Ecosystem

19

Future Backbone Networks?

“Ethernomics”– The Ecosystem is cost sensitive

20+ Gbps λ DWDM?– Lower cost than 40 Gb/s– Re-use of existing infrastructure?– Return on Investment?

100 Gbps λ DWDM– Return on Investment?– Future capacity requirements

ResearchNetworks

BroadbandAccess

Networks

EnterpriseNetworks

ContentNetworks



ResearchNetworksResearchNetworks

BroadbandAccess

Networks

EnterpriseNetworksEnterpriseNetworks

ContentNetworksContent

Networks



20+ Gbps λ DWDM?

100 Gbps λDWDM?

Future Ethernet Ecosystem

20

Thoughts Related to Backbone

40 Gbps λ DWDM– Not widely deployed– Interim solution to bandwidth crunch– Support seen by

– Transport equipment providers – Service providers

Choice of 100 Gbps– Service providers are evaluating Ethernet for transport– Maps into existing deployed 10 Gbps λ DWDM– Support for 100 Gbps λ DWDM seen by

– Transport equipment providers– Service providers

– Recent decision by IEEE– Future discussions and work needed

21

Other Bodies

The Ethernet AllianceOptical Internetworking Forum – 20Gbps to 25Gbps electrical interface project.– 100Gbps to 160Gbps system interface project.

ITUIETFJEDEC

22

Agenda


23

Chassis Slot Density

400Gbps to 500Gbps2009

120Gbps2006/7 – in design now

Based on max back plane thickness of 300mils, 20TX and 20RX differential pipes.

60Gbps2004

40Gbps2000

Slot densityYear System Introduced

24

Chassis Design

Chassis design issues:– Backplane and channel

signaling– Lower system BER– Connectors– N+1 switch fabric – Reduced EMI– Clean power routing

architecture– Thermal and cooling– Cable management

All design aspects must also meet local regulatory standards

25

Backplane Considerations

Slot CapacitySwitching CapacityPerformance– Signal coding– BER– Impacts SerDes Design

Design and technology drives scalability– Advanced fiberglass materials– Unique conductor layers– Engineered trace geometries

26

Anatomy of a 100 Gbps Solution: Designing for EMI Compatibility

EMI is Electro-magnetic Interference, i.e. noiseToo much noise interfere can cause bit errorsConcerned about two types of EMI– Conducted through

power– Radiated through air

0

10

20

30

40

50

60

30M 50 60 80 100M 200 300 400 500 800 1G

Leve

l in

dBµV

/m

Frequency in Hz

EN 55022 Class A Electric Field S

E300 Power Supply Radiated Emission for 30 MHz to 1 GHz

27

Power Considerations

End User Restrictions?Total system wattage?Input power quality not specifiedHigher speeds require lower noise

28

Thermal Management Considerations

Cooling capacity per slot?Front to back filtered airflow for carrier deploymentsCooling redundancyHeat can affect material performance which affects high-speed signaling performance

29

Line Card Components

PHY Network Processor

SRAMCAM

MACFabric

Interface

SwitchFabric 1

SwitchFabric N

MediaInterface

DRAMLine Card

Pluggable or fixed media interfaces

Ethernet PHYsical Layer

• Line drivers/receivers• Encoders/decoders• Timing

Ethernet Media Access Control

• Framing• Addressing• Error handling• Flow control

Packet lookup memory

Packet buffer memory

Packet lookup, classification and forwarding functions

Interfaces to switch fabrics

30

High Speed SignalingInterfaces will use SERDES

– Short Reaches– Long Reaches

SERDES will replace parallel busing ASIC’s need new higher speed SERDES

– 6.25 Gbps today– 10 Gbps emerging

Discrete higher speed SERDES for 20 Gb/s?

PHY Network Processor

SRAMCAM

MACFabric

Interface

SwitchFabric 1

SwitchFabric N

MediaInterface

DRAMLine Card

SERDES

31

Higher Speeds Drive Switch/Router Requirements

Driving architectural requirementsMassive hardware and software scalability– >200 Gbps/slot switch fabric capacity – Support for several thousand interfaces– Multi-processor, distributed architectures

Fast packet processing at line-rate– 100 GbE is ~149 Mpps or 1 packet every 6.7 ns(10 GbE is only ~14.9 Mpps or 1 packet every 67 ns)

32

Higher Speeds Drive Density

100 Gbps Ethernet will benefit allDrives 10 GbE port density up and cost downPossible line-rate combinations– 1 x 100 GbE port– 10 x 10 GbE ports– 100 x 1 GbE ports– And even more oversubscribed port density…

The more things change the more they stay the same….

33

System Port Count Cycle

2002 2004 2006 2010*2008

100 GE 56 PortsStandard In Development

GE 336 Ports 672 Ports 1260 Ports 2240 Ports ?

10 GE 28 Ports 56 Ports 224 Ports 560 Ports 2240 Ports

34

Conclusions

Aggregation is the Killer Application– From carrier to the consumer– Not talking desktop

100G Ethernet is coming!The more things change, the more they stay the same– Lower speeds (1G and 10G)

– Higher density– Integration and cost optimization

– Port density of 100G– Higher scalability– Integration and cost optimization

35

Thank You

Documents

Ethernet Evolution: The Path to 100 Gigabit Ethernet · The Path to 100 Gigabit Ethernet John D’Ambrosia - Scientist, Components Technology Chair, IEEE 802.3 HSSG ... – Cable