IEEE 802.3ap Backplane Ethernet & Industry Testing Needs University of New Hampshire InterOperability Laboratory (UNH-IOL) Bob Noseworthy and Kristin Harris

IEEE 802.3ap Backplane Ethernet

&Industry Testing Needs

University of New Hampshire InterOperability Laboratory (UNH-IOL)

Bob Noseworthy and Kristin Harris+1-603-862-0090 [email protected]

XILINXTom Palkert

Advanced TCA Summit Oct. 17-19, 2006

2

Backplane Ethernet and Industry Testing Needs

UNH-IOL Workshop @ AdvancedTCA Summit 2006

Acknowledgements

• Special thanks to the following individuals for their contributions, input and support– Adam Healey, Agere Systems– John D’Ambrosia, Force10 Networks– Tom Palkert, Xilinx

3



Who we are• Bob Noseworthy

– Technical Director, University of New Hampshire InterOperability Lab (UNH-IOL)

– Active member, 802.3 Working Group– Developed early Fast, Gigabit, 10Gigabit Ethernet test

solutions• Tom Palkert

– System Architect, XILINX– Editor Clause 72 (10GBASE-KR), IEEE 802.3ap Backplane

Ethernet – Chair of the Fibre Channel T11.2 physical layer working

group• Kristin Harris

– Marketing, UNH InterOperability Lab (UNH-IOL)– Active with Ethernet Alliance Marketing Committee & various

tradeshows (Interop, GlobalComm, etc)– Focus on promoting new efforts, including Backplane

Ethernet

4



What you will learn

• Review the current state of the IEEE 802.3ap Backplane Ethernet Draft Standard

• Discuss the current activities addressing testing needs for Backplane Ethernet / AdvancedTCA

5



What will not be covered

• Pricing - We don’t know, we don’t care

• Specific product features – We’re not here to sell.

• Low-level fine grain details of: 10GBASE-KR, 10GBASE-KX4, 1000BASE-KX, XAUI, etc (feel free to ask afterwards)

6



Presentation Overview

• 13min: Overview of the draft Backplane Ethernet Standard

• 15min: 10GBASE-KR Highlights– Presented by Tom Palkert, Xilinx

• 10min: Backplane Ethernet in the AdvancedTCA landscape

• 7min: Facilitating Adoption of Backplane Ethernet

• 5min: Q&A

7



UNH-IOL Overview• The UNH-IOL is dedicated to improving interoperability between

devices and technologies.

• Leading provider of 3rd party testing services for the networking and data communications industry.

• 100% industry funded

• 150 + member companies provide market motivation

• De-facto test house for the 802.3 Ethernet community

• $20,000,000+ interoperability test beds

• Educates future engineers for the networking industry

8



Ethernet AdoptionFrom the Standard to the Marketplace

• IEEE develops new 802.3 standard, like 802.3ap amendment

• Ethernet equipment manufacturers develop new devices in conformance with 802.3 standard

• Companies bring devices to the UNH-IOL for IEEE standards based conformance and interoperability testing

• Conformance testing verifies product functionality

• Interoperability testing validates that product will work with other devices in the customer network

• Functional and interoperable devices accelerate the adoption of the technology in the marketplace

• UNH-IOL offers feedback to IEEE to improve standards

9



IEEE 802.3ap: Broad Market Potential

• Broad set(s) of applications

• Backplane is still an Ethernet Network

• PICMG adopted Ethernet as a fabric (PICMG 3.1)

• Balanced Cost (LAN vs. attached stations)• Evolving Ethernet price-performance makes Backplane Ethernet

more economically feasible than other interconnect solutions

• Multiple vendors, multiple users• 156 participants representing 33 distinct companies attended the

Ethernet over Backplane Call for Interest. This level of involvement indicates market growth through active product development and standards adoption

10



IEEE 802.3ap: Economic Feasibility

• Ethernet Backplane standardization will increase deployment and diversity of supply base to further reduce costs

• Ethernet IP re-use will lower implementation costs

• Cost factors known, reliable data• Integration of functionality will reduce cost

• Component costs will benefit from cost reduction associated with Moore’s Law

• Cost assessment for backplanes based on well-known material and component costs

11



Review of IEEE 802.3ap• Status• Objectives• Features

Per IEEE-SA Standards Board Operations Manual, January 2005

• At lectures, symposia, seminars, or educational courses, an individual presenting information on IEEE standards shall make it clear that his or her views should be considered the personal views of that individual rather than the formal position, explanation, or interpretation of the IEEE.

Note• Currently, IEEE P802.3ap is under review per the IEEE

802.3 Sponsor ballot process and is subject to change. The work in this presentation is per IEEE802.3ap Draft 3.0.

12



13



Current Objectives

a) Support full-duplex operation only.b) Provide for Auto-Negotiation among Backplane

Ethernet physical layer signaling systems.c) Not preclude compliance to CISPR/FCC Class A

for RF emission and noise immunity.d) Support operation over differential, controlled

impedance traces on a printed circuit board with 2 connectors and total length up to at least 1 m consistent with the guidelines of Annex 69B.– i) a 1 Gb/s PHY– ii) a 4-lane 10 Gb/s PHY– iii) single-lane 10 Gb/s PHY

e) Support a BER of 10-12 or better.

14



802.3ap Feature Overview

• 1m operation on improved FR-4 • 3 New Backplane Specific Phy types

– 1000BASE-KX: 1 Gb/s, – 10GBASE-KX4: 4 lanes, 10Gb/s– XAUI-

like– 10GBASE-KR: 1 lane, 10Gb/s

• New Auto-Negotiation• Forward Error Correction

15



Backplane Ethernet Channel

• Backplane Ethernet is primarily intended to operate over differential, controlled impedance traces up to 1 m, including two connectors, on printed circuit boards residing in a backplane environment. The performance of such an interconnect is highly dependent on implementation.

16



Definition of Improved FR-4

• Refers to the effective dielectric properties of the fabricated board

• Temperature and Humidity Tolerance (0-55°C, 10-90% non-condensing): Dk ±0.04, Df ±0.0010

• Resin Tolerance: Dk ±0.02, Df ±0.0005

• Source: F10_SBS_041906.pdf

f (GHz) Dk Df

0.1 3.60 0.010

1.0 3.57 0.010

2.0 3.50 0.011

5.0 3.50 0.011

10.0 3.40 0.012

20.0 3.20 0.0125

17



BP Ethernet Auto-Negotiation

• Defined by new Clause 73. – Clause 28-like Auto-Negotiation (like that used

by traditional 10/100/1000 Ethernet Phys)• Sits below PMDs

– allows optimal link selection and configuration before link is established via Differential Manchester Encoded signaling at 312.5MHz

– allows parallel detection for legacy Phys– still allows for Out-of-band configuration– performed only on Lane 0 (in case of KX4)

• Very fast – ~339.2ns page width (vs ~10ms for Clause 28)

18



1000BASE-KX

• Signaling speed: – 1.25Gbaud on 2 lanes (TX and RX)

• 8B10B coding identical to other Gigabit Ethernet Phys (1000Base-SX, LX, CX)

• No transmitter or receiver equalization

• New Auto-Negotiation support• No Forward Error Correction

19



10GBASE-KX4

• Signaling speed: – 3.125Gbaud on 8 lanes(4 TX, 4 RX)

• Fixed transmitter pre-emphasis– Similar to 10GBASE-CX4

• Receiver equalization• New Auto-Negotiation support• No Forward Error Correction (FEC)

20



10GBASE-KR

• Signaling speed: 10.3125Gbaud on 2 lanes (TX,RX)• Mandatory Start-Up Protocol (Training) to bring

linkup– Follows Auto-negotiation– Process trains RX equalizers and allows link partner’s

receiver to optimally tune the transmit equalizer over the backplane interconnect

• Optional support for FEC– Adds no overhead by exploiting 64B/66B coding overhead– Intended to improve an already good link (2dB Coding

Gain) Corrects up to 11bit burst errors– Suitable for DFE-based Rx Equalizers to mitigate error

propagation

21



10GBASE-KR Highlights

• Following Presentation by Tom Palkert, Xilinx

10G-BASEKR highlights

Tom PalkertSystem Architect

XilinxOct. 2006

Outline

• Application space

• Normative points

• Backplane challenges and solutions

Modular Platform Industry Trends

• Ethernet has been the default backplane of choice for modular platforms

Application and Data Application and Data ServicesServices

Access, Edge & Access, Edge & Infrastructure ServicesInfrastructure Services

Compute DensityCompute DensityIO DensityIO Density

Car

rier

Car

rier

(CO

-NE

BS

)(C

O-N

EB

S)

Ent

erpr

ise

/E

nter

pris

e /

Dat

a C

ente

rD

ata

Cen

ter

NetworkNetwork ServersServers

WirelessWirelessAccessAccess

EdgeEdgeMulti-Multi-

ServicesServicesDatacenterDatacenterSwitch &Switch &

AppliancesAppliances

PedestalPedestalServersServers

LargeLargeSMPSMP

ServersServers

HPCHPCClustersClusters

VolumeVolumerackrack

IA serversIA serversModular PlatformsModular Platforms

LAN

SAN

Standard form factor Telco (aTCA) OR Modular Server Platforms

Typical 802.3ap application

2’’

16’’

2’’

E/O

Switc

h

802.

3ap

PHY

16’’

802.

3ap

PHY

E/O

XAU

IR

etim

er

2’’

1/10G Ethernet port

10G Production XENPAK*

802.3ap802.3ap

good eye

bad eye

*Third-party brands and names are the property of their respective owners.

TXPre-emphasis

IntervalNormalInterval

1.0 (0 dB)

0.707 (3 dB)

0.5 (6 dB)

0

Normative and Informative pts

RX

Launch: Via StubsVia Adjacency

Trace dimensionsTrace proximityBoard materialVia stubsConnector:

Return lossCrosstalk

Normative RXNormative TX Informative channel

Rise time, AmplitudeIntersymbol interference

Reflections, crosstalk

Insertion Loss (S21)Return Loss (S11)

Insertion Loss deviation (ILD)

Rise timeEqualization

Amplitude, Ripple

TX compliant waveform

Recommended S21 and Insertion Loss limits for channel

Insertion Loss Deviation

10GBASE-KR RX characteristics

10GBASE-KR interference tolerance parameters

-60

-40

-20

0

0 2 4 6 8 10

Kap2 - HmzdN13SITdyneGbxN13SIM34 - PicmgDF - XlnxGbxFr4E - Xlnx0xtFr4F40 - WinSipRogG - Tier1HmzdF40 - WinSipN13Agreed Ch Ad Hoc

Frequency (GHz)

SD

D2

1 (

dB

)Compare Backplanes

With Channel Ad Hoc Curve

ATCA Backplanes wrt 10GBASE-KR channel model

Conclusion

• ATCA is a prime candidate for 10GBASE-KR applications– Feasibility demonstrated for multiple designs

• 10GBASE-KR specifies normative TX and RX with an informative channel– Difficult/impossible to specify a ‘typical’

backplane due to differences in materials and design parameters

34



Backplane Ethernet in the AdvancedTCA landscape

• PICMG 3.1 was created in the absence of an industry standard for Backplane Ethernet– 1000BASE-KX is expected to supersede 1000BASE-BX– 10GBASE-KX4 is expected to supersede 10GBASE-BX4– Desirable to have electrical compatibility with PICMG 3.1

• IEEE P802.3ap supplies PICMG 3.1 with a roadmap– 10GBASE-KR supports 10 Gb/s serial transmission over

an ATCA backplane (up to 4 channels per blade)– Auto-negotiation protocol to facilitate incremental

upgrades to chassis already in the field

• Source: DesignCon2006_8023ap_Agere_Tyco.pdf

35



Side note on 1000Base-T vs 1000BASE-KX

• 1000BASE-T is designed to support 100 m of unshielded twisted pair cabling (Category 5e) at 1Gb/s– Robust, mature technology; DSP intensive– Small-scale integration, relatively high power (~750 mW

per channel)– 4 lanes, 125baud, Nyquist of 62.5 MHz

• 1000BASE-KX is targeted for backplane applications– Robust & mature technology; 8B10B block code, Serdes– Large-scale integration, low power (< 100 mW per channel)– Backplane complexity decreases but signal integrity

concerns increase– 2 lanes (TX & RX), 1250baud, Nyquist of 625 MHz

• Source: DesignCon2006_8023ap_Agere_Tyco.pdf

36



14 slot Dual-Star ATCA System

• 1000BASE-KX – 4Gbps per linecard, 48Gbps per

backplane

• 10GBASE-KX4– 10Gbps per linecard, 120Gbps per

backplane

• 10GBASE-KR– 40Gbps per linecard, 480Gbps per

backplane

37



Backplane Ethernet Channel• Efforts such as PICMG’s Interconnect Channel

Characterization (ICC) Subcommittee are expected to normatively define channel requirements for a given application– 802.3ap addresses a full system – not just a backplane

channel– Channel is informatively defined in 802.3ap D3.0 Annex

69B– IEEE 802.3ap D3.0 defines only the TX and RX

characteristics, leaving the backplane channel designer flexibility in developing a channel that can support the normative TX and RX specs

– 802.3ap D3.0 does not define number or type of con-nectors, tracelengths for backplanes or linecards, etc.

– From IEEE 802.3ap D3.0, it would be impossible to declare a given backplane-only channel as Backplane Ethernet conformant

38



Backplane Ethernet in the AdvancedTCA landscape, par deux

– IEEE 802.3ap – PICMG 3.1

• Current PICMG Ethernet Fabric specification

– PICMG ICC • Developing methodologies for channel definition within

PICMG

– PICMG RES (Requirements Engineering SubCom)• Map specification requirements and generate use

profiles

– AIW (ATCA InterOperability Workshops)• #14 coming up soon, focused on PICMG needs (such

as ATCA/AMC/TCA)

39



Backplane Ethernet in the AdvancedTCA landscape, par deux,

cont.– CP-TA (Communications Platform Trade

Assoc.)• Promote ATCA adoption by advancing

interoperability testing “CP-TA plans to drive a mainstream market for open industry standards-based communications platforms by certifying interoperable products.” Souce: http://www.cp-ta.org/about/

– SCOPE• “help, enable and promote the availability of open

carrier grade base platforms based on Commercial Off The Shelf (COTS) hardware / software and Free Open Source Software (FOSS) building blocks, and to promote interoperability to better serve Service Providers and consumers” Souce: http://www.scope-alliance.org/about.html

40



Facilitating Adoption of Backplane Ethernet

• Draft IEEE 802.3ap soon to be an approved standard

• Ethernet Alliance will be promoting Backplane Ethernet through tradeshows, plugfests and other venues

• PICMG, CP-TA, SCOPE are expected to embrace the approved 802.3ap standard

• Need for validation of emerging Backplane Ethernet solutions remains

41



Benefits of InterOperability

• Near-term:– Closed systems – enable system designers w/

increased component selection freedom

• Long-term/Ideal:– Fully open system – user-level freedom to

select approved/compatible components– Nirvana – each component has an

independent specification to ensure conformance and interoperability (will PICMG ICC lead the way)

42



Industry Testing Needs

• 3rd party vendor-neutral environment

• Confidential test of products• Detailed debugging and reporting• Need for an InterOperability Test

Bed: Multitude of representative test channels and link partners to validate interoperability

43



Conformance vs InterOperability

• Conformance test: – Predicts future interoperability– For Backplane Ethernet applies only to TX & RX not

Channel– Can never be 100%– Even a 100% conformant component does not ensure that

a system utilizing the component will also be conformant• Interoperability test:

– Demonstrates interoperability with set of chosen devices– Includes real system effects that may not be fully included

in conformance requirements– For AdvancedTCA, can demonstrate component

functionality in an ATCA environment (line cards, backplanes, connectors, etc)

– Can never be 100%• Need for both to establish sufficient confidence in

solutions

44



UNH-IOL Involvement

• The InterOperability Laboratory will continue to be engaged with the IEEE 802.3, the Ethernet Alliance and other interested forums to meet the industry’s need for test and evaluation of Backplane Ethernet / AdvancedTCA products.

• UNH-IOL Backplane Ethernet test effort forming, contact us for more info

45



References & Further Reading

• SBS 2006: The State of IEEE 802.3ap Backplane Ethernet – John D’Ambrosia, Force10 Networks– http://www.ethernetalliance.org/technology/presentations/F10_SBS_041906.pdf

• DesignCon 2006: The Impact of the Ethernet Ecosystem on Backplane and System Design

– http://www.ethernetalliance.org/technology/white_papers/The_Impact_of_the_Ethernet_Ecosystem_on_Backplane_and_System_Design_-_DesignCon2006.pdf

• DesignCon 2006: The State of the IEEE 802.3ap Backplane Ethernet – Adam Healey, Agere Systems; John D’Ambrosia, Force10– http://www.ethernetalliance.org/technology/white_papers/

DesignCon2006_8023ap_Agere_Tyco.pdf

46



Q&A

47



Improving Networks Worldwide.

• Trusted, third party product validation

• Detailed interoperability testing and debugging

• Industry developed, trusted test procedures

• Global engagement and industry involvement

48



Bios• Bob Noseworthy has been a part of the UNH-IOL for over a

decade, and has contributed to the 802.3z, ab, ae, ak, an and ap standards efforts from their inception. Activities include proving the technical feasibility of XAUI for IEEE 802.3ae, demonstrations for the 10 Gigabit Ethernet Alliance, and developing PMD and PCS test solutions for XAUI and related 10GE technologies with fellow 10GE Consortium staff. Today Bob serves as Technical Director of the UNH-IOL, and holds both an MS and BS in Electrical Engineering. Responding to industry interest, Bob will also be organizing testing efforts for Backplane Ethernet technologies, please feel free to contact him for more information.

• Tom Palkert is active in the architecture and standardization of high speed serial I/O for Fibre Channel, Ethernet, SONET and backplane applications. He is currently the Chair of the Fibre Channel T11.2 physical layer working group and editor of 10GBASE-KR. He is a past member of the Board of directors of the Optical Internetworking Forum (OIF). He received his degrees and patents so long ago that they no longer matter.

Documents

IEEE 802.3ap Backplane Ethernet & Industry Testing Needs University of New Hampshire InterOperability Laboratory (UNH-IOL) Bob Noseworthy and Kristin Harris