INTEROPERABILITY –THE FOUNDATION OF ETHERNET …...Apr 16, 2020 · Ethernet –Success through interop and flexibility •Ethernet is the dominant solution for high speed interconnect

© 2017 Ethernet Alliancewww.ethernetalliance.org

April 16, 2020

INTEROPERABILITY – THE FOUNDATION OF ETHERNET SUCCESS

Presenters:Shawn Nicholl, Xilinx – Enabling Interoperability Today for Next-Generation Ethernet SolutionsJeff Twombly, Credo – Interoperability – 400G Beyond – The Case for Active Cables Paul Brooks, Viavi – Driving Interoperability Through Test & MeasurementRay Nering, Cisco – Interoperability – The Foundation of Ethernet Success – A Systems Vendor Perspective

Moderator: David J. Rodgers, Teledyne LeCroy

© 2017 Ethernet Alliance

Presentation Disclaimer

The views being presented in this educational material on the respective IEEE 802.3 standards under

consideration are the views of the author(s), and do NOT represent a formal position or interpretation of the

respective standard by The Ethernet Alliance.

2


The Ethernet AllianceA Global Community of End Users, System Vendors,

Component Suppliers and Academia

3

§ Our Mission• To promote industry awareness, acceptance and advancement

of technology and products based on, or dependent upon, both existing and emerging IEEE 802 Ethernet standards and their management.

• To accelerate industry adoption and remove barriers to market entry by providing a cohesive, market-responsive, industry voice.

• Provide resources to establish and demonstrate multi-vendor interoperability.


Ethernet Alliance Strategy

§ Global Outreacho Cultivate diverse, worldwide membership

§ Technology Advancemento Support and advocacy of standards

developmento PoE Certification Program

§ Thought Leadershipo Annual Ethernet Roadmapo Technology Exploration Forums

4

§ Collaborationo Facilitate interoperability testing eventso Industry Plug Fests supporting member and

technology initiativeso Partnerships with peer organizationso Involvement with SIGs and MSAs

§ Promotion and Marketing of Etherneto Industry media and analyst engagement o Tradeshows and eventso Blogs, white papers, industry articles

Expanding the Ethernet Ecosystem and Supporting Ethernet Development


• Plugfests in the last 12 months– High Speed Networking

• Past Trade Shows – OFC– SC– ECOC– Other

• PoE Certification Program– Gen 1 (.3af / .3at)– Gen 2 (.3bt)

5

The Ethernet Alliance - Investment in Multi-vendor Interoperability


April 16, 2020

ETHERNET INTEROPERABILITY –TEST AND MEASUREMENT CONSIDERATIONS

David J. Rodgers – Teledyne LeCroy PSG


Test and Measurement considerations for High-Speed Ethernet applications

§ Ethernet Standards Evolving at Breakneck Pace• 25GbE to 100GbE, now PAM4 50GbE, 200&400GbE• Soon, 100GbE to 800GbE• Automotive, Industrial, Commercial

§ Ethernet Fabrics Fueling Storage Explosiono Speed and Optimization meeting QOS Expectations

• NVM-oF, iSCSI, FCoE, NFS, IBXoE, FCIP, iSER, iWARP, RoCE, Routable RoCE (v2), and so on, and so on…

§ Standards beget Interoperability?o Interpretation and implementation differences abound

7


Effective Observation

8

Fabric Management Utility/HypervisorTraffic Tap and DPI

(Wireshark®)Line RateAnalysis


“You Can’t Test What You Can’t Measure!”

9

• Hardware Test tools have to change to keep up with market and technology demands

• New Speeds adding new complexities– NRZ to PAM4 signaling

• There is a “protocol” to the Phy– Auto-negotiation– Link Training– FEC

• No two vendors implementations are identical


Test and Measurement – The Bell Weather§ T&M vendors; We’re On the Leading Edge!

• Partners in Pain of “being first”• Traditional – Signal Integrity Tools• New to the Scene – Protocol Specific Tools

§ Purpose Built Protocol Tools!o Compliment to, not replacement for Traditional Tools

• Optimized for the Fabric/Device under testo Becoming essential in all HW/SW test environments

§ The Goal; Testing must be “standardized” and repeatableo Interop PlugFests, 3rd party testing services

10


Key Interoperability Challenges

11

• Identifying Participants– Characterizing Functionality of All Ecosystem Players

• Determining Root Cause– Eliminate “finger pointing”

• Crafting the Solution• Remediation Validation– Test the fix

• Timely Resolution!


• Ubiquitous deployment requires vendor interoperability• Establishing the “link” has become more

complicated than ever• It’s imperative to know what’s “on the wire”– Testing no longer “ends” at the connector

• Integrating legacy and new Ethernet technologies creating new challenges

4/16/2012

Test & Measurement – Navigating the #NextEthernetEra


Test & Measurement – ReCap

13

• Ethernet is a Juggernaut• Content Delivery and Storage Demands are High• Consistent and Predictable Interoperation

is Mandatory• Speed adds Exponential Influences on the

Eco-System• Testing, Testing, Testing!– Tool Sets and Methodologies Must Evolve


April 16, 2020

ENABLING INTEROPERABILITY TODAY FOR NEXT-GENERATION ETHERNET SOLUTIONS

Shawn Nicholl - Xilinx


Programmability for Early Adopters• Early developers of Ethernet solutions are

constantly challenged• Specs are in flux• Proof of Concept development• Interoperability testing

• Programmable platforms (FPGA and ACAP) are the solution• Early implementations without "locking in" the final

solution• Test & Measurement equipment

• Programmability enables compliance testing through evolving standards• Vendors may need to tweak implementations

15


Programmability and Continuous Development

• Recent trend to in-fill Ethernet rates• 25GE, 50GE, 200GE• NBASE-T (2.5GE, 5GE)

• Continuous evolution of products• Evolution of supported Ethernet rates for a port• Evolution of connectivity, form factors (QSFP-DD/OSFP) and break-out• Evolution of applications requiring wire-speed support

• Programmability enables continuous integration• Agile development for products

16

Source: QSFP-DD MSA( http://www.qsfp-dd.com/ )

Source: OSFP MSA( https://osfpmsa.org )

http://www.qsfp-dd.com/

https://osfpmsa.org/


Building Blocks for Interoperability• All network traffic passes over electrical interfaces of

silicon devices• Cutting edge transceivers operate at 56G PAM4 and 112G PAM4

serial signaling rates• Ethernet applications are emerging to take advantage of those

rates, while also expanding to leverage lower rates

• Applications need flexible Serdes• PAM4 and NRZ signaling• Wide range of rates

• Forward Error Correction• PAM4 Bit Error Ratio (BER) demands use of FEC

17


Creating Interoperable Solutions

• Judicious hardening of functions • 16nm devices have 100 Gbps FEC in the Serdes• 7nm devices have 600 Gbps protocol blocks• Area and power efficient

• Platform for interoperability• Develop new Ethernet products• Ensure legacy applications continue to work

• Adaptable engines • Future-proof for emerging protocols• Programmability for applications evolution

18


5G Wireless Application – Radio over Ethernet (RoE)

19

IEEE 1914.3 RoE CPRI

CPRI

eCPRI

eCPRI

IEEE 802.3brFrame Preemption

IEEE 802.1CMTSN

Ethernet Backhaul


Protocol Offload and NFV Acceleration

20

Unencrypted 100GE

Encrypted 100GE

Unencrypted 50GE

Encrypted 25GE

Encrypted 400GE

PCIe to Host

Distributed Network Functions

SecureEthernet


Ethernet Aggregation Application – FEC Variety

21

100G CAUI-4RS(528,514) FEC

200GAUI-4RS(544,514) FEC

25GAUIRS(528,514) FEC

SFICL74 KR FEC

400GAUI-8RS(544,514) FEC


Ethernet-Centric World

• World is becoming Ethernet-dominated• Countless new signaling and port rates• Longevity of legacy rates

• Desire a scalable paradigm• Supportable by everyone - from developers

to testers• Supportable over a range of markets• Ethernet link interoperability and reliability

should be a given for developers

• Testing Automation • Need automation at all levels

22


Test Automation with Programmable Devices

• Pre-hardware simulations• Powerful regressions • Constrained random test• Shared vectors ensure interoperability

• Hardware-based Proof of Concept• Enables early interop and in-system testing• In-house or canned platform

• Ethernet test equipment automation hooks• Script-based controls• Library re-use over multiple generations

23


Standards Development and Interoperability

• Industry must unite to develop interoperable networking standards• Standard and industry bodies like IEEE, OIF, EA, and ETC foster a culture of

openness• Create an improved solution without re-inventing the wheel• Leverage existing solutions for newer market requirements like 800GE

• Strong, well-written standards lead to interoperability• Whole industry benefits from hard work done at the ground floor• Faster bring-ups and less cost wasted in field debug• Companies should become contributors to standards

• Commercial benefits• Common goal and common development focus• Volume improves cost (economies of scale)

24


Tradeshows and Interop events

• Multi-vendor interoperability testing is critical • Customers want the plug-and-play nature of Ethernet

• Ethernet Alliance has taken a leadership role• Live demonstrations at tradeshows • Hot-plug events

• Interoperability events bring many vendors together • Ensure that their equipment talks with each other

25


Summary

• Developers of interoperable Ethernet technologies benefit from programmable platforms• Programmability enables continuous evolution of products• FPGA and ACAP devices provide the building blocks

• In an Ethernet-dominated world, interoperability is supported through:• Efficient test automation• Strong contribution to standards and industry bodies • Live tradeshow demonstrations and hot plug events

• Get involved!

26


April 16, 2020

INTEROPERABILITY –400G BEYOND – THE CASE FOR ACTIVE CABLES

Jeff Twombly – CREDO


Direct Attached Cables – Old Trusty Work Horse

• DAC has been the industry’s favorite– Low cost– Draws no power– Availability

• But, as our industry moves beyond 25Gb NRZ, DACs have become problematic– Performance, interoperability, gauge density, routing challenges,…

28


400G and Beyond - Problems with DACs

• DAC places burden of equalization on the switches– What works on one port may not on the other

• There are established training protocols; however, – “No two vendors implementations are identical”

• Routing challenges for emerging DDC routers and switches– 100’s of fabric connections required

• As transfer rates increase to 800G– Testing and interoperability issues become more problematic

29


400Gb and 800Gb Ethernet Link Budgets & Cu Gauge

30www.ethernetalliance.org

800Gb Ethernet LR Link Budget

400Gb Ethernet LR Link Budget OD (mm)

Scale Cross

Section

Bend Radius (mm)

400G DAC 1m – 30 AWG 2 x 6.7 33.5

400G DAC 2m – 28 AWG 2 x 7.9 39.5

400G DAC 2.5m – 26 AWG 11.4 55.0

OD (mm)

Scale Cross

Section

Bend Radius (mm)

800G DAC 1m – 28 AWG 2 x 6.7 33.5

800G DAC 2m – 26 AWG 11.4 55.0


100G NRZ

400G PAM4 time

Adoption of 400Gb In the Data Center

31

2020 DDC 400G Applications• DR4 Optics for Spine Interconnects• 400G CLOS AEC for Fabric Interconnects

31

64xDD Fabric64xDD Fabric64xDD Fabric64xDD Fabric

32 Ports

32 Ports

32 Ports

32 Ports

32 Ports

32 Ports

32 Ports

32 Ports

128xDDFiber

128xDDFiber

256 –312 Cable CLO

S Fabric

19” Rack

Fibe

r


100Tb Ethernet DDC – Cable Management

32

OD (mm)

Scale Cross

Section

Bend Radius (mm)

400G DAC 1m – 30 AWG 2 x 6.7 33.5

400G DAC 2m – 28 AWG 2 x 7.9 39.5

400G DAC 2.5m – 26 AWG 11.4 55.0

HiWire AEC – 30 AWG 2 x 6.5 32.5

HiWire CLOS <1.5m – 34 AWG 6 24

HiWire CLOS 1.5m–3m – 32 AWG 8 32

• HiWire CLOS AECs consume up to 70% less volume than DACs• HiWire CLOS can be routed with 15cm Cable Managers• DACs require >50cm cable managers

• Can break cages due to weight and stress• Mixed DACs and Optics have shown signal integrity issues

DACs are too thick and bend radius is too largefor 400G the Distributed Disaggregated Chassis Use


32 Ports

32 Ports

32 Ports

32 Ports

32 Ports

32 Ports

32 Ports

32 Ports

128xDDFiber

128xDDFiber

Ingr

ess/

Egre

ss F

iber


32 Ports

32 Ports

32 Ports

32 Ports

32 Ports

32 Ports

32 Ports

32 Ports

128xDDFiber

128xDDFiber

Cable Fabric

128

VerticalCable

Manager19” Rack #1

Fiber/Cu divider

19” Rack #2

Switch Racks stacked next to each other at the T2Cross section of 128 cable fabric going up/down

(256 Total Fabric Cables Required)

Even NIC-TOR – 400G DACs don’t fit well and won’tsee mainstream adoption for 3-4 more years


At 400Gb and 800Gb we need Active Cables• At 400Gb, an active cable (such as a HiWire CLOS AEC):

– Consumes 70% less volume than a passive DAC (1x6mm vs. 1x11.4mm)– Consumes 70% less power than an AOC– Enables routing densities to >500 cables per rack

• At 800Gb, the choice is more stark– Retimed front panel (Transceivers, AOC, AEC)– Switches that support the DAC LR reach will consume 15-20% more power

• For what purpose? Opportunity for ASICs to move to optimized C2M reach & power…– Improbable to route 800Gb DACs at even TOR-NIC densities in a data center

• The time has come to move away from the complexities of managing DAC– Move to fully deterministic and persistent connections– Better TTM and reduced OpEX

33


Active Cable – Testing and Interoperability

• Leverage Ethernet Alliance’s plugfest to ensure interoperability among active cable, switch, server, and test equipment suppliers.

• Leverage industry consortium to ensure simplified application use cases and guarantee plug & play performance.

34


April 16, 2020

DRIVING INTEROPERABILITY THROUGH TEST & MEASUREMENT

Paul Brooks – VIAVI Solutions


Ethernet – Success through interop and flexibility• Ethernet is the dominant solution for high speed interconnect

of communication and computing devices.• With a choice of PMDs to go a few cms over a backplane or

10’s km over fiber• Pluggable optical modules have allowed Ethernet to address

dynamic needs in a flexible ‘pay and adopt’ as you grow approach

• Ensuring the validation, functionality of pluggable optical modules across an open ecosystem has been a key element.

36


Client Pluggable Optics Today

10GECPRI up to 9SFP+

25GECPRI10 SFP28

50GE SFP56

100GE SFP56-DD

100GECFP

100GECFP2

SFP – ‘single lane’

100GECFP4

100GE QSFP28

200GE QSFP56

400GE QSFP56-DD

40GE QSFP+

QSFPx– ‘multi-lane’

• Client optical interfaces are deployed in volumes of millions as pluggable, multi vendor, commoditized products based on robust and clear standards to rates of 400G today.

• The vast majority of the photonic interfaces use direct detect optical links using simple NRZ modulation. From 400G there has been a move to PAM-4 based signalling for both the electrical and optical links.

Standard mechanical form factors and module management and host interfaces allow interchange of modules and ‘pay as you grow’ equipment.

400GE & 800GE OSFP


Inside a 400G QSFP-DD

Double stacked connectorSI issues at 25Gbd PAM-4Mechanical robustness

DSP – host at PAM-4 25Gbd, line side 100G/lambda or ZR coherent

ROSA, TOSA BW, linearity, power, performance

Mechanical integration

Complex module management MIS – real time software to manage events like LOS. Multiple complex parameters like OSNR and PMD to be read-off and checked against alarm levels. Pre-FEC BER to be tracked.

Photonic inter-op & margin

Host AUI-8 inter-op & margin. Equalizer set-up

Thermal – module & system


Critical Areas for Inter-op

Electrical - Host sideOptical - Line side

Data interfaceFEC, eCPRI, Ethernet

Electrical interfacePhy, skew

Management interfaceMIS, I^2C, peek/poke

DSP, controller,

F/W & S/W etc

Photonics

Pluggable module

Mechanical & thermal environment

Optical i/fPMD, optics


Succesful inter-op means an Integrated Environment

The elements required for successful inter-op must be validated together. The close interaction of host (power/thermal/mechanical, data & phy/electrical and the module management are key to the open ecosystem. Also critical is the inter-operability of the optical side. With the emergence of pluggable coherent modules, especially the expected wide scale take up of ZR modules at 400G will cause addition challenges.

PowerCoolingPHY layerEthernet & PCS MISTriggersFECPhotonic


Legacy or State of the art?• A modern host has to consistently

manage and support a wide range of potential clients.

• Take a QSFP-DD slot in a network element– QSFP+ 40G– QSFP28 – 100G– QSFP56 – 200G– QSFP-DD – 400G – Optical or DAC– SFF or CMIS– I^2C speed & termination– Breakout or ?

• A modern host must be able to manage this in a dynamic and consistent manner

• Consistent behaviour under fault conditions like LOS?

41

• What type of module?

• SFF or CMIS• Supported modes• AUI set-up• LOS• Traffic• power


Summary - Ethernet Interop • Pluggable optics give Ethernet the powerful ability to scale and

grow with the right reach (PMD) for each application– From copper DAC to ZR optics for 80km+

• Pluggable optics have evolved significantly from the simple E/O and O/E converters of past generations to highly integrated and complex systems with demanding management and control as well as high speed electrical & photonic interfaces

• All of these need to be orchestrated together to continue the open ‘plug, play & grow’ ecosystem that has made Ethernet the success it is.

42


April 16, 2020

INTEROPERABILITY – THE FOUNDATION OF ETHERNET SUCCESS, A SYSTEMS VENDOR PERSPECTIVE

Ray Nering – Cisco Systems


Industry Relies on Ethernet Interoperability

• Cisco has relied on Ethernet requirements since it first product in 1986

• Today, Cisco’s product line is broad, diverse and still relies on Ethernet requirements

• Any product must be able to interoperate with other vendors to be a relevant player in the market

• Ethernet standards, industry standards organizations (IEEE, etc), MSAs and industry collaboration are the key to maintaining an ecosystem that promotes interoperability

• At 100GE and below Interop wasn’t trivial• PAM4 technology, everything is new: Optics, SERDES, CMIS, form factors, standards


Optical Signaling (module <-> module)Electrical Signaling (module<-> line-card High Speed)

Control Plane (module<-> linecard Low Speed)

Successful implementation on 1-100 GbE client optics, i.e. Switch to Switch, Switch to Router interfacesGenerally within same buiding, campus or data center (<10 km)

Standards (IEEE, MSA, ) try to guarantee interop and uniformity

Pluggable Optics are GREAT


100GbE Pluggable Optic Interoperation • Industry has been releasing platforms w/ 100GbE ports since 2010 timeframe• OEMs have been producing platforms w/ 100GbE ports since 2010

• They have a range of 100G port form factors and all must interoperate optically for all time• CFP, CFP2, CPAK, CFP4, QSFP28 and QSFP-DD…

• A CFP 100BASE-LR4 shipped in 2011 must be able to interoperate with all other 100GBASE-LR4 modules no matter the vendor or the form factor

• Customer may buy a 100G-LR4 and it should work in any platform w/ a compatible port


1, 10, 40 GbE

100 GbE

Equalization on electrical channel

Fixed or adaptive Equalization

Low power management

CDR

Laser turn on

Diagnostic monitoring

Optical and environmental monitoring

Provisioning based on state machine

100GbE was the beginning of more complex host/module interactions

25Gbit/s I/O 3.5 W power

CDR Strong Equal. @ Rx

Complex interactions

with Host


What happens if……Vendor A does not test full compliance and has little margin on equalization settings

On this unit, no errors

On this unit, CRC errors for no apparent reason

Error free area guardband


What happens if……Vendor does test full compiance and has plenty of margin



Error free area guardband


What happens if……

Vendor A has limited acceptance for fast electrical signals from linecard

Error free only in this condition: performance depended on port used

Error free wide range: perfomance consistent across all ports

Vendor B has wide acceptance for fast electrical signals from linecard


Impact on the Transition to PAM4 • Industry turned to PAM4 to increase data rate beyond 25Gb/s• PAM4 is much more sensitive to:

• Noise• Reflections• Non-linearities• Baseline wander

• Receiver design is much more complicated• IEEE has done a great job to embrace PAM4 with standards• 802.3bs/cd/ck/cm/cn/ct/cw/cp/cu….• There will be improvements as we learn more


400 GbE Increases Complexity Further 100 GbE (LR4) 400 GbE (FR4)

IDEAL

MEASURED

POST DSP EQUALIZER

• 400G PAM4 requires DSP and strong equalizations to enable client optics.

• DSPs are not as powerful (and power-hungry) as the ones used on line-side optics.

• Yet interop between different DSPs is a critical item


Tx A Tx B Tx C Tx A Tx B Tx CTx A Tx B Tx C

Rx A Rx B Rx C

400G DSP interop• Interop testing over three 400G vendors• Vendor A and C use same DSP• Vendor B uses different DSP • Tests identified significant issues with

interoperating with anything but itself


Best practice: module level testingOptical

Interoperability

Electrical

Control

• TX• RX

• TX• RX

• Alarms• Monitoring (DOM)

Across full range of

Temperature Voltage DC noise


Best practice: system level testingSoftware Interactions

Module & Fiber Plug /Remove

Power Cycle

Shut/No Shut

Across full range of

Temperature Voltage Platforms


Tier 1 optics supplier

Tier 2 optics supplier

Bareboneoptics

supplier

Basic optical compliance

Full optical compliance

Test over worst case fibers

Full electrical compliance

Basic Monitoring and control compl.

Full Monitoring and control compliance

Full testing/regressionover MCN

Interoperability with other optics

System level testing

Test across all T/V corners

Coverage

Key takeaways

Critical features for seamless interop between modules

on different platform

Faster speeds = More complexity

More sensistive linecards/module interactionsBest

Practice


Conclusion• Ethernet interoperability is literally the foundation of

networking today• Lower speeds (>100GbE, NRZ) interop was straight forward• Higher speeds (50G+) w/ PAM4 signaling interop has become

much more complicated• Interactions between optics, platforms and SW are critical • Corner testing not only at the optical level but also across

platforms & SW are critical for robust interoperability

57


If you have any questions or comments, please email [email protected]

Visit the Ethernet Alliance on Facebook

Ethernet Alliance: visit www.ethernetalliance.orgJoin the Ethernet Alliance LinkedIn groupFollow @EthernetAllianc on Twitter

www.ethernetalliance.org 58

http://ethernetalliance.org

https://www.facebook.com/pages/Ethernet-Alliance/175317229233420

http://www.ethernetalliance.org/

http://us.linkedin.com/company/ethernet-alliance

Documents

INTEROPERABILITY –THE FOUNDATION OF ETHERNET …...Apr 16, 2020 · Ethernet –Success through interop and flexibility •Ethernet is the dominant solution for high speed interconnect