Backplane Technology Overview for AdvancedTCA

B k l T h l O i Backplane Technology Overview for AdvancedTCA

Theresienstr. 9080290 MunichGermany

Hui ChenSupervisor: Rainer Ohlendorf

www.lis.ei.tum.deJuly, 13th 2007

© Institute forIntegrated Systems

Outline

Introduction to AdvancedTCAI t t P t lInterconnect Protocols① Ethernet② InfiniBand③ StarFabric④ PCI Express⑤ RapidIO

Comparison of Interconnect Protocols① Feature② Protocol Efficiency② Protocol Efficiency③ Effective Bandwidth④ System Level Consideration

C l i


Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 2

Conclusion

What is AdvancedTCA ?

• AdvancedTCA (Advanced Telecom Computing Architecture)Standard chassis/ backplane solution for high-availability communications equipmentAdopted in 2003

• AdvancedTCA Chassis



Base Spec of AdvancedTCA

• Board dimension1.2 inches in width 280 mm deep, ~322 mm high

• Power supply & Cooling-48 V power feeds200 Watts per board

Board of AdvancedTCA

• Backplane architectureZ 1 P M tZone 1: Power, ManagementZone 2: Data Transport InterfaceZone 3: User-defined I/O



Backplane of AdvancedTCA

Data Transport Interface

• Base Interface: Dual StarSupports only Ethernet

• Fabric Interface: Dual Star/ Full MeshSupports various protocolsMax. 4 channelsMax. 4 channels

• Dual StarRedundant switch: eliminates single point of

(a) Dual Star configuration

Redundant switch: eliminates single point of failureEffective for large redundant systems

F ll M h• Full Mesh+) Higher data throughput-) Higher pins count & costEff ti f ll t



Effective for small systems(b) Full Mesh configuration

Outline

Introduction to AdvancedTCA

Interconnect ProtocolsInterconnect Protocols① Ethernet② InfiniBand③ StarFabric④ PCI Express⑤ RapidIO⑤ RapidIO

Comparison of Interconnect Protocols① Feature② Protocol Efficiency③ Effective Bandwidth④ System Level Consideration



Conclusion

Protocol ①― Ethernet

• Application:Initial in Local Area NetworkSuccessful in backplanebut defective in congestion management

• Congestion managementCongestion managementPAUSE function: all data flows are paused when only one flow is congested→ congestion spreading, when complex data pathsg p g, p p

• TopologyP2P connections: Ethernet Switch ↔ Endpoints Ethernet Topology

• Frame



Header: large

Protocol ②― InfiniBand ⑴

• ApplicationInitial in System Area Network

l i b k l h i t h i

Processor Node

now also in backplane, chassis-to-chassisSupercomputer, Linux cluster

• Topology• TopologyInfiniBand Switches + Channel AdaptersHCA: Host Channel Adapter in processor node

TCATCA: Target Channel Adapter in I/O nodeHCA talks TCAs to enable remotestorage and connectivity InfiniBand Topology

I/O Node

• Packet format



Larger payload size than Ethernet

Protocol ②― InfiniBand ⑵

• Three link widths x1

x4Links x1 x4 x12 x4

x12

Links x1 x4 x12

Data Rate (Gb/s) 2 8 24

• Supports Quality of Service (QoS)QoS: a method to prioritize network traffic

Ensures most important data gets thru the network as quickly as possibleEnsures most important data gets thru the network as quickly as possibleDefines traffic to many streams, streams to differentiated classes;reserves bandwidth to streams and classes



Protocol ② ― InfiniBand ⑶

• Supports Quality of ServiceThru 16 Virtual Lanes (VLs)

Multiplex independent data streams → the same physical link

Represent a set of send & receive pbuffers in a port

Each port: one management VL (VL15) & up to 15 data VLs

Priorities

VL15 (highest)

Virtual Lanes

VL14

VL0

…© Institute for

Integrated SystemsHauptseminar am Lehrstuhl für Integrierte Systeme

Hui Chen, July 13th 2007 10

Protocol ③ ― StarFabric ⑴

• ApplicationBackplane, chassis-to-chassisEmbedded distributed processing(e.g. multiprocessor systems)

• TopologyStarFabric Switch: six-port

Connects other switches/ bridges

StarFabric TopologyCascadable

StarFabric BridgeTranslates other protocols (e.g. PCI) intoSt F b i t ffiStarFabric traffic

• Not scalableO t l t 2 Gb/



Operates only at 2 Gb/s

Protocol ③ ― StarFabric ⑵

• Packet format

• Supports Quality of Service7 classes of traffic7 classes of traffic

asynchronous traffic (data)multicast…isochronous traffic (voice & video)

• Backwards compatible to PCIConverting a serial fabric to parallel PCI



Protocol ④ ― PCI Express ⑴

• Successor to PCIBackplane, chip-to-chip interconnectPC (e.g. Graphics cards)

• TopologySwitchesMultiple Endpoints: I/O devicesHost Bridge

To create an I/O hierarchy

• Six link widths:PCI Express Topology

Links x1 x2 x4 x8 x16 x32

Data Rate (Gb/s) 2 4 8 16 32 64



Protocol ④ ― PCI Express ⑵

• Packet format

• Supports Quality of ServiceThr 8 Traffic Classes (TCs) & 8 Virt al Channels (VCs)Thru 8 Traffic Classes (TCs) & 8 Virtual Channels (VCs)

Each port : up to 8 VCs. VC0 (lowest priority), VC7 (highest)

TCs: differentiated servicesE h k t TC t d t i hi h VC b ff t d k t i tEach packet a TC: to determine which VC buffer to drop packet into

TC/VC mapping: ☺ TC[0:6]→VC0 TC7→VC[0:1]



An Example of TC/VC mapping

Protocol ⑤― RapidIO ⑴

• ApplicationBackplane, chip-to-chip interconnectEmbedded systems (e.g. DSPs)Networking (e.g. Carrier Cards)

T l• TopologyA Switch, multiple EndpointsLarge systems: one / more host processors

For system exploration initializationFor system exploration, initializationMonitors system-level activity

• Two link widths (x1 x4)RapidIO Topology

Two link widths (x1, x4)

Links x1 x4

Data Rate (Gb/s) 1 2 2.5 4 8 10



Protocol ⑤― RapidIO ⑵

• Packet format

Header: smaller than Ethernet:1-byte source address ~ 28 Endpoints → sufficient for backplane applicationsPayload size: relatively small

• Supports Quality of ServiceSix logical flows: as prioritized classes



Outline


Interconnect ProtocolsInterconnect Protocols① Ethernet② InfiniBand③ StarFabric③ StarFabric④ PCI Express⑤ RapidIO

Comparison of Interconnect Protocols① Feature② Protocol Efficiency② Protocol Efficiency③ Effective Bandwidth④ System Level Consideration



Conclusion

Comparison – Feature

Feature GigE InfiniBand StarFabric PCI Expr RapidIOFeature GigE InfiniBand StarFabric PCI Expr. RapidIO

Transport Media

PCB, Copper & Fiber

PCB, Copper & Fiber

PCB &CAT5 cable

PCB, Copper & Optical PCB only

Scalable Link Widths None Yes None Yes Yes

PCI Migration None Encapsulated, requires SW

Yes, transparent

Yes, transparent

Yes,requires SWq p p requires SW

Max Signal Channel Rate 1 Gbps 2 Gbps 2 Gbps 2 Gbps 2.5 Gbps

Max Payload 1500 B tes 4096 B t 128 B t 4096 B t 256 B tesySize 1500 Bytes 4096 Bytes 128 Bytes 4096 Bytes 256 Bytes



Comparison – Protocol Efficiency

• Protocol Efficiency = Payload / Packet size

• Payload < ~500B : RapidIO, highest efficiency smallest overheadPayload > ~500B : PCI Express highestPayload > ~500B : PCI Express, highest



Comparison – Effective Bandwidth

• Effective Bandwidth = Protocol efficiency × Bandwidth

• InfiniBand x4; PCIe x4; RapidIO 4x 2Gbps

O ll Gi E l t• Overall: GigE, lowest



Comparison – System Level

• Quality of Service (QoS)

Gi E I fi iB d St F b i PCI E R idIOGigE InfiniBand StarFabric PCI Expr. RapidIO

Prioritized Traffic Classes － 16 7 8 6

P f• Performance

GigE InfiniBand StarFabric PCI Expr. RapidIO

M B d idth fMax Bandwidth for AdvancedTCA, x4 1 Gbps 8 Gbps 2 Gbps 8 Gbps 10 Gbps

Latency High (~100 us)

Medium (1.29-2.6 us)

Medium (< 3 us)

Low (0.3-0.7 us)

Low (< 100 ns)



Outline


Interconnect ProtocolsInterconnect Protocols① Ethernet② InfiniBand③ StarFabric③ StarFabric④ PCI Express⑤ RapidIO

Comparison of Interconnect Protocols① Feature② Protocol Efficiency③ Effective Bandwidth④ System Level Consideration

Conclusion



Conclusion

Conclusion

• Low bandwidth backplane applicationsEthernet +) well understood, low risk

) hi h h d hi h l-) high overhead, high latency-) not scalable

• High bandwidth (tradeoffs of flexibility vs. latency, overhead)g ( y y, )InfiniBand and StarFabric

Complex header: +) for complex routing-) reduces effective bandwidth

RapidIOSmaller packets: +) to lower latency

-) overhead: a higher percentage of the packet) d ff ti b d idth-) reduces effective bandwidth

PCI ExpressMiddle: +) design goal: software transparency

-) constrains the protocol raises latency



-) constrains the protocol, raises latency

Thank you for your attention !

Any questions ?

Special thanks to:

Rainer Ohlendorf



Education

Backplane Technology Overview for AdvancedTCA