© 2002 Foundry Networks, Inc.
Técnicas de Alta Disponibilidade para NAPs
Marcelo Molinari – Foundry Networks do [email protected]
©2002 Foundry Networks, Inc.
Agenda
• LINX topology overview
• AMS-IX topology overview
• Metro Ring Protocol
• Virtual Switching Redundancy Protocol
© 2002 Foundry Networks, Inc.
London Internet Exchange (LINX)
©2002 Foundry Networks, Inc.
LINX Topology
©2002 Foundry Networks, Inc.
LINX Topology• The LINX Network consists of two separate high-
performance Ethernet switching platforms installed across seven locations.
• Switches from two equipment vendors are deployed in two separate networks to provide an extra level of fault-tolerance, the logic being that both systems shouldn't develop the same fault at the same time.
©2002 Foundry Networks, Inc.
LINX Topology
• Two switches are installed in every LINX location, and the locations are interconnected by multiple 10 gigabit Ethernet circuits to form two physically separate backbone rings.
• Most LINX members connect to both switching platforms, which reduces the impact of any downtime on a single network element.
• Management of the logical redundancy of the network is done using MRP (Metro Ring Protocol). In the event of the loss of a network segment, MRP activates a redundant link within tenths of a second and restore connectivity.
©2002 Foundry Networks, Inc.
LINX Aggregated Traffic Statistics
© 2002 Foundry Networks, Inc.
Amsterdam Exchange (AMS-IX)
©2002 Foundry Networks, Inc.
AMS-IX Topology
©2002 Foundry Networks, Inc.
AMS-IX Topology• AMS-IX is a distributed exchange, currently present at five
independent co-location facilities in Amsterdam.
• The AMS-IX topology is built around two hub/spoke arrangements.
• The core switches are Foundry Networks NetIron MLX-32 switches.
• Members connected with GigE, 100Base-TX or 10Base-T ports are connected to Foundry Networks BigIron 15000 and BigIron RX-8 switches.
• Members with a 10GbE port are connected to Glimmerglass Networks photonic cross-connects. These L1 switches connect the member 10GbE ports to BigIron RX-16 or NetIron MLX-16 switches.
©2002 Foundry Networks, Inc.
AMS-IX Topology• The two core switches run VSRP (Virtual Switch
Redundancy Protocol) to define the active hub/spoke and to automatically fail over to the other, based on pre-defined triggers (e.g. link failure).
• All edge switches from Foundry follow VSRP automatically. The Glimmerglass switches follow the VSRP failover based on software developed at AMS-IX.
• Members can connect to the AMS-IX infrastructure at any of the five AMS-IX co-locations, at 100 Mbit/s, 1 Gbit/s or 10 Gbit/s.
©2002 Foundry Networks, Inc.
AMS-IX Traffic Statistics
© 2002 Foundry Networks, Inc.
Metro Ring Protocol
©2002 Foundry Networks, Inc.
Metro Ring Protocol (MRP)
• Metro Ring Protocol is a Layer 2 protocol designed to provide SONET-like, high speed, fault tolerant, fast recovery for Metro Ethernet networks.
• MRP SONET-like features provide: – Sub-second failover– Efficient use of bandwidth with topology groups (802.1s
based)– Scalable protection for multiple VLANs– Large Scale L2 MANs with multi-ring support– Highly flexible network designs– Works with other L2 features
• Runs on all Ethernet and PoS/SDH interfaces, including 10 Gigabit
©2002 Foundry Networks, Inc.
How it works
• A single node is defined as the Ring Master Node
• All other nodes are defined as Ring Member Nodes
• The Master Node prevents loops by blocking its secondary port
• Ring Hello Packets are generated by the Master Node to check ring integrity
• As long as the master sees its own Hello packets on the secondary port, ring health is verified, and secondary port remains blocked
Secondary (Blocking)
Primary (Forwarding)
RHPRHP
RHPRHP
©2002 Foundry Networks, Inc.
Rapid Failover• Hello packets are hardware
forwarded by the nodes in the ring to ensure fastest failure detection.
• Master considers ring broken if no Hello packets are received within 300 ms (3 consecutive Hellos are lost).
• If no Hellos are received, Master transitions secondary port into forwarding state to restore ring connectivity.
• To provide reliable flushing of stale MAC entries, Master sends 3 consecutive TCN notifications.
• By changing timers and using messages sent by the node where the ring broke, it is possible to achieve recovery times from 150 ms to 200 ms.
RHP
NetIron 400
NetIron 400
NetIron 400 NetIron 400
RHP
RHPRHP
TC
NetIron 400
NetIron 400
NetIron 400 NetIron 400
TC
FAULT
TC
©2002 Foundry Networks, Inc.
Link RestorationPreventing Temporary Loops• When an MRP port goes up, it goes
into pre-forwarding mode to avoid the creation of temporary loops.
• In pre-forwarding mode the port forwards no data, but only the ring hello packets from the master.
• Master sees its own RHP, detects that ring integrity has been restored, puts its secondary port in blocking mode.
• From that point onwards, master sends RHPs with the Forwarding flag bit set, indicating that members should transition their ports from pre-forwarding to forwarding.
• The Forwarding flag bit is always set, as long as the master is blocking its secondary port.
NetIron 400
NetIron 400
NetIron 400 NetIron 400
RHPRHP
NetIron 400
NetIron 400
NetIron 400 NetIron 400
Link restored
RHPRHP
RHPRHP
PF
PF
RHPRHP
F
F
©2002 Foundry Networks, Inc.
Topology Groups
Topology Group
Master VLAN
Member VLAN
Member VLAN Group
A VLAN running a control protocol (or more) that controls the active topology for the whole topology group. Control protocols: STP, RSTP, MRP, VSRP.
A VLAN running NO control protocol of its own but rather follows the active topology of the master VLAN..
A group of VLANs running NO control protocol of their own but rather follow the active topology of the master VLAN. VLAN groups are defined via the “vlan-group” command.
©2002 Foundry Networks, Inc.
Efficient use of Ring Bandwidth• MRP Supports Multiple Topology groups within a Ring
• An MRP Node can be both a Master node and Member Node for different topology groups
• Each Topology group contains a Master VLAN and Member VLANs
• Master VLANs generate Hello packets and block secondary ports
• 4094 VLANs can be divided among up to 255 Topology groups
VLAN Group 2
All links are utilized!VLAN Group 1
2
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
1
TopoGroup 1
TopoGroup 2
©2002 Foundry Networks, Inc.
Using Multiple Rings• There are 3 ring scenarios:
• Single ring
• Rings that don’t overlap
• Overlapping rings that share links
• Each Ring runs its own instance of MRP
• A ring node can be Master for Multiple rings
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000BigIron 4000BigIron 4000
BigIron 4000BigIron 4000 BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
BigIron 4000BigIron 4000
Single Ring Non-Overlapping Ring Overlapping Rings
Phase I
Phase II
©2002 Foundry Networks, Inc.
Example Scenarios – Phase I
– High-speed 10 GE trunks for Metro rings or IXPs
– Provides sub-second fault-detection and fail-over
– Superior scalability: no limit on maximum number of nodes per ring
– Counter rotating topology groups provide efficient use of bandwidth
BigIron 4000
BigIron 8000
BigIron 4000
BigIron 4000
BigIron 8000
BigIron 4000
S5
S6
S1
S4
S3
S2
MasternodePrimary
port
Secondary port
BigIron 4000
BigIron 4000
BigIron 4000
BigIron 4000
BigIron 4000
Ring1
Ring4
Ring2
Masternode
BigIron 4000
BigIron 4000
S5
S4
Ring3
©2002 Foundry Networks, Inc.
Example Scenarios – Phase II
– Shared ring support– Increased reliability – Increased bandwidth
BigIron 4000
BigIron 8000
BigIron 4000
BigIron 4000
BigIron 8000
BigIron 4000
S5
S6
S1
S4
S3
S2
Masternode
Primary port
Secondary port
BigIron 4000
BigIron 4000
BigIron 4000
BigIron 4000
BigIron 4000
BigIron 4000
Ring1
Ring4
Ring2
Masternode
Masternode
BigIron 4000
BigIron 4000
S5
S4
Ring3
©2002 Foundry Networks, Inc.
Interface Flexibility
BigIron 4000
BigIron 8000
BigIron 4000
BigIron 4000
BigIron 8000
BigIron 4000
S5
S6
S1
S4
S3
S2
Masternode
Primary port
Secondary port
BigIron 4000
BigIron 4000
BigIron 8000
S5
S6
S4
BigIron 4000
S3
Masternode
Primary port
Secondary port
BigIron 8000
S2
BigIron 4000
S1
• Support for Mixed interfaces- 10Gig & Gig- Gig & 10/100- 10Gig & PoS/SDH
Slower link
• Support for Trunked interfaces- 10Gig & Gig- PoS/SDH
©2002 Foundry Networks, Inc.
MRP – Summary of Benefits
• Fast, sub-second, predictable fail-over functionality
• Maximizes ring bandwidth utilization
• Cost effective scalable solution for MAN resiliency
• Attractive alternative to STP
• Utilizes 802.3 Ethernet Packet standards and MACs
• Can be combined with other Foundry features to provide complete end to end MAN designs
© 2002 Foundry Networks, Inc.
Virtual Switch Redundancy Protocol
©2002 Foundry Networks, Inc.
Virtual Switch Redundancy Protocol• VSRP provides an alternative to Rapid Spanning Tree
Protocol (RSTP) in dual homed/mesh configurations, providing sub-second fail-over and recovery.
• VSRP features provide: – Sub-second fail-over– Efficient use of mesh bandwidth – no blocked links– Block and unblock ports at the per-VLAN group level– Large scale L2 MANs with multi-tiered support– Highly flexible network designs– Configurable tracking options– Works with other L2 features
• Works with all Ethernet interfaces, including 10 Gigabit
• VSRP is based on VRRP-E & can provide L2 and L3 backup
©2002 Foundry Networks, Inc.
How it works
BigIron 8000
S1
BigIron 4000
S3
BigIron 4000
S5
VSRPMaster
• VSRP uses an election process to select a Master switch and up to 4 backup switches for each VLAN: higher configured priority wins; if equal, higher IP address wins.
• Only the Master switch forwards data, while Backup switches block traffic on all VSRP configured interfaces within the VLAN (or the topology group).
• Master switch sends Hello packets to all backup switches
• Switches do not have to be VSRP aware. VSRP aware provides faster failover.
• VSRP can track ports and decrease the priority of VSRP active switch in case a tracked port goes down.
BigIron 4000
S4
S2
VSRPAware
Hello
Hello
FBBFF B
BigIron 8000
VSRPBackup
©2002 Foundry Networks, Inc.
Rapid Failover• A VSRP Backup switch monitors Hellos
from the Master.
• If no Hellos are received for Master Dead Interval (default 300 ms), Backup goes into Hold Down state, starts sending periodic Hellos.
• Hold Down interval is by default 300 ms, and it allows for the election of a new master.
• If the switch is elected as Master, it sets its port into forwarding state, sends 3 TCNs.
• A VSRP aware switch receives TCN, and looks for the new master. Hellos of the new master will be received on a different port.
• A VSRP aware switch shifts the MAC addresses learned on the failed port to the new port.
Mac Type:D = Dynamic, S = Static, H = Host, R = Router
MAC Port Age Type
0060.f320.23a8 2 0 D
0030.1b07.0694 2 0 D
00d0.b758.88dc 2 0 D
0004.8039.5f00 2 0 D
NetIron 400
NetIron 800 NetIron 800 NetIron 800
BF B
Master Backup Backup
Hello
HelloHe
llo
NetIron 400
NetIron 800 NetIron 800 NetIron 800
BDn F
Master Backup
Hello
FAULT
FAULT
Hello
©2002 Foundry Networks, Inc.
Link RestorationSwitching Back to Original Master• When the failed link is restored, the
original Master remains as a Backup.
• Original Master receives inferior Hello from the current Master, so it immediately replies with its own Hello, switches into Hold Down state (300 ms), starts sending periodic Hellos
• Current Master receives superior Hello, so it switches into Backup mode.
• If no superior Hellos are received during Hold Down interval, original Master considers itself the new current Master, and sets its port in Forwarding mode.
• New Master sends out 3 TCNs.
• A VSRP aware switch receives TCN, and looks for the new master. Hellos of the new master will be received on a different port.
Mac Type:D = Dynamic, S = Static, H = Host, R = Router
MAC Port Age Type
0060.f320.23a8 1 0 D
0030.1b07.0694 1 0 D
00d0.b758.88dc 1 0 D
0004.8039.5f00 1 0 D
NetIron 400
NetIron 800 NetIron 800 NetIron 800
BB F
Master(Orig. Master)
Backup Backup
Hello
Hello
NetIron 400
NetIron 800 NetIron 800 NetIron 800
BF B
Master Backup
Hello HelloHello
Hello
Backup
Link restored
Hello
Hello
Hello
• A VSRP aware switch shifts the MAC addresses to the new port.
©2002 Foundry Networks, Inc.
Efficient use of Uplink Bandwidth
BigIron 8000BigIron 8000
S1
BigIron 4000
S3
BigIron 4000
S5
Master topology group 1Backup topology group 2
HelloPackets
BigIron 4000
S4
S2
Master topology group 2Backup topology group 1
VSRPAware
Topology group 1 = Master VLAN 1Member VLANs 2 to 2048
Topology group 2 = Master VLAN 2049Member VLANs 2050 to 4096
• VSRP supports topology groups to fully utilize switches and links
• Topology groups are a collection of VLANs
• Each yopology group contains a Master VLAN and Member VLANs
• VSRP configured switches can be Master for some topology groups while backup other for others
• 4094 VLANs can be divided among up to 255 Topology groups
©2002 Foundry Networks, Inc.
VSRP Domains• VSRP can be configured in
separate domains within the same VLAN to allow for larger topologies.
• Topology groups can be designed to use unique paths in each domain.
• A TTL value within the VSRP Hello packet controls how deep the packet goes into the network.
• TTL is being decremented by 1 at each VSRP aware switch.
• Default TTL is 2, which allows Hello to traverse one VSRP aware switch to go to another VSRP active switch.
NetIron 400
NetIron 800 NetIron 800 NetIron 800
NetIron 400
NetIron 400 NetIron 400
NetIron 400 NetIron 400
VSRP Aware
VSRP Active
VSRP Active
VSRP Active
VSRP Aware
VSRP Aware
VSRP Domain 1(VRID 1)
VSRP Domain 2(VRID 2)
VSRP Domain 3(VRID 3)
©2002 Foundry Networks, Inc.
Intelligent Port Level Control
• VSRP can be configured to run only on designated ports
• Only VSRP configured ports are placed in blocking for Backup switches
• Supports host attachments and dual port NICs: host facing ports are configured as VSRP free.
• Works in combination with MRP to provide flexible Metro / Enterprise Ethernet designs.
NetIron 800 NetIron 800
MRPMaster Backup
VSRP AwareDual Homed Servers
NetIron 400 NetIron 400
NetIron 800 NetIron 800
©2002 Foundry Networks, Inc.
VSRP-Aware Switches
• Both VSRP-Aware and Non VSRP-Aware switches can be used as edge devices
• VSRP-Aware switches recognize the VSRP Hello packet sent by the Master, and then create an table which contains the VRID of the VLAN which sent the VSRP Hello plus the incoming port where the VSRP Hello was received.
• When the VSRP-Aware switch sees a Hello packet coming from a different port, it quickly moves the MAC address table entries to the new port.
BigIron 8000
S1
BigIron 4000
S3
BigIron 4000
S5
VSRPMaster
BigIron 4000
S4
S2
VSRPAware
Hello
Hello
FBBFF B
BigIron 8000
VSRPBackup
VSRPAware
VSRPAware
©2002 Foundry Networks, Inc.
Non VSRP-Aware Switches
• Non VSRP-Aware switches can be used as edge devices, but they do not recognize VSRP Hello packets.
• MAC entries will age out or they will eventually be learned from the new port.
• This results in slow convergence when the Master fails.
• Solution is to configure “VSRP Fast Start” in the VSRP Master and Backup nodes.
• VSRP Fast Start disables and re-enable the ports before transitioning from Master to Backup. This causes a MAC address flush in the edge devices which makes convergence faster.
BigIron 8000
S1
BigIron 4000
S3
BigIron 4000
S5
VSRPMaster
BigIron 4000
S4
S2
Non VSRPAware
FBBFF B
BigIron 8000
VSRPBackup
Non VSRPAware
Non VSRPAware
©2002 Foundry Networks, Inc.
VSRP – Summary of Benefits
• Fast, sub-second protection without Spanning Tree
• Combines both switching and routing redundancy– Provides default gateway redundancy if
needed
• Supports topology groups for full link utilization
• Can be combined with other Foundry features to provide complete end to end MAN designs
©2002 Foundry Networks, Inc.
References
• LINX - https://www.linx.net/
• AMS-IX - http://www.ams-ix.net/
• “I can feel your traffic” - http://www.jasinska.de/
• MRP- http://www.foundrynet.com/pdf/wp-mrp.pdf
• VSRP - http://www.foundrynet.com/pdf/wp-vsrp.pdf
© 2002 Foundry Networks, Inc.
Thank You !Marcelo Molinari – Foundry Networks do Brasil