Designing a Layer 3 Data Center Network with the QFabric … · 2011-09-06 · Interface Naming Conventions for QFabric Architecture ... Designing a Layer 3 Data Center Network with

IMPLEMENTATION GUIDE

Copyright © 2011, Juniper Networks, Inc. 1

DEsIGNING A LAyEr 3 DATA CENTEr NETwOrk wITh ThE QFAbrIC ArChITECTUrE how to build a Data Center Network with QFabric Products Acting as a Layer 3 switch

Although Juniper Networks has attempted to provide accurate information in this guide, Juniper Networks does not warrant or guarantee the accuracy of the information provided herein. Third party product descriptions and related technical details provided in this document are for information purposes only and such products are not supported by Juniper Networks. All information provided in this guide is provided “as is”, with all faults, and without warranty of any kind, either expressed or implied or statutory. Juniper Networks and its suppliers hereby disclaim all warranties related to this guide and the information contained herein, whether expressed or implied of statutory including, without limitation, those of merchantability, fitness for a particular purpose and noninfringement, or arising from a course of dealing, usage, or trade practice.

2 Copyright © 2011, Juniper Networks, Inc.

IMPLEMENTATION GUIDE - Designing a Layer 3 Data Center Network with the QFabric Architecture

TableofContents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Design Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

QFabric basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Node Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

QFabric Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Defining Node Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Example 1: sNG configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Example 2: rsNG configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Example 3: NNG configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Interface Naming Conventions for QFabric Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Interface Type Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Access Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Trunk Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

routed Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Layer 3 LAG Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

VLAN Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Trunk Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Design Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Connecting Layer 3 Device to QFabric Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

route Lookup and Forwarding Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

QFabric and VrrP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

Layer 3 Design Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Use Case 1: static Default route Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Use Case 2: Putting QFabric Architecture into an OsPF Area. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

Use Case 3: Putting QFabric Architecture into OsPF stub Area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Use Case 4: Connecting One-Armed srX series Device as Active/Active with QFabric Architecture . . . . . . . . . . . . . . . 22

Use Case 5: Connecting One-Armed srX series as Active/backup with QFabric Architecture . . . . . . . . . . . . . . . . . . . . . 25

Use Case 6: Connecting One-Armed srX series Gateway to QFabric Architecture (VrF-based steering Mode) . . . 28

Use Case 7: QFabric Architecture back-to-back Extension with L3 LAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

About Juniper Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31



TableofFigures

Figure 1: Juniper’s data center solution with QFabric architecture, MX series, srX series,

vGw Virtual Gateway, and Junos space. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Figure 2: QFabric logical and physical configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Figure 3: LAG support between node groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Figure 4: Different types of redundancy for rack servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Figure 5: Different deployment scenarios with embedded blade switches in blade chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Figure 6: Layer 3 devices can be located anywhere in the QFabric architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

Figure 7: NNG connecting to MX series with LAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Figure 8: QFabric technology in OsPF area0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19

Figure 9: srX series one-armed deployment in a two-tier architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Figure 10: One-armed srX series active/active deployment with QFabric technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Figure 11: One-armed srX series active/active deployment with QFabric architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Figure 12: Applying security policy to inter-VrF routing on QFabric architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Figure 13: back-to-back extension with LAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29



Introduction

As people become more adept at employing virtualization technologies, and as applications become more efficient,

the need for a high-performance and scalable data center infrastructure becomes increasingly critical. Today’s data

center network architecture has too many layers and is too rigid to meet those requirements. Juniper has developed a

new technology called Juniper Networks® QFabric™ architecture that addresses the inefficiencies of legacy data center

networks. QFabric technology eliminates network complexity by reducing the number of switch layers and managed

devices, while providing optimal network utilization and a pay-as-you-grow model that doesn’t compromise overall

network performance.

Scope

This document will discuss the design of a data center network where QFabric architecture acts as the Layer 3 switch.

It will describe the overall network topology and provide relevant configuration templates for QFabric solutions.

The target audiences for this document are architects, network engineers or operators, and individuals who

require technical knowledge, although every effort has been made to make this document appeal to the widest

possible audience. It is assumed that the reader is familiar with Juniper Networks Junos® operating system and is

knowledgeable about the QFabric family of products. Also, reading the “Designing a Layer 2 Data Center Network with

the QFabric Architecture” implementation guide is highly recommended.

DesignConsiderations

One of the biggest challenges with today’s data center is keeping the network simple while enabling it to grow without

making uncomfortable trade-offs. Adding new switches is the typical response to network growth, but that means more

devices to manage and, more importantly, a potentially negative impact on network performance due to switch locations.

Juniper Networks has introduced QFabric technology to address these challenges. QFabric technology has the unique

ability to reduce complexity by flattening the network to a single tier, providing any-to-any connectivity that ensures

every device is no more than a single hop away from any other device. Increasing port counts with QFabric architecture

does not increase complexity or add devices to manage, since all QFabric solution components are managed as a

single device.



Figure1:Juniper’sdatacentersolutionwithQFabricarchitecture,MXSeries,SRXSeries,vGWVirtualGateway,andJunosSpace.

QFabricBasics

Juniper Networks QFabric architecture is composed of three components: QFabric Director, QFabric Interconnect, and

QFabric Node. Each component plays a vital role. The QFabric Director functions as a routing Engine (rE) in a modular

switch, where it is responsible for managing the overall QFabric system as well as distributing forwarding tables to the

QFabric Nodes and QFabric Interconnects. The QFabric Interconnect is equivalent to a fabric, acting like the backplane

of the switch and providing a simple, high-speed transport that interconnects all of the QFabric Nodes in a full-mesh

topology to provide any-to-any port connectivity. The QFabric Node is equivalent to a line card, providing an intelligent

edge that can perform routing and switching between connected devices.

Figure2:QFabriclogicalandphysicalconfiguration

MX SeriesRemoteData Center

SRX Series

SRX5800

Servers NAS FC Storage

VMware vSpherevGW

QFabric Interconnect

QFabric Director

CPE

QFabric Node #1

QFabric Node #2

QFabric Node #3

QFabricNode #128

• • •



NodeGroups

A node group is nothing more than an abstraction of a single or set of QFabric Nodes that are logically grouped with

similar attributes. Node groups are not bound by physical location but by common traits. There are three different types

of QFabric Nodes: server node group (sNG), redundant server node group (rsNG), and network node group (NNG).

• sNG is a single QFabric Node that is connected to servers, blade chassis, and storage devices (it may also be

referred to as host-facing ports). Typically, host devices require a subset of protocols1 such as Link Aggregation

Control Protocol (LACP) and Link Layer Discovery Protocol (LLDP). Therefore, sNGs will only need to support host

type protocols. Layer 2 or Layer 3 networking protocols2 such as spanning Tree Protocol (xsTP) and OsPF are not

supported and cannot be configured on sNG ports.

• rsNG is similar to sNG with a couple of differences. First, an rsNG requires two QFabric Nodes to be grouped.

second, it can support cross-member (node) link aggregation groups (LAGs), as shown in Figure 3.

• NNG is a set of QFabric Nodes connected to wAN routers, other networking devices, or service appliances such

as firewalls or server load balancers. because such devices will be connected to an NNG, all protocol stacks are

available on these ports. The QFabric architecture requires at least one QFabric Node to be a member of an NNG

(up to eight devices are allowed). while defined as an NNG, it does not limit connections to service appliances or

networking devices; server and/or storage devices can also connect to an NNG.

Figure3:LAGsupportbetweennodegroups

Table1:NodeGroupsSupportMatrix

NoDeGRoupS MAX.NuMBeRoFMeMBeRSpeRNoDeGRoup

MAX.NuMBeRoFNoDe

GRoupSWIThINTheQFABRIC

ARChITeCTuRe

SAMeMeMBeRLAG

CRoSS-MeMBeRLAG(ACTIVe/

ACTIVe)

SuppoRThoST-FACINGpRoToCoLS3

SuppoRTNeTWoRkING-

FACINGpRoToCoLS4

single node

group (sNG)1 127 3 3

redundant

server node

group (rsNG)

2 63 3 3 3

Network node

group (NNG)8 1 3 3 3 3

QFabricConfiguration

This document will not go over the deployment or bring-up of the system. It is assumed that the QFabric architecture

has already been brought up by a certified specialist and is ready to be configured. This section will cover how to define

node groups and how to configure port types (access or trunk), VLANs, LAGs, and VLAN membership.

All management and configuration is done through the QFabric Director. There is no need to go into individual QFabric

devices and configure them. The entire QFabric architecture can be managed from a single IP address that is shared by

the QFabric Directors.

1 host-facing protocols are LLDP, LACP, Address resolution Protocol (ArP), Internet Group Management Protocol (IGMP) snooping, Data Center bridging (DCbx).2 Network-facing protocols are xsTP, OsPF, L3 unicast and multicast protocols, and IGMP.3 host-facing protocols are LLDP, LACP, ArP, IGMP snooping, DCbx.4 Network-facing protocols are xsTP, L3 unicast and multicast protocols, and IGMP.

SNG

QFabricNode

QFabricNode

RSNG NNG

QFabricNode

QFabricNode

QFabricNode

QFabricNode



DefiningNodeGroups

Node groups are a new concept for the Junos operating system and are only relevant to QFabric technology. Therefore,

a new stanza has been introduced to help manage QFabric Nodes and node groups. by default, all QFabric Nodes are

identified by serial number. serial numbers can be easily managed with a spreadsheet, and it is not humanly possible

to manage without one. QFabric Nodes can be aliased with a more meaningful name, such as the physical location of

the QFabric Node (row and rack), as shown with the example below.

[edit fabric]netadmin@qfabric# set aliases node-device ABCD1230 row1-rack1

Just as in configuration mode, “fabric” has been introduced into the operational command to provide QFabric

architecture-related administrative “show” commands. below is an example of a serial number-to-alias assignment.

The Connection and Configuration columns provide the current state of the QFabric Node.

netadmin@qfabric> show fabric administration inventory node-devices Item Identifier Connection ConfigurationNode devicerow1-rack1 ABCD1230 Connected Configured row1-rack2 ABCD1231 Connected Configured row1-rack3 ABCD1232 Connected Configuredrow21-rack1 ABCD1233 Connected Configured

QFabric Nodes—even single devices—need to be assigned to a node group. Any arbitrary name can be assigned to an

xsNG. NNG is the exception to this rule, as it already has a name (Nw-NG-0) which cannot be changed. A QFabric

Node can only be part of one node group type; it cannot be part of two different node groups.

Typically members within node groups are close in proximity, but that is not a requirement. Members of a node group

can be in different parts of the data center.

example1:SNGconfiguration

[edit fabric]netadmin@qfabric# set resources node-group SNG-1 node-device row1-rack1

example2:RSNGconfiguration

[edit fabric]netadmin@qfabric# set resources node-group RSNG-1 node-device row1-rack2netadmin@qfabric# set resources node-group RSNG-1 node-device row1-rack3

Note: Up to two QFabric Nodes can be part of an rsNG.

example3:NNGconfiguration

[edit fabric]netadmin@qfabric# set resources node-group NW-NG-0 network-domainnetadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack1

Note: Up to eight QFabric Nodes can be part of an NNG.



A corresponding “show” command, shown below, provides overall node group membership and status.

netamdin@qfabric> show fabric administration inventory node-groups Item Identifier Connection ConfigurationNode group NW-NG-0 Connected Configured row21-rack1 ABCD1233 Connected Configured RSNG-1 Connected Configured row1-rack2 ABCD1231 Connected Configured row1-rack3 ABCD1232 Connected Configured SNG-1 Connected Configured row1-rack1 ABCD1230 Connected Configured

Another helpful command, “show fabric administration inventory,” combines both node device and node groups.

InterfaceNamingConventionsforQFabricArchitecture

The standard Junos Os port naming convention is a three-level identifier—interface_name-fpc/pic/port_no. The fpc is the

first level, and it provides slot location within the chassis. For QFabric architecture, the three-level identification poses

a big challenge for management because QFabric technology can scale to include up to 128 QFabric Nodes, and there

is no concept of a “slot” with QFabric Nodes. Therefore, the QFabric interface naming convention has been enhanced to

include four levels, where a chassis-level identifier is added. The new interface name scheme is QFabric Node:interface_

name-fpc/pic/port. The QFabric Node can either be the serial number or the alias name that has been assigned.

netadmin@qfabric> show interfaces row1-rack1:xe-0/0/10 Physical interface: row1-rack1:xe-0/0/10, Enabled, Physical link is Up Interface index: 49182, SNMP ifIndex: 7340572 Link-level type: Ethernet, MTU: 1514, Speed: 10Gbps, Duplex: Full-Duplex, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled Interface flags: Internal: 0x0 CoS queues : 12 supported, 12 maximum usable queues Current address: 84:18:88:d5:b3:42, Hardware address: 84:18:88:d5:b3:42 Last flapped : 2011-09-06 21:10:51 UTC (04:20:44 ago) Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps)

Note: This interface naming convention only applies to physical interfaces. For logical interfaces such as LAGs, it is

node-group:interface_name-fpc/pic/slot. routed VLAN interfaces (rVIs) follow the standard naming convention used

by Juniper Networks EX series Ethernet switches: vlan.x.

InterfaceTypeConfiguration

The next few sections will cover common configurations—ports and VLANs. QFabric architecture follows the same

configuration context as EX series switches. Those who are familiar with configuring the EX series will find the next

few sections very familiar, with the only difference being the interface naming convention.

There are three different interface types—access, trunk, and routed interface. Just as with any other Junos Os platform,

interface configurations are done under the interface stanza. The access and trunk ports can be configured on any node

groups. routed interfaces are limited to rVI or NNG ports.



Accessport

[edit interfaces]netadmin@qfabric# set row1-rack1:xe-0/0/0.0 family ethernet-switching port-mode access

Note: Port mode access is optional. If port mode is not defined, the default port mode is “access.”

The standard “show interfaces” command is available. Another helpful interface command for Layer 2 port is “show

ethernet-switching interfaces <QFabric Node:interface_name-fpc/pic/slot>”. An example output is shown below:

netadmin@qfabric> show ethernet-switching interfaces row1-rack1:xe-0/0/0 detail Interface: row1-rack1:xe-0/0/0.0, Index: 82, State: up, Port mode: AccessEther type for the interface: 0x8100VLAN membership: default, untagged, unblockedNumber of MACs learned on IFL: 0

Trunkport

[edit interfaces]netadmin@qfabric# set row1-rack1:xe-0/0/0.0 family ethernet-switching port-mode trunk

below is a sample “show output” command on a trunk interface:

netadmin@qfabric> show ethernet-switching interfaces row1-rack1:xe-0/0/1 detail Interface: LC2:xe-0/0/1.0, Index: 89, State: down, Port mode: TrunkEther type for the interface: 0x8100Number of MACs learned on IFL: 0

RoutedInterface

As mentioned earlier, routed interfaces can either be rVI or Layer 3 ports on NNG. rVI provides routing between VLANs

as well as between physical routed interfaces on the NNG. The following example shows physical Layer 3 interface

configurations on both NNG and rVI.

example1:L3routedportonNNG

[edit interfaces]netadmin@qfabric# set row21-rack1:xe-0/0/0.0 family inet address 1.1.1.1/24

below is a sample “show output” command on a “show interface” for a Layer 3 route interface on an NNG:

netadmin@qfabric> show interfaces row21-rack1:xe-0/0/0 Physical interface: row1-rack4:xe-0/0/0, Enabled, Physical link is Up Interface index: 131, SNMP ifIndex: 1311224 Link-level type: Ethernet, MTU: 1514, Speed: 10Gbps, Duplex: Full-Duplex, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled Interface flags: Internal: 0x4000 CoS queues : 12 supported, 12 maximum usable queues Current address: 84:18:88:d5:e7:0c, Hardware address: 84:18:88:d5:e7:0c Last flapped : 2011-09-07 12:53:59 UTC (00:21:30 ago) Input rate : 0 bps (0 pps) Output rate : 0 bps (0 pps) Logical interface row21-rack1:xe-0/0/0.0 (Index 86) (SNMP ifIndex 1311280)



Flags: 0x4000 Encapsulation: ENET2 Input packets : 0 Output packets: 1 Protocol inet, MTU: 1500 Destination: 1.1.1/24, Local: 1.1.1.1, Broadcast: 1.1.1.255

example2:RVI

step 1. Configuring the rVI interface

[edit interfaces]netadmin@qfabric# set vlan1250.0 family inet address 10.83.100.1/24

step 2. binding the rVI interface to the VLAN

[edit interfaces]netadmin@qfabric# set vlans v1250 l3-interface vlan.1250

below is a sample “show output” command on a “show interface” for an rVI:

root@qfabric> show interfaces vlan Physical interface: vlan, Enabled, Physical link is Up Interface index: 128, SNMP ifIndex: 1311221 Type: VLAN, Link-level type: VLAN, MTU: 1518, Speed: 1000mbps Link type : Full-Duplex Current address: 84:18:88:d5:ee:05, Hardware address: 00:1f:12:31:7c:00 Last flapped : Never Input packets : 0 Output packets: 0 Logical interface vlan.1250 (Index 88) (SNMP ifIndex 2622001) Flags: 0x4000 Encapsulation: ENET2 Input packets : 0 Output packets: 1 Protocol inet, MTU: 1500 Destination: 10.83.100/24, Local: 10.83.100.1, Broadcast: 10.83.100.255

Layer3LAGConfiguration

Link aggregation provides link redundancy as well as increases bandwidth. QFabric architecture supports both static

and dynamic LAGs, which can be configured on any QFabric Node. There are two typical LAG deployments—same

member and cross member. same member LAGs are where all of the LAG child members are terminated on the

same QFabric Node. Cross member LAGs are where child member LAGs are split between node group members. As

discussed in the Defining Node Groups section, same member LAGs can be configured on any node group, while cross

member LAGs are only supported on rsNGs and NNGs.

Table2:NodeGroupsLAGSupportMatrix

NoDeGRoupS SAMeMeMBeRLAG CRoSS-MeMBeRLAG(ACTIVe/ACTIVe)

sNG 3

rsNG 3 3

NNG 3 3



example1:SamememberLAGconfiguration

step 1. Define number of supported LAGs per node group

while the example below is for an sNG named sNG-1, the same configuration is applicable to rsNG or NNG—the

configuration will just need to reflect the correct node group name. All node groups support the same member LAG

configuration.

netadmin@qfabric# set chassis node-group SNG-1 aggregated-devices ethernet device-count 1

step 2. Assign the interface to a LAG interface

Note: The chassis identifier name is the QFabric Node.

[edit interfaces]netadmin@qfabric# set row1-rack1:xe-0/0/46 ether-options 802.3ad ae0 netadmin@qfabric# set row1-rack1:xe-0/0/47 ether-options 802.3ad ae0

step 3. Configure the LAG interface

All common LAG parameters across child LAG members such as LACP, speed, duplex, and so on are centralized to the

LAG interface itself. while the example below is for a Layer 2 interface, for Layer 3 the “family” needs to change from

ethernet-switching to inet (L3 is only supported on NNG). For static LAGs, omit the LACP configuration. One thing to

note is that the node identifier is the node group, not the QFabric Node.

[edit interfaces]netadmin@qfabric# set SNG-1:ae0 aggregated-ether-options lacp activenetadmin@qfabric# set SNG-1:ae0 unit 0 family ethernet-switching port-mode trunk

some relevant commands for LAG:

• show lacp ## applicable to dynamic LAG only ##

• show interface terse | match node_group:interface_name ## example – sNG-1:ae0 ##

• show interface node_group:interface_name

step 4. Assign IP address to LAG interface

example2:CrossmemberLAGconfiguration

step 1. Define the number of supported LAGs per network node group

netadmin@qfabric# set chassis node-group NW-NG-0 aggregated-devices ethernet device-count 10

step 2. Assign the interface to a LAG interface

Note: The interface name is the QFabric Node.

[edit interfaces]netadmin@qfabric# set row1-rack2:xe-0/0/0 ether-options 802.3ad ae0 netadmin@qfabric# set row1-rack3:xe-0/0/0 ether-options 802.3ad ae0



step 3. Configure the LAG interface and assign it an IP address

All common LAG parameters across child LAG members such as LACP, speed, duplex, and so on are centralized to the

LAG interface itself. while the example below is for a Layer 2 interface, for Layer 3 the family needs to change from

ethernet-switching to inet (L3 is only supported on NNG). For static LAGs, omit the LACP configuration. One thing to

note is that the node identifier is the node group and not the QFabric Node.

[edit interfaces]netadmin@qfabric# set NW-NG-0:ae0 aggregated-ether-options lacp activenetadmin@qfabric# set NW-NG-0:ae0 unit 0 family ethernet-switching port-mode trunk

some relevant commands for LAG:

• show lacp ## applicable to dynamic LAG only ##

• show interface terse | match node_group:interface_name ## example – Nw-NG-0:ae0 ##

• show interface node_group:interface_name

Once the LAG interface is configured for Layer 2 link, change the family to inet and assign an IP address.

[edit interfaces]netadmin@qfabric# set NW-NG-0:ae0.0 family inet address 192.168.0.1/24

VLANConfiguration

VLANs allow users to control the size of a broadcast domain and, more importantly, group ports in a Layer 2 switched

network into the same broadcast domain as if they were connected on the same switch, regardless of their physical

location.

QFabric architecture is no exception. VLANs can be contained to a single node group or spread across the same and/or

different types of node groups. The steps below outline how to define VLANs and assign VLAN port membership.

step 1. Define the VLAN

VLANs are defined under the VLAN stanza. Minimum configuration is VLAN name and vlan-id.

[edit vlans]netadmin@qfabric# set default vlan-id 1

below is an example of “show vlan output.” The asterisk denotes that the interface is up.

netadmin@qfabric> show vlans Name Tag Interfacesdefault 1 row1-rack1:xe-0/0/0.0*, row1-rack1:xe-0/0/0.1*, row1-rack2:xe-0/0/3.0*, RSNG-1:ae0.0*, NW-NG-0:ae0.0*

step 2. VLAN port membership

If VLAN membership is not explicitly configured on the access ports, then it reverts back to the “default” VLAN. For

trunk ports, explicit configuration is required. There are two methods for assigning a port to a VLAN—port centric and

VLAN centric. Either method is valid, but if interface range or group profile isn’t being used, then for ease of VLAN

management, Juniper recommends that VLAN membership for the access port should be done under the VLAN

method and under the port method for the trunk port.



Method1:VLANcentric

[edit vlans]netadmin@qfabric# set default interface row1-rack1:xe-0/0/0.0

Method2:portcentric

Either the vlan-name or vlan-id (802.1Q) can be used.

[edit interfaces]netadmin@qfabric# set row1-rack1:xe-0/0/0.0 family ethernet-switching vlan members 1

Trunkport

On trunk ports, VLAN ranges are supported for ease of configuration (i.e., 1-100). For nonsequential VLANs, enclose the

membership with squared brackets and use a space for separation (i.e., 1-10 21 50-100).

[edit interfaces]netadmin@qfabric# set row1-rack1:xe-0/0/0.0 family ethernet-switching port-mode trunk vlan members [1-10 21 50-100]

In the above configuration, all VLANs are tagged on the interface. For hybrid trunks, untagged and tagged traffic use

the “native-vlan-id” keyword for untagged. below is an example trunk interface configured for VLAN 1 to be untagged

and VLANs 2-25 to be tagged. Note that VLAN 1 is not part of the “vlan members” configuration.

[edit interfaces]netadmin@qfabric# set row1-rack1:xe-0/0/0.0 family ethernet-switching port-mode trunk native-vlan-id 1 vlan members [2-25]

some helpful VLAN membership commands are:

• show vlans

• show vlans vlan-name detail

• show ethernet-switching interfaces brief

• show ethernet-switching interfaces node_identifier:interface_name-fpc/pic/port

below is an example of the media access control (MAC) address table for the QFabric:

netadmin@qfabric> show ethernet-switching table Ethernet-switching table: 3 entries, 1 learned VLAN MAC address Type Age Interfaces default * Flood - NW-NG-0:All-members default 00:10:db:ff:a0:01 Learn 51 NW-NG-0:ae0.0 default 84:18:88:d5:ee:05 Static - NW-NG-0:Router

Additional useful MAC address table commands include:

• show ethernet-switching table summary

• show ethernet-switching table interface node_identifier:interface_name-fpc/pic/port

• show Ethernet-switching table vlan



DesignuseCases

This section will describe various Layer 3 design uses cases deploying QFabric technology. For cable deployment, there

are few options—top-of-rack (TOr), middle-of-row (MOr), or end-of-row (EOr)—each of which has pros and cons.

QFabric architecture offers benefits with all three types of deployments, including lower cabling costs, modularity and

deployment flexibility, as well as fewer (one logical) devices to manage and a simplified sTP-free Layer 2 topology.

while QFabric architecture can be deployed as TOr, EOr, or MOr, for the following design use cases, the deployment

of choice will be TOr.

how the rack server or blade chassis is connected to the TOr depends on the high availability strategy, i.e., is it at the

application, server/network interface card (NIC), or network level? For rack servers, there are three different types of

connections and levels of redundancy, which are explained below.

• Single-attached: The server only has a single link connecting to the switch. In this model, there is either no

redundancy, or the redundancy is built into the application.

• Dual-attached: The server has two links connecting to the same switch. NIC teaming is enabled on the servers,

where it can be either active/standby or active/active. The second link provides the second level of redundancy. The

more common deployment is active/active with a static LAG between the switch and rack server.

• Dual-homed: The server has two links that connect to two different switches/modules in either an active/standby

or active/active mode. This is a third level of redundancy; in addition to link redundancy there is spatial redundancy.

If one of the switches fails, then there is an alternate path. In order to provide an active/active deployment, the NIC

needs to be in different subnets. If they are sharing the same IP/MAC, then some form of stacking or multichassis LAG

technology needs to be supported on the switches so that a LAG can be configured between the switches and server.

Figure4:Differenttypesofredundancyforrackservers

Depending on how the servers are connected and how NIC teaming is implemented, the QFabric Node should be

configured with the appropriate node group. The table below shows the relationship between node group and server

connections.

Table3:NodeGroupSelectionMatrixforRackServersorBladeSwitcheswithpass-ThroughModules

ACTIVe/pASSIVe ACTIVe/ACTIVe

single-attached sNG N/A

Dual-attached sNG sNG

Dual-homed rsNG rsNG

Single-attached Dual-attached Dual-homed

(L) Active/Standby(R) Active/Active

(L) Active/Standby(R) Active/Active



Network redundancy is not specific to TOr deployment, as it also exists for MOr or EOr. The same deployment

principles apply to TOr, EOr, and MOr, with minor exceptions for MOr or EOr where, in a dual-homed connection

scenario using modular switches, the second link can be connected to either a different module or a different chassis,

depending on cost and rack space.

In the case where blade chassis are used instead of rack servers, physical connectivity may vary depending on the

blade chassis intermediary connection, pass-through module, or blade switches. Juniper recommends the pass-

through module as it provides a direct connection between the servers and the QFabric architecture. This direct

connection eliminates any oversubscription and the additional switching layer that is seen with blade switches. The

deployment options for pass-through are exactly the same as described for rack servers.

As for blade switches, depending on the vendor, they all have one thing in common—they represent another device to

manage, which adds complexity to the overall switching topology. Figure 5 shows the common network deployment

between blade switches and access switches.

Figure5:Differentdeploymentscenarioswithembeddedbladeswitchesinbladechassis

• Single-homed: Each blade switch has a LAG connection into a single access switch. In this deployment, there are no

Layer 2 loops to worry about or manage.

• Dual-homed(active/backup): In this deployment, each access switch is a standalone device. since there are

potential Layer 2 loops, the blade switch should support some sort of Layer 2 loop prevention sTP or active/backup-

like technology, which will effectively block any redundant link to break the Layer 2 loop.

• Dual-homed(active/active): This deployment provides the most optimized deployment, as all links between the

blade and access switches are active and forwarding and provide network resiliency. The connection between the

blade switch and access switch is a LAG, which means the external switches must support either multichassis LAG

or some form of stacking technology. since LAG is a single logical link between the blade and external switches,

there are no Layer 2 loops to worry about or manage.

Note: Figure 5 assumes that blade switches are separate entities and are not daisy-chained or logically grouped

through a stacking technology.

since QFabric architecture is a distributed system that acts as a single logical switch, the two most likely deployments

are single-homed or dual-homed (active/active). The QFabric Nodes will be configured as sNG for single-homed and

rsNG for dual-homed (active/active).

Table4:NodeGroupSelectionMatrixforBladeChassiswithembeddedBladeSwitches

ACTIVe/pASSIVe ACTIVe/ACTIVe

single-homed sNG N/A

Dual-homed (active/backup) sNG or rsNG sNG or rsNG

Dual-homed (active/active) rsNG rsNG

Dual-homedDual-homedSingle-homed

BladeSwitch

BladeChassis

Active/Backup Active/Active



In this document, the first hop router is the QFabric architecture. Use cases where a wAN edge router such as one

of Juniper Networks MX series 3D Universal routers, a security device such as one of Juniper Networks srX series

services Gateways, or any other service layer devices (load balancer, wAN optimizer, service gateway) connect to the

QFabric architecture as Layer 3 devices are discussed below.

ConnectingLayer3DevicetoQFabricArchitecture

when Layer 3 devices connect to the QFabric architecture, the network node group port must be used for the physical

connection. Network node group members do not have to be deployed physically close together, they can span the

data center. however, only eight QFabric Nodes can be in a network node group. It is impossible to have multiple

network node groups per QFabric architecture configuration.

Figure6:Layer3devicescanbelocatedanywhereintheQFabricarchitecture

RouteLookupandForwardingDecisions

In the QFabric architecture, all of the data plane intelligence is distributed to each QFabric Node. In other words, if the

packet comes into one of the QFabric Nodes and it requires Layer 3 lookup, the QFabric Node consults its routing table

and decides on the destination QFabric Node. The ingress QFabric Node sends a packet to the 40 Gbps uplink. Once

the egress QFabric Node receives the packet, it references its own Address resolution Protocol (ArP) table to select an

appropriate port.

QFabricandVRRp

In traditional data center architectures, Virtual router redundancy Protocol (VrrP) is typically required to secure the

gateway redundancy for any Layer 3 devices. however, moving onto the QFabric architecture, VrrP is not necessary

since a QFabric solution is a single logical switch, meaning that there is no need to have multiple devices running as

gateways. within a network node group, the high availability of a gateway has already been built in. For example, the

srX series cluster in Figure 6 connects to two QFabric Nodes which are part of NNG. To the srX series cluster, it is

the same as connecting to different ports on different line cards on a single switch. These line cards and ports are

fully synchronized at the QFabric Director level. There is no need to run “protocols” to ensure the switchover between

devices; therefore, users do not have to configure VrrP among network node groups.

Junos Pulse Gateway

WX Series

MX Series

SRX Series

SNG SNG

Load Balancer

NNG: Network Node Group

NNG



Layer3DesignuseCases

useCase1:StaticDefaultRouteConfiguration

In the case where an MX series device is present and provides most of the rich routing functionality, and the QFabric

architecture just needs to provide basic routing, a static default route configuration will apply. Three QFabric Nodes in

NNG will provide inter-VLAN routing and upstream LAG access to redundant MX series devices. On the MX series side,

there are two ways to provide one unique gateway IP to QFabric architecture—one is Virtual Chassis technology on the

MX series and the other is VrrP between the two.

Figure7:NNGconnectingtoMXSerieswithLAG

step 1. Define QFabric Node alias and NNG

MX Series

NNG

VLAN1104VLAN1100

VLAN1101 VLAN1103

VLAN1102

[edit fabric]netadmin@qfabric# set aliases node-device ABCD1252 row21-rack1netadmin@qfabric# set aliases node-device ABCD1253 row21-rack2netadmin@qfabric# set aliases node-device ABCD1254 row21-rack3

netadmin@qfabric# set resources node-group NW-NG-0 network-domain netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack1netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack2netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack3

step 2. Define Layer 2 configuration

[edit vlans]netadmin@qfabric# set v1100 vlan-id 1100netadmin@qfabric# set v1100 vlan-id 1101netadmin@qfabric# set v1100 vlan-id 1102netadmin@qfabric# set v1100 vlan-id 1103netadmin@qfabric# set v1100 vlan-id 1104



step 3. Define LAG configuration NNG connecting to MX series device

[edit]netadmin@qfabric# set chassis node-group NW-NG-0 aggregated-devices ethernet device-count 24

[edit interfaces]netadmin@qfabric# set interface-range LAG-ae0 member row21-rack1:xe-0/0/[0-1] netadmin@qfabric# set interface-range LAG-ae0 member row21-rack2:xe-0/0/[0-1]netadmin@qfabric# set interface-range LAG-ae0 member row21-rack3:xe-0/0/[0-1]netadmin@qfabric# set interface-range LAG-ae0 ether-options 802.3ad ae0netadmin@qfabric# set interface-range LAG-ae1 member row21-rack1:xe-0/0/[2-3]netadmin@qfabric# set interface-range LAG-ae1 member row21-rack2:xe-0/0/[2-3] netadmin@qfabric# set interface-range LAG-ae1 member row21-rack3:xe-0/0/[2-3] netadmin@qfabric# set interface-range LAG-ae1 ether-options 802.3ad ae0

netadmin@qfabric# set NW-NG-0:ae0 aggregated-ether-options lacp activenetadmin@qfabric# set NW-NG-0:ae1 aggregated-ether-options lacp active

step 4. Assign IP address to LAG interfaces

[edit interfaces]netadmin@qfabric# set NW-NG-0:ae0.0 family inet address 192.168.0.1/24netadmin@qfabric# set NW-NG-0:ae1.0 family inet address 192.168.0.2/24

step5: Configure rVI for five VLANs

[edit interfaces]netadmin@qfabric# set vlan1100.0 family inet address 10.84.100.1/24netadmin@qfabric# set vlan1101.0 family inet address 10.85.100.1/24netadmin@qfabric# set vlan1102.0 family inet address 10.86.100.1/24netadmin@qfabric# set vlan1103.0 family inet address 10.87.100.1/24netadmin@qfabric# set vlan1104.0 family inet address 10.88.100.1/24

step 6. bind the rVI interface to the VLAN

[edit interfaces]netadmin@qfabric# set vlans v1100 l3-interface vlan.1100netadmin@qfabric# set vlans v1101 l3-interface vlan.1101netadmin@qfabric# set vlans v1102 l3-interface vlan.1102netadmin@qfabric# set vlans v1103 l3-interface vlan.1103netadmin@qfabric# set vlans v1104 l3-interface vlan.1104

step 7. Configure default routes to the MX series

[Assumes that 192.168.0.254 is the address of the MX series Virtual Chassis configuration]

[edit routing-option]netadmin@qfabric# set routing-options static route 0.0.0.0/0 next-hop 192.168.0.254



step 8. Verify default route configuration

netadmin@qfabric> show route terse

inet.0: 16 destinations, 16 routes (16 active, 0 holddown, 0 hidden)+ = Active Route, - = Last Active, * = Both

A Destination P Prf Metric 1 Metric 2 Next hop AS path* 0.0.0.0/0 S 5 192.168.0.254* 10.84.100.0/24 D 0 NW-NG-0:vlan.1100* 10.84.100.1/32 L 0 Local* 10.85.100.0/24 D 0 NW-NG-0:vlan.1101* 10.85.100.1/32 L 0 Local* 10.86.100.0/24 D 0 NW-NG-0:vlan.1102* 10.86.100.1/32 L 0 Local* 10.87.100.0/24 D 0 NW-NG-0:vlan.1103* 10.87.100.1/32 L 0 Local* 10.88.100.0/24 D 0 NW-NG-0:vlan.1104* 10.88.100.1/32 L 0 Local* 192.168.0.0/24 D 0 NW-NG-0:ae0.0 NW-NG-0:ae1.0* 192.168.0.1/32 L 0 Local* 192.168.0.2/32 L 0 LocalNote: The MX series Virtual Chassis configuration will not be covered, since it is out of the scope of this document.

Please visit www.juniper.net for more information about Virtual Chassis technology.

useCase2:puttingQFabricArchitectureintoanoSpFArea

Another use case is to run OsPF on the QFabric architecture. This scenario is applicable where the user wants more

granular control over advertised/advertising routes. In the following example, QFabric technology is deployed in OsPF

Area0 and upstream MX series devices will advertise the default route.

Figure8:QFabrictechnologyinoSpFarea0

MX Series

NNG

OSPF Area0

VLAN1104VLAN1100

VLAN1101 VLAN1103

VLAN1102



step 1. Define QFabric Node alias and NNG

[edit fabric]netadmin@qfabric# set aliases node-device ABCD1252 row21-rack1netadmin@qfabric# set aliases node-device ABCD1253 row21-rack2netadmin@qfabric# set aliases node-device ABCD1254 row21-rack3

netadmin@qfabric# set resources node-group NW-NG-0 network-domain netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack1netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack2netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack3

step 2. Define five VLANs

[edit vlans]netadmin@qfabric# set v1100 vlan-id 1100netadmin@qfabric# set v1100 vlan-id 1101netadmin@qfabric# set v1100 vlan-id 1102netadmin@qfabric# set v1100 vlan-id 1103netadmin@qfabric# set v1100 vlan-id 1104

step 3. LAG configuration NNG connecting to MX series device

[edit]netadmin@qfabric# set chassis node-group NW-NG-0 aggregated-devices ethernet device-count 24

[edit interfaces]netadmin@qfabric# set interface-range LAG-ae0 member row21-rack1:xe-0/0/[0-1] netadmin@qfabric# set interface-range LAG-ae0 member row21-rack2:xe-0/0/[0-1]netadmin@qfabric# set interface-range LAG-ae0 member row21-rack3:xe-0/0/[0-1]netadmin@qfabric# set interface-range LAG-ae0 ether-options 802.3ad ae0netadmin@qfabric# set interface-range LAG-ae1 member row21-rack1:xe-0/0/[2-3]netadmin@qfabric# set interface-range LAG-ae1 member row21-rack2:xe-0/0/[2-3] netadmin@qfabric# set interface-range LAG-ae1 member row21-rack3:xe-0/0/[2-3] netadmin@qfabric# set interface-range LAG-ae1 ether-options 802.3ad ae0

netadmin@qfabric# set NW-NG-0:ae0 aggregated-ether-options lacp activenetadmin@qfabric# set NW-NG-0:ae1 aggregated-ether-options lacp active

step 4. Assign IP address to LAG interfaces

[edit interfaces]netadmin@qfabric# set NW-NG-0:ae0.0 family inet address 192.168.0.2/30netadmin@qfabric# set NW-NG-0:ae1.0 family inet address 192.168.1.2/30

step 5. Configure rVI for five VLANs

[edit interfaces]netadmin@qfabric# set vlan1100.0 family inet address 10.84.100.1/24netadmin@qfabric# set vlan1101.0 family inet address 10.85.100.1/24netadmin@qfabric# set vlan1102.0 family inet address 10.86.100.1/24netadmin@qfabric# set vlan1103.0 family inet address 10.87.100.1/24netadmin@qfabric# set vlan1104.0 family inet address 10.88.100.1/24




[edit interfaces]netadmin@qfabric# set vlans v1100 l3-interface vlan.1100netadmin@qfabric# set vlans v1101 l3-interface vlan.1101netadmin@qfabric# set vlans v1102 l3-interface vlan.1102netadmin@qfabric# set vlans v1103 l3-interface vlan.1103netadmin@qfabric# set vlans v1104 l3-interface vlan.1104

step 7. Enable OsPF and include LAG interface and rVI to area 0

[edit]netadmin@qfabric# set protocols ospf area 0.0.0.0 interface NW-NG-0:ae0.0netadmin@qfabric# set protocols ospf area 0.0.0.0 interface NW-NG-0:ae1.0netadmin@qfabric# set protocols ospf area 0.0.0.0 interface vlan.1100netadmin@qfabric# set protocols ospf area 0.0.0.0 interface vlan.1101netadmin@qfabric# set protocols ospf area 0.0.0.0 interface vlan.1102netadmin@qfabric# set protocols ospf area 0.0.0.0 interface vlan.1103netadmin@qfabric# set protocols ospf area 0.0.0.0 interface vlan.1104

step 8. Verify OsPF neighbor

[edit]root@SV-POC-QF> show ospf neighborAddress Interface State ID Pri Dead192.168.0.3 NW-NG-0:ae0.0 Full 13.13.13.1 128 36192.168.0.4 NW-NG-0:ae1.0 Full 12.12.12.1 128 31

step 9. Verify routing table

netadmin@qfabric> show route terse

inet.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)+ = Active Route, - = Last Active, * = Both

A Destination P Prf Metric 1 Metric 2 Next hop AS path* 10.84.100.0/24 D 0 NW-NG-0:vlan.1100* 10.84.100.1/32 L 0 Local* 10.85.100.0/24 D 0 NW-NG-0:vlan.1101* 10.85.100.1/32 L 0 Local* 10.86.100.0/24 D 0 NW-NG-0:vlan.1102* 10.86.100.1/32 L 0 Local* 10.87.100.0/24 D 0 NW-NG-0:vlan.1103* 10.87.100.1/32 L 0 Local* 10.88.100.0/24 D 0 NW-NG-0:vlan.1104* 10.88.100.1/32 L 0 Local* 192.168.0.0/24 D 0 NW-NG-0:ae0.0 NW-NG-0:ae1.0* 192.168.0.1/32 L 0 Local* 192.168.0.2/32 L 0 Local* 0.0.0.0/0 O 10 1 >192.168.0.3 192.168.0.4* 224.0.0.5/32 O 10 1 MultiRecv



useCase3:puttingQFabricArchitectureintooSpFStubArea

Another use case is to run QFabric architecture in an OsPF stub area. This scenario is applicable where the user wants

to minimize the routing table size.

Most of the configurations are the same as those in Use Case 2. The only difference is to configure the OsPF area as

stub at step 7, then add rVI interfaces to the stub area. Note that it is possible not to advertise summary routes in the

stub area by adding the “no-summaries” option.

[edit]netadmin@qfabric# set protocols ospf area 0.0.0.1 stub no-summariesnetadmin@qfabric# set protocols ospf area 0.0.0.1 interface NW-NG-0:ae0.0netadmin@qfabric# set protocols ospf area 0.0.0.1 interface NW-NG-0:ae1.0netadmin@qfabric# set protocols ospf area 0.0.0.1 interface vlan.1100netadmin@qfabric# set protocols ospf area 0.0.0.1 interface vlan.1101netadmin@qfabric# set protocols ospf area 0.0.0.1 interface vlan.1102netadmin@qfabric# set protocols ospf area 0.0.0.1 interface vlan.1103netadmin@qfabric# set protocols ospf area 0.0.0.1 interface vlan.1104

useCase4:Connectingone-ArmedSRXSeriesDeviceasActive/ActivewithQFabricArchitecture

It is frequently required to connect firewalls to the core/aggregation device. The next two use cases will discuss how

srX series services Gateways can be deployed with QFabric solutions. The diagram below shows a typical deployment

in which two Juniper Networks srX5800 services Gateway devices running in active/active mode connect to an EX

series/MX series device in a one–armed fashion.

Figure9:SRXSeriesone-armeddeploymentinatwo-tierarchitecture

EX4200Virtual Chassis

SRX5800_BSRX5800_A

EX4200Virtual Chassis

EX Series/MX Series

Core/Edge Tier

VLAN 500, 1001, 1003, 1005 VLAN 600, 1000, 1002, 1004 VLAN 1000 VLAN 1001



when a customer migrates to the QFabric architecture, the one-armed deployment will appear as in Figure 10. There

is no need to change the configuration on the srX5800 side. The fundamental QFabric solution configuration is the

same as on the EX series/MX series devices in Figure 9.

Figure10:one-armedSRXSeriesactive/activedeploymentwithQFabrictechnology

In this example, srX5800_A and srX5800_b connect to the QFabric solution as “one-armed” devices, deployed as an

active/active cluster.

The first VLAN trunk is handling VLANs 500, 1001, 1003, and 1005, while the second trunk handles VLANs 600, 1000,

1002, and 1004. This VLAN traffic will be distributed to the srX5800 cluster srX5800_A and srX5800_b. A solid line

denotes the primary link for the given VLAN, while a dotted line indicates the backup. with the virtual router functions

of QFabric architecture, inter-VLAN routing won’t ensure that these two groups are totally isolated at the Layer 3 level.

This is feasible in a multi-tenant environment. VLANs 500 and 600 will be used for uplink connections to the wAN

edge router from the srX series under the “set security zones security-zone uplink interface” stanza. here the first

VLAN trunk is in virtual router instance 10 (Vr10) while the second VLAN trunk is in Vr20. In addition, rVI VLANs 500

and 600 will be in Core Vr to provide uplink connection to the wAN edge routers. servers just need to send packets to

the VrrP address on the srX series gateway in each rVI VLAN (1000 through 1005).

Note that srX series configuration details are not covered since they are out of scope for this document. The following

configuration examples focus on network node group configuration. Please review previous use case or the L2 design

guide for server node group configuration information.

step 1. Define QF/Node alias and NNG

to WAN EdgeSRX5800_A

SRX5800_B

VLAN 500, 1001, 1003, 1005

VLAN 600, 1000, 1002, 1004

VLAN 1000

VLAN 1001

[edit fabric]netadmin@qfabric# set aliases node-device ABCD1252 row21-rack1netadmin@qfabric# set aliases node-device ABCD1253 row21-rack2netadmin@qfabric# set aliases node-device ABCD1254 row21-rack3netadmin@qfabric# set aliases node-device ABCD1255 row21-rack4netadmin@qfabric# set resources node-group NW-NG-0 network-domain netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack1netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack2netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack3netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack4



step 2. Define VLANs

[edit vlans]netadmin@qfabric# set v500 vlan-id 500netadmin@qfabric# set v600 vlan-id 600netadmin@qfabric# set v1000 vlan-id 1000netadmin@qfabric# set v1001 vlan-id 1001netadmin@qfabric# set v1002 vlan-id 1002netadmin@qfabric# set v1003 vlan-id 1003netadmin@qfabric# set v1004 vlan-id 1004netadmin@qfabric# set v1005 vlan-id 1005

step 3. Map VLAN to interface

set interfaces row21-rack3:xe-0/0/20 unit 0 family ethernet-switching port-mode trunkset interfaces row21-rack3:xe-0/0/20 unit 0 family ethernet-switching vlan members v1000set interfaces row21-rack3:xe-0/0/20 unit 0 family ethernet-switching vlan members v1002set interfaces row21-rack3:xe-0/0/20 unit 0 family ethernet-switching vlan members v1004set interfaces row21-rack3:xe-0/0/21 unit 0 family ethernet-switching vlan members v500set interfaces row21-rack4:xe-0/0/20 unit 0 family ethernet-switching port-mode trunkset interfaces row21-rack4:xe-0/0/20 unit 0 family ethernet-switching vlan members v1001set interfaces row21-rack4:xe-0/0/20 unit 0 family ethernet-switching vlan members v1003set interfaces row21-rack4:xe-0/0/20 unit 0 family ethernet-switching vlan members v1005set interfaces row21-rack4:xe-0/0/21 unit 0 family ethernet-switching vlan members v600


[edit interfaces]netadmin@qfabric# set vlans v500 l3-interface vlan.500netadmin@qfabric# set vlans v600 l3-interface vlan.600netadmin@qfabric# set vlans v1000 l3-interface vlan.1000netadmin@qfabric# set vlans v1001 l3-interface vlan.1001netadmin@qfabric# set vlans v1002 l3-interface vlan.1002netadmin@qfabric# set vlans v1003 l3-interface vlan.1003netadmin@qfabric# set vlans v1004 l3-interface vlan.1004netadmin@qfabric# set vlans v1005 l3-interface vlan.1005

step 5. Configure rVI for VLAN 500 and 600

[edit interfaces]netadmin@qfabric# set vlan500.0 family inet address 10.84.100.1/24netadmin@qfabric# set vlan600.0 family inet address 10.84.101.1/24



step 6. Create virtual router instance and include rVIs

Configuring VR10netadmin@qfabric# set routing-instances VR-TEN instance-type virtual-routernetadmin@qfabric# set routing-instances VR-TEN interface vlan.1001netadmin@qfabric# set routing-instances VR-TEN interface vlan.1003netadmin@qfabric# set routing-instances VR-TEN interface vlan.1005netadmin@qfabric# set routing-instances VR-TEN protocols ospf area 0.0.0.0 interface all

Configuring VR20netadmin@qfabric# set routing-instances VR-TWENTY instance-type virtual-routernetadmin@qfabric# set routing-instances VR-TWENTY interface vlan.1000netadmin@qfabric# set routing-instances VR-TWENTY interface vlan.1002netadmin@qfabric# set routing-instances VR-TWENTY interface vlan.1004netadmin@qfabric# set routing-instances VR-TWENTY protocols ospf area 0.0.0.0 interface all

Configuring CoreVRnetadmin@qfabric# set routing-instances core instance-type virtual-routernetadmin@qfabric# set protocols ospf area 0.0.0.0 interface row21-rack1:xe-0/0/10.0netadmin@qfabric# set protocols ospf area 0.0.0.0 interface row21-rack1:xe-0/0/11.0netadmin@qfabric# set protocols ospf area 0.0.0.0 interface row21-rack2:xe-0/0/10.0netadmin@qfabric# set protocols ospf area 0.0.0.0 interface row21-rack2:xe-0/0/11.0netadmin@qfabric# set routing-instances core interface vlan.500netadmin@qfabric# set routing-instances core interface vlan.600netadmin@qfabric# set routing-instances core protocols ospf area 0.0.0.0 interface all

useCase5:Connectingone-ArmedSRXSeriesasActive/BackupwithQFabricArchitecture

The srX series can also be deployed in an active/backup manner. Again, the configuration is simple with QFabric

technology because it uses the same approach as the EX series switches. Users simply need to create a VLAN for

terminating server connections, create an rVI (VLAN 100 in this case as shown in Figure 11) for uplink connection, and

put rVI in L3 routing. The srX series devices are configured as the primary security gateway for their respective VLANs,

so servers just need to send packets to the VrrP address of the srX series in each VLAN.



Figure11:one-armedSRXSeriesactive/activedeploymentwithQFabricarchitecture



SRX5800_B

VLAN 1001, 1003, 1005

VLAN 1000, 1002, 1004

VLAN 100

[edit fabric]netadmin@qfabric# set aliases node-device ABCD1252 row21-rack1netadmin@qfabric# set aliases node-device ABCD1253 row21-rack2netadmin@qfabric# set aliases node-device ABCD1254 row21-rack3netadmin@qfabric# set aliases node-device ABCD1255 row21-rack4netadmin@qfabric# set resources node-group NW-NG-0 network-domain netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack1netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack2netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack3netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack4

step 2. Define VLANs

[edit vlans]netadmin@qfabric# set v100 vlan-id 100netadmin@qfabric# set v1000 vlan-id 1000netadmin@qfabric# set v1001 vlan-id 1001netadmin@qfabric# set v1002 vlan-id 1002netadmin@qfabric# set v1003 vlan-id 1003netadmin@qfabric# set v1004 vlan-id 1004netadmin@qfabric# set v1005 vlan-id 1005



step 3. Map VLAN to interface

set interfaces row21-rack3:xe-0/0/20 unit 0 family ethernet-switching port-mode trunkset interfaces row21-rack3:xe-0/0/20 unit 0 family ethernet-switching vlan members v1000set interfaces row21-rack3:xe-0/0/20 unit 0 family ethernet-switching vlan members v1002set interfaces row21-rack3:xe-0/0/20 unit 0 family ethernet-switching vlan members v1004set interfaces row21-rack3:xe-0/0/21 unit 0 family ethernet-switching vlan members v100set interfaces row21-rack4:xe-0/0/20 unit 0 family ethernet-switching port-mode trunkset interfaces row21-rack4:xe-0/0/20 unit 0 family ethernet-switching vlan members v1001set interfaces row21-rack4:xe-0/0/20 unit 0 family ethernet-switching vlan members v1003set interfaces row21-rack4:xe-0/0/20 unit 0 family ethernet-switching vlan members v1005set interfaces row21-rack4:xe-0/0/21 unit 0 family ethernet-switching vlan members v100


[edit interfaces]netadmin@qfabric# set vlans v100 l3-interface vlan.100

step 5. Configure rVI for VLAN 100

[edit interfaces]netadmin@qfabric# set vlan100.0 family inet address 10.84.100.1/24

step 6. Enable OsPF and include uplink interfaces and rVI to area 0

[edit]netadmin@qfabric# set protocols ospf area 0.0.0.0 interface row21-rack1:xe-0/0/10.0netadmin@qfabric# set protocols ospf area 0.0.0.0 interface row21-rack1:xe-0/0/11.0netadmin@qfabric# set protocols ospf area 0.0.0.0 interface row21-rack2:xe-0/0/10.0netadmin@qfabric# set protocols ospf area 0.0.0.0 interface row21-rack2:xe-0/0/11.0netadmin@qfabric# set protocols ospf area 0.0.0.0 interface vlan.100



useCase6:Connectingone-ArmedSRXSeriesGatewaytoQFabricArchitecture(VRF-BasedSteeringMode)

If a customer would like to create a security zone per VrF basis and apply those security policies to inter-VrF traffic,

QFabric needs to act as the first hop router, and the srX series will be used for services delivery only. with this model,

it is important to note that the QFabric solution routes a significantly higher volume of traffic that doesn’t need

services, so that needs to be taken into consideration to avoid capacity or scaling problems. For example, Figure 12

shows that intra-VrF traffic (vlan.1001) won’t hit the srX series, while inter-VrF traffic (vlan.1000 and vlan.1004) will.

Figure12:Applyingsecuritypolicytointer-VRFroutingonQFabricarchitecture

Vr-Zone-A contains VLAN 1000, 1001, and 1002. Vr-Zone-b includes VLAN 1003, 1004, and 1005. Each Vr has

a default route entry which is pointing to the srX series. The configuration is described below. Note that it is still

necessary to have separate Vr instances to connect to the wAN edge, which is configured as Core Vr. (Only the Vr

portion will be covered here.)

Configuring Vr-ZONE-A


SRX5800_B

VLAN 500, 1001, 1003, 1005

VLAN 600, 1000, 1002, 1004

VLAN 1000

VLAN 1001

VLAN 1004

netadmin@qfabric# set routing-instances VR-ZONE-A instance-type virtual-routernetadmin@qfabric# set routing-instances VR-ZONE-A interface vlan.1000netadmin@qfabric# set routing-instances VR-ZONE-A interface vlan.1001netadmin@qfabric# set routing-instances VR-ZONE-A interface vlan.1002netadmin@qfabric# set routing-instances VR-ZONE-A routing-options static route 0.0.0.0/0 next-hop x.x.x.x [VRRP address of each RVI on SRX]netadmin@qfabric# set routing-instances VR-ZONE-A protocols ospf area 0.0.0.0 interface all



Configuring Vr-ZONE-b

netadmin@qfabric# set routing-instances VR-ZONE-B instance-type virtual-routernetadmin@qfabric# set routing-instances VR-ZONE-B interface vlan.1003netadmin@qfabric# set routing-instances VR-ZONE-B interface vlan.1004netadmin@qfabric# set routing-instances VR-ZONE-B interface vlan.1005netadmin@qfabric# set routing-instances VR-ZONE-B routing-options static route 0.0.0.0/0 next-hop x.x.x.x [VRRP address of each RVI on SRX]netadmin@qfabric# set routing-instances VR-ZONE-B protocols ospf area 0.0.0.0 interface all

Configuring Core Vr

netadmin@qfabric# set routing-instances core instance-type virtual-routernetadmin@qfabric# set protocols ospf area 0.0.0.0 interface row21-rack1:xe-0/0/10.0netadmin@qfabric# set protocols ospf area 0.0.0.0 interface row21-rack1:xe-0/0/11.0netadmin@qfabric# set protocols ospf area 0.0.0.0 interface row21-rack2:xe-0/0/10.0netadmin@qfabric# set protocols ospf area 0.0.0.0 interface row21-rack2:xe-0/0/11.0netadmin@qfabric# set routing-instances core interface vlan.500netadmin@qfabric# set routing-instances core interface vlan.600netadmin@qfabric# set routing-instances core protocols ospf area 0.0.0.0 interface all

useCase7:QFabricArchitectureBack-to-BackextensionwithL3LAG

Currently, there is a 150 meter distance limitation between a QFabric Node and the QFabric Interconnect due to the

QsFP+ specification. however, there is a way to increase this distance through the use of a back-to-back extension. In

Figure 13, QFabric 1 can reach QFabric 2, which is up to 300 meters away, with sFP+, sr and 640GbE bandwidth. The

solution consists of eight QFabric Nodes in a network node group, allowing it to form eight LAGs between the remote

NNGs. Currently, QFabric architecture supports eight-way equal-cost multipath (ECMP). Following the configuration

example in Figure 13 only covers the L3 LAG extension portion; please review previous use cases and the L2 design

guide for other configuration options.

Figure13:Back-to-backextensionwithLAG

160GbpsFabric/

QFabric Node

160GbpsFabric/

QFabric Node

QFabric_1 QFabric_2

8x10GbE LAG

8x10GbE LAG

8x10GbE LAG

8x10GbE LAG

8x10GbE LAG

8x10GbE LAG

8x10GbE LAG

8x10GbE LAG

QFabricDirector

QFabricDirector




[edit fabric]netadmin@qfabric# set aliases node-device ABCD1252 row21-rack1netadmin@qfabric# set aliases node-device ABCD1253 row21-rack2netadmin@qfabric# set aliases node-device ABCD1254 row21-rack3netadmin@qfabric# set aliases node-device ABCD1255 row21-rack4netadmin@qfabric# set aliases node-device ABCD1256 row21-rack5netadmin@qfabric# set aliases node-device ABCD1257 row21-rack6netadmin@qfabric# set aliases node-device ABCD1258 row21-rack7netadmin@qfabric# set aliases node-device ABCD1259 row21-rack8

netadmin@qfabric# set resources node-group NW-NG-0 network-domain netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack1netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack2netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack3netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack4netadmin@qfabric# set resources node-group NW-NG-0 node-device row21-rack5netadmin@qfabric# set resources node-group Nw-NG-0 node-device row21-rack6

netadmin@qfabric# set resources node-group Nw-NG-0 node-device row21-rack7

netadmin@qfabric# set resources node-group Nw-NG-0 node-device row21-rack8

step 2. LAG configuration NNG connecting to QFabric 2

[edit]

netadmin@qfabric# set chassis node-group Nw-NG-0 aggregated-devices ethernet device-count 24

[edit interfaces]

netadmin@qfabric# set interface-range LAG-ae0 member row21-rack1:xe-0/0/[0-7]

netadmin@qfabric# set interface-range LAG-ae0 ether-options 802.3ad ae0

















netadmin@qfabric# set Nw-NG-0:ae0 aggregated-ether-options lacp active









step 3. Add IP address to LAG interfaces

[edit interfaces]

netadmin@qfabric# set Nw-NG-0:ae0.0 family inet address 192.168.0.1/24








Summary

The exponential data center demands exponential power, flexibility, and control, along with exponential reductions in

energy consumption and TCO. The QFabric architecture with provides just such a flexible solution for deploying a fabric

across the data center, enabling unique network designs that fundamentally simplify while maintaining any-to-any

connectivity, reducing the number of managed devices and connections, and centralizing data center management.

by following this design and implementation guide, Layer 3 QFabric architecture can be successfully deployed. The

designs suggested in this document will help establish complete data center solutions by integrating MX series, srX

series, and Juniper Networks Virtual Gateway products in a way that not only solves the increasing problems of scale

and data center economics, but has the potential to enable dramatic new levels of computing for years to come.

AboutJuniperNetworks

Juniper Networks is in the business of network innovation. From devices to data centers, from consumers to cloud providers,

Juniper Networks delivers the software, silicon and systems that transform the experience and economics of networking.

The company serves customers and partners worldwide. Additional information can be found at www.juniper.net.



8010083-001-EN Nov 2011

Copyright 2011 Juniper Networks, Inc. All rights reserved. Juniper Networks, the Juniper Networks logo, Junos, Netscreen, and screenOs are registered trademarks of Juniper Networks, Inc. in the United states and other countries. All other trademarks, service marks, registered marks, or registered service marks are the property of their respective owners. Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice.

eMeAheadquarters

Juniper Networks Ireland

Airside business Park

swords, County Dublin, Ireland

Phone: 35.31.8903.600

EMEA sales: 00800.4586.4737

Fax: 35.31.8903.601

ApACheadquarters

Juniper Networks (hong kong)

26/F, Cityplaza One

1111 king’s road

Taikoo shing, hong kong

Phone: 852.2332.3636

Fax: 852.2574.7803

CorporateandSalesheadquarters

Juniper Networks, Inc.

1194 North Mathilda Avenue

sunnyvale, CA 94089 UsA

Phone: 888.JUNIPEr (888.586.4737)

or 408.745.2000

Fax: 408.745.2100

www.juniper.net

Printed on recycled paper

To purchase Juniper Networks solutions,

please contact your Juniper Networks

representative at 1-866-298-6428 or

authorized reseller.

Documents

Designing a Layer 3 Data Center Network with the QFabric … · 2011-09-06 · Interface Naming Conventions for QFabric Architecture ... Designing a Layer 3 Data Center Network with