NET1949BU Seamless Network Connectivity for …...Enhanced Networking and Security for AWS Native Workloads Refer to CNA1091BU - One-Stop Container Networking: Cloud Foundry, Kubernetes,

Suresh ThiruSridhar Subramanian

NET1949BU

VMworld 2017 - NET1949BU

Seamless Network Connectivity for Virtual and Bare-metal Workloads with NSX

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

2



bution

Agenda

1 NSX Everywhere

2 Bare-metal Use Cases

3 NSX Solutions

4 Design Considerations and Best Practices

3



bution

Works across Hypervisors, Clouds,

Application frameworks

Infrastructure

independent

Security wrapped around the VM, container, microservice

Application Drives InfrastructureWhat does this mean for Networking and Security?



bution

Evolution of Server Computing

Explosion of VM and

mobility led to network

virtualization

Native container network

with multi-tenancy,

micro-segmentation, and

common tools for day 2

operations

Full visibility and control

with consistent operation

across private and public

cloud

Seamless connectivity and

security for physical

workloads (Legacy App, DB,

Storage, Security Appliance)

Container Workload

Dynamic Static

Bare-metal Workload

Public Cloud Workload

VM Workload

Networking And Security Services

Introduces new Networking and Security requirements



bution

Container Workload

Bare-metal Workload


VM Workload

New Silos Leads to Operational InefficienciesChallenges: Different technology stacks, processes, teams, and expertise



bution

• Uniform Networking & Security services across private & public clouds

• Single pane of glass management

• Supports any physical network infrastructure

PV FW

Container Workload

Bare-metal Workload


VM Workload

NSX Everywhere: A Platform For All Workloads



bution

8

Cloud

Consumption

Control Plane

Management

Plane

Data Plane

Hypervisor

Virtual Switch

SW RT FW

Container on

Hypervisor or

Bare-metal*

Virtual Switch

SW RT FW

Guest VM in

Public Cloud

Virtual Switch

SW RT FW

NSX Edge

Services

Edge Router

RT FW LB VPN

ESXi

Bare-metal Server

PV

Virtual

Switch

OVSDB

TOR

NSX Manager

NSX Controller

* NSX support for containers on bare-metal is planned for future release

NSX Architecture Extended to Support All WorkloadsCentral Management to manage networking and security policies



bution

NSX Platform Journey

Public

Cloud

Multi

Hypervisor ContainersvSphere Baremetal

Delivered entire

networking and

security services in

software for vSphere

Extended NSX to

KVM and Openstack

Integrated NSX with

PaaS and Container

orchestrator for

cloud-native apps

Extended NSX to

native cloud

workload and cloud

services

NSX benefits extended to

bare-metal



bution

Focus For Rest of The Session

Public

Cloud

Multi

Hypervisor ContainersvSphere Baremetal

Refer to

NET1510BU -

Introduction to NSX-T

Architecture

Refer to

NET1535BU -

NSX Design—

Reference Design for

SDDC

with NSX and vSphere

Refer to

MMC2046BU -Using VMware NSX for

Enhanced Networking and

Security for AWS Native

Workloads

Refer to

CNA1091BU -One-Stop Container

Networking: Cloud Foundry,

Kubernetes, Docker, and More

This SessionVMworld 2017 Content: Not fo


bution

Bare-metal Use Cases



bution

Use Case 1: Integration of non-Virtualized Workloads

• Typically necessary for integrating a non-virtualized appliance

• L2 as well as L3

• A gateway takes care of the on ramp/off ramp

12

Overlay-backed Workloads

OverlayVLAN

Virtual To PhysicalGATEWAY

PhysicalWorkloads



bution

Use Case 2: Migration Of Physical To Virtual

• Physical workloads migrated in phases to virtual form factor

• Temporary, bandwidth not critical

BEFORE DURING MIGRATION AFTER

3 Physical Workloads


OverlayVLAN


1 VirtualWorkloads


OverlayVLAN


3 VirtualWorkloads


OverlayVLAN

0 VirtualWorkloads



bution

Use Case 3: Migration Of VLAN-Backed Virtual Workloads

• VLAN-backed Virtual workloads Migrated in phases to Overly-backed Virtual workloads

• Temporary, bandwidth not critical

BEFORE DURING MIGRATION AFTER

3 VLAN-backed

Workloads


OverlayVLAN

2 VLAN-backed

Workloads

1 Overlay-backedWorkload


OverlayVLAN

0 VLAN-backed

Workloads

3 Overlay-backedWorkload


OverlayVLAN

0 Overlay-backedWorkloads



bution

NSX Solutions



bution

Guiding Principle: Routing Vs Bridging

Routed Connectivity to Physical workloads

• Standard Routing protocols (OSPF and BGP)

• ECMP Scale-out, failure isolation with routing

Bridged Connectivity to Physical workloads

• Flat Broadcast domain limiting size and scale

• Single Active bridge for a VXLAN-VLAN pair

16

Route when you Can, Bridge when you Must !

Overlay VLAN


Physical Workloads

VLAN-backed Virtual workload

L2

…

Overlay VLAN

Physical Workloads

L3

L2L2 VMworld 2017 Content: Not fo


bution

Web1 App1

External

Network

Oracle ExadataPhysical Server

L3

Edge

Services

GatewayVPN

Distributed Logical Router

NET1416BU NSX-T Logical Routing

Practical Example with Exadata Server in separate L3 Subnet

• Web and App Tier in Overlay

• App Tier and Exadata Are in different Subnets

• Edge Gateway provides routed North-South to physical network

• Performance & Scale with ECMP

• Most commonly deployed by Oracle & Enterprise



bution

Web1 App1

External

Network

L3

Edge

Services

GatewayVPN

Distributed Logical Router


Oracle ExadataBare-metal Server

App tier and Exadatashare the same subnet

Practical Example when Exadata and APP Server in same Subnet

• Web and App Tier in NSX Overlay

• App to Exadata Bridging via the “Virtual to Physical Gateway” realizable in Two ways

1. NSX Software Bridge Design in a separate VM

2. HW Gateway Design by enabling Top-Of-Rack Network switch to provide the function



bution

Summary of Bridging Options For Virtual To Physical Connectivity

19

SW Agent*

Pros

• Common NSX Stack for workload connectivity across Bare-metal servers, Hypervisors, Containers and Public Cloud

• Paves way for security Of workload at OS layer

Cons

• Legacy OS versions not supported

*This is NOT a shipping option today and is in exploration stage

SW Bridge

Pros

• Independent Of Physical Switch

HW or SW

• Scale-out with little investment

• High performance VXLAN to VLAN

gateway in hypervisor kernel

Cons

• Density of Physical workloads mapping to different VxLAN –VLAN pairs

HW Gateway

Pros

• Offers Higher Bandwidth and port-density for workloads

• Useful in racks where no Hypervisor can be deployed

• Fast Failover and Redundancy features from HW Vendors

Cons

• Reduces Virtualization benefits by introducing Hardware dependency



bution

Software Bridge - Recorded Demo

192.168.1.10192.168.1.20

VLAN 16

Overlay-backed Workload

PhysicalWorkload

OverlayVLAN

NSX SW Bridge hosted in a

Hypervisor Instance



bution

Software Bridge DEMO With NSX-T



bution

Hardware Bridge – Recorded Demo

22

172.16.10.10

VLAN 160

Ethernet18

Database Logical Switch

VNI 5000

PhysicalWorkload

NSX Controllers10.114.221.235-237

10.114.221.196Arista Switch as Hardware Gateway10.114.211.105

HV1

VTEP

Overlay-backed Workload

172.16.10.11



bution

Up Next: Configuration of the Arista Hardware Gateway

23

Configuration of the Replication Cluster



bution

Up Next: Registration of the Arista Hardware Gateway into NSX

24

Configuration of the Arista Hardware Gateway



bution

Up Next: Binding a Logical Switch to a Physical Port/VLAN

25

Registration of the Arista Hardware Gateway into NSX



bution

26

Binding a Logical Switch to a Physical Port/VLAN



bution

Customer Case Study

• Deployment Region: Global

• Deployment Scale :

– 1st Phase: 26 Hosts

– 2nd Phase: 30 Hosts in 2nd DC

• Management: Log Insight

• NSX version: 6.2.3

• 3rd Party Integration: Arista Hardware VTEP

• NSX Features Used:

– HW Gateway

– DFW

27

Large Electronics Manufacturing Company



bution

Network Topology for Case Study

28

Key Takeaways

• Use Case:- Shared storage service

with controlled access for compute

Rack VMs.

• Problem: Compute rack VMs need

shared storage access from a non-

virtualized disk.

• Conditions: VMs cannot talk to each

other

• Solution:

– HW Gateway Solution used to

bridge VM traffic to VLAN X on

which storage disk attached to get

shared service

– DFW used to prevent VM to VM

communication

Storage

Disk

VLAN XDatabase RacksCompute Racks

VXLAN

VLAN

VXLAN ID 500X

Arista HW Gateway



bution

Design Considerations and Best Practices



bution

Software Bridge vs. Hardware Gateway

30

• A single bridging instance per Logical Switch

• Bandwidth limited by single bridging instance

• L2 network must be extended to reach all the

physical devices

• Several Hardware Gateways can be deployed at several

locations simultaneously

• With Hardware Gateways, VLANs can be kept local to a

rack and don’t need to be extended

VXLAN

VLAN

VLAN 10 VLAN 20VLAN 10 VLAN 10Database Racks

VLAN extended

between racks

Non-virtualized

devices (part of the

same L2 segment)

Database RacksCompute Racks Compute Racks



bution

Redundancy considerations With Software Bridge

31

PhysicalWorkload VLAN Y

Control VM-0

SW Bridge

Hypervisor

ACTIVE

Control VM-1

SW Bridge

Hypervisor

STANDBY VirtualWorkload

(Logical Switch X)

PhysicalWorkload VLAN Y

Control VM-0

SW Bridge

Hypervisor

DOWN

Control VM-1

SW Bridge

Hypervisor

ACTIVE VirtualWorkload

(Logical Switch X)

• SW Bridge functionality for a given VLAN/VXLAN par can only be active on a single Hypervisor

Recommendation:

• Introduce redundancy by selecting Standby Hypervisor that will host SW Bridge and take over upon failure

• Standby is determined by location of Control VM that the User configures



bution

Hardware Based Solution re-Introduces Hardware Dependency

32

The Hardware Based model invalidates the benefits of virtualization

VXLAN

VLAN

VLAN 10 VLAN 20Database Racks

Compute Racks

Physical switch HW and SW versions need to be certified with NSX

Hardware Gateway does not natively support Distributed Routing or Distributed Firewall



bution

Redundancy Consideration With Hardware Gateway

• The OVSDB based mechanism is currently not aware of any form of redundancy

• Several Hardware Gateways can be active for the same Logical Switch

• A backdoor connection could result in a loop

• Recommendation: Only connect hosts to the Hardware Gateway

33

VLAN 10Hardware

Gateway1

LS VNI:5000

VLAN 10

Network

Switch in

backdoorPhysical

workload 1

Physical

workload 2

Loop

Hardware

Gateway2



bution

Best Practices For Redundancy With Hardware Gateway

34

• Active/Standby uplink

• No L2 connection must be made between switches

Host Based Redundancy

VLAN 10

Hardware

Gateway1

Hardware

Gateway2

LS VNI:5000

VLAN 10

Physical

workload



bution

Best Practices For Redundancy With Hardware GatewayPort Channel Based Redundancy

Data Plane: Physical View

HV

Several physical Hardware

Gateways presented as a

single logical one to NSX

Data Plane: Logical View

HV

Most Hardware vendors offer a distributed port channel based solution



bution

Security Considerations For Bare-metal Workloads

36

• Distributed or Edge FW can regulate V-P traffic

• NSX integration with Partner FW manager can regulate V-P traffic closest to the Physical workload

PhysicalWorkload

VPN

Edge Firewall

PhysicalNetwork

STOP

STOP

Virtual To Physical communication

STOP

PhysicalWorkload

VPN

Edge Firewall

NSX Manager

Security Groups

PhysicalNetwork

STOP

Eg.Partner Firewall

Mangement Console

STOP STOP



bution

Security Considerations For Bare-metal Workloads

37

IPFIX (from

vSphere)

NetFlow (from

physical)

Search, Analytics and Micro-segmentation Modeling

Across Virtual, Physical & Cloud

Public Cloud VirtualPhysical

AWS Flow

Logs

Physical To Physical Flows Analysis & ACL Recommendations*

*This is NOT a shipping option today and is currently under development

• vRealize Network Insight (vRNI) leverageable analyzing flows from virtual, physical (Netflow) and cloud. V-to-V, V-to-P and P-to-P

• Micro-segmentation models, application tier definition and firewall/ACL rules recommendation for physical end points / IPs

• Scale out architecture for large scale flow collection. No agents.

PAR4377BU NSX Advanced Security



bution

Key Takeaways

38

PV FW

ContainerWorkload

Bare-metalWorkload

PublicCloudWorkload

VMWorkload

• Route when you can and Bridge only when you must

• Recommended Order Of Bridging Solutions For Bare-metal workloads

• SW Bridge

• Hardware Gateway

• Secure Bare-metal servers with native NSX solution or with NSX integrated partner solution



bution

Relevant Sessions and References

▪ Sessions

▪ References

NSX for vSphere Network Virtualization Design Guide (Ver 3.0)

https://communities.vmware.com/docs/DOC-27683

39

NET1535BU

NET1536BU

Reference Design for SDDC with NSX and vSphere: Part 1 & 2

NET1863BU NSX-T Advanced Architecture Concepts

NET1416BU NSX-T Logical Routing

CNA1091BU One-Stop Container Networking: Cloud Foundry, Kubernetes, Docker,

and More

MMC2046BU Using VMware NSX for Enhanced Networking and Security for AWS

Native Workloads



bution

https://communities.vmware.com/docs/DOC-27683



bution



bution

Documents

NET1949BU Seamless Network Connectivity for …...Enhanced Networking and Security for AWS Native Workloads Refer to CNA1091BU - One-Stop Container Networking: Cloud Foundry, Kubernetes,