©2016 Open-NFP 1

Stacks and Layers: Integrating P4, C, OVS and OpenStack

Johann Tönsing September 21, 2016

©2016 Open-NFP 2

Agenda

• Review of Traditional Cloud Networking StacksOVS (with/without connection tracking), Contrail vRouter, SR-IOV, VirtIO…

• Issues: Flexibility, Performance, CPU Utilization • Accelerating and Offloading Cloud Networking using Agilio™ SmartNICs

• Implementation Details: Hardware and Software Architectures • Introducing Additional Programmability

• Implications of Supporting P4, C and eBPF Extensions • Results: Performance Comparisons • Update on Integration / Open Sourcing Activities • Conclusions

©2016 Open-NFP 3

Traditional OpenStack Networking Options

Forwarding /Virtual Switching Technology

OpenStack Control Plane Forwarding Overlays (Tunnels)

Security Groups (Microsegmentation, Stateful Firewalling)

SR-IOV with regular NIC

Vendor specific plugin Limited E.g. MAC/VLAN based directing

None None

OVS

ML2 plugin (optional: SDN controller) OVN driver

L2 / L3 using OpenFlow match/action tables

VXLAN, GRE, (for OVN) GENEVE …Emerging: NSH (for NFV)

OVS 2.5+ can interface to Linux conntrack (Previously Linux bridge with iptables/nftables)

Contrail vRouterContrail driver (via Contrail controller)

L2 / L3 using dedicated forwarding tables

MPLS in UDP, MPLS in GRE,VXLAN …

Built in, via flow table

Additional features: load balancing, NAT etc.

©2016 Open-NFP 4

OpenStack Networking Options Evaluated

Forwarding /Virtual Switching Technology

Traditional Approach Agilio™ SmartNIC Accelerated Approach

SR-IOV

Limited expressiveness to direct traffic to VMs (no support for general match/action rules, tunnel termination, stateful firewalling) High throughput No VM migration support

Full OVS or vRouter virtual switching incl. tunnel termination, stateless/stateful firewalling

and SR-IOV based data delivery to VMs High throughput VirtIO integration, supporting VM migration

OVS andContrail vRouter

High expressiveness - match/action, tunnels, stateless/stateful firewalling etc. Limited throughput High CPU utilization (e.g. 50% of cores)

Same expressiveness - match/action, tunnels, stateless/stateful firewalling etc.

Higher throughput (~5x higher) Lower CPU utilization (~10x lower)

©2016 Open-NFP 5

Agilio™ CX SmartNIC Hardware

• Optimized for standard server based cloud data centers • Low Profile Half Length PCIe form factor, power < 25W • Based on Netronome’s NFP-4xxx silicon (72 C programmable cores, 8 threads each) • 2GB DRAM for lookup tables / state tables (millions of entries) • Dataplane fully implemented in software

1x 40GbE 2x 40GbE2x 10GbE 2x 25GbE

Also available: Agilio™ LX 2x40G / 1x100G with dual PCIe interfaces, 120 cores, 8GB DRAM…

©2016 Open-NFP 6

Offload Model: Agilio™ OVS Acceleration

Open vSwitch Subsystem

OVS Agent

OpenFlow

Virtual MachineVirtual Machine

Virtual Machine

x86 Kernel

x86 Userspace

PCIe

Virtual Machine

SR-IOV / VirtIO VFs

SR-IOV / VirtIO VFs

Agilio™SmartNIC

Apps

Apps

1

1

netdev or DPDK

netdev or DPDK

Apps

netdev or DPDK

Apps

netdev or DPDK

Apps

netdev or DPDK

OVS CLI Callable API

1 Configuration via controller, CLI, or Callable API

2

2 OVS userspace agent populates kernel cache

(Nova, Neutron)

Execute Action

Open vSwitch Datapath

Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)

Conn track

3 Offload datapath: copy match tables, sync stats

3

5 Offload connection tracking: synchronize state

5

Conn track

FTPSIP

4 Flow tracking: per-microflow state learning

4

Self Learning Exact Match Flow Tracker

Miss

Hit

OVS Kernel DP Match/Act

OVS Kernel DP Match/Act

Miss

Miss

©2016 Open-NFP

vRouter Datapath

vRouter Flow

Classify

Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)

vRouter ForwardingvRouter

Forwarding

7

Agilio™ vRouter Acceleration

vRouter Subsystem

vRouter Agent

Virtual MachineVirtual Machine

Virtual Machine

x86 Kernel

x86 Userspace

PCIe

Virtual Machine

SR-IOV / VirtIO VFs

SR-IOV / VirtIO VFs

Agilio™SmartNIC

Apps

Apps

1

1

netdev or DPDK

netdev or DPDK

Apps

netdev or DPDK

Apps

netdev or DPDK

Apps

netdev or DPDK

vRouter CLI

1 Configuration via controller or CLI: BGP over XMPP

3

3 Userspace agent populates kernel tables

(Nova, Neutron)

Execute Action

4 Offload flows: copy tables, sync stats

4

vRouter Flow

Classify

Miss

Miss

vRouter ForwardingvRouter

Forwarding

Notes: - Control protocol: BGP over XMPP - Tunnel protocols: VXLAN, MPLS over UDP / GRE - Distinct policy (flows - ACLs) and forwarding (IP/MPLS) - Flow table is used to implement stateful firewalling

2

2 Offload forwarding: pre-emptively copied

©2016 Open-NFP 8

Offload Existing Datapath: Characteristics

•Flexibility defined by server’s existing datapath software •OVS: Configure match/action tables (forwarding / policies) •Contrail vRouter: Configure forwarding and policies separately

•Integration via drivers / plugins •OVS: OpenStack ML2 plugin (with/without SDN controller) •OVS: OpenStack driver for OVN •Contrail vRouter: OpenStack driver for ContrailExtend OpenStack to support new concept - SR-IOV path directly to VM while offloading virtual switching to NIC

©2016 Open-NFP 9

SmartNIC Firmware: Pre-programmed or Custom

Compiler Debugger

Run-Time

app.P4 app.C

Editor

• Firmware can be developed in P4 and/or C • To discuss: integrating custom programmed

datapaths (implemented in P4 / C etc.) into existing code (OVS / OpenStack…)

• SmartNIC with dynamically downloadable firmware

Contrail OVS

OpenStack ONOS ODL

Linux BSD

• OVS / Contrail / Linux eBPF datapath on host can be accelerated by SmartNICD

P D K

eBPF

©2016 Open-NFP 10

Example: P4 “main” implementing a simple NICheader_type eth_hdr { fields { dst : 48; src : 48; etype : 16; } } header eth_hdr eth;

parser start { return eth_parse; }

parser eth_parse { extract(eth); return ingress; }

action drop_act() { drop(); }

action fwd_act(port) { modify_field(standard_metadata.egress_spec, port); }

table in_tbl { reads { standard_metadata.ingress_port : exact; } actions { fwd_act; drop_act; } }

control ingress { apply(in_tbl); }

©2016 Open-NFP

Datapath Program

in HL lang e.g. P4

• Configtime

Compiler back end

• Protocol Independent Forwarding (PIF)Configured (“Programmed”) Datapath • Programs in language(s) like P4 describe datapath • Parse tree => protocol independent •Match/action tables (control flow arranges table sequence) • Packet metadata, per table or global state •QoS

Datapath Program

in IR

Compiler front end��������������������������������� ��

�$��#��!����(,�)�����)�%�)��)$����%'$�'�""����,��%'$%$(��)��)�������'�!��(�$#��#���()'��)��$',�'��#��"$��!��#��!�#�*������(��#���(%�������!!-��$'�)��(�%*'%$(��������#��������#���%�#�!$,���"%!$-��#���()'��)��$',�'��#��"$��!���(���$#���)�����)�$#����!�(�������)��)��$#)��#�'*!�(��$'�%'$��((�#��%�� �)(�'����+����-�)���(,�)��������*'������%��)(���(�"%!�������$'"�$���#���������()'��)��$',�'��#��"$��!��

�

���*'��� ���()'��)�%�� �)��$',�'��#���#��#����

��� �)(�'����+����-�)���(,�)����'����'()�%�'(���)$�$�)��#�)����#�)��!������'��$#)�#)(��������'��$�#�.�������'����!�(�'�!�+�#)�)$�%�� �)�%'$��((�#����#��(�)��#�)��!�%�� �)�"�)���)����������#�'�((�%$')�#*"��'������ �)�%'$��((�#��%'$����(�)�'$*�����(�&*�#���$�����(�������$��,�����"�-�%�'�$'"���)�$#(�)��)���#�����((�%�� �)���)���%�� �)�"�)���)����#���!$��!���)����������!$,�()�)��%'�(�'+����-�)���(,�)�����#��"�-�%�'�$'"�����)�$#�!�%�� �)�%�'(�#�����#�!!-����%�� �)��(��((��#���)$�.�'$�$'�"$'��&*�*�(�)$����(����*!����$'�)'�#("�((�$#�$#���'�((�%$')(�����*'�������#)����(�($"���((�#)��!��$"%$#�#)(�$���#���()'��)��$',�'��#��"$��!���$,�+�'��)��'���'����+�'��)-�$��,�-(�)��)�)��(���$"%$#�#)(���#����'��!�.����#��#��"%!�"�#)�)�$#��#��)������*'���(�#$)��#)�#����)$��"%!-��#-�%�')��*!�'��"%!�"�#)�)�$#�()'*�)*'���

�������#��!�(���%'$�'�""�'�)$�%'$�'�"�)���%�� �)��$',�'��#���#��#���$#���*'�)�$#������#������%'$+������()�')�#��%$�#)��$'���+�!$%�#�����$"%!�)��������!�#�*����������$!!$,�#��(��)�$#(���(�'����)����((�#)��!����)*'�(�$��)���������!�#�*�����

/�����%�#���),$' �#���$*#��)�$#���!!�'���)(�'�(�'+���� 11

Protocol Independent Forwarding => Flexibly Programmed Datapath

• Predefined Protocol Forwarding • OpenFlow specification defines protocols / fields,

match/action behavior, overall control flow(tables can influence) • Set of supported protocols fixed by

implementation

OpenFlow

• Either way - need “southbound” interfacefor run-time interaction with switch... •Populate tables, receive statistics + events

OpenFlow++OpenFlow Controller

• Run time

©2016 Open-NFP 12

Extending OVS using P4/C Plugins

Open vSwitch Subsystem

OVS Agent

OpenFlow

Virtual MachineVirtual Machine

Virtual Machine

x86 Kernel

x86 Userspace

PCIe

Virtual Machine

SR-IOV / VirtIO VFs

SR-IOV / VirtIO VFs

Agilio™SmartNIC

Apps

Apps

1

1

netdev or DPDK

netdev or DPDK

Apps

netdev or DPDK

Apps

netdev or DPDK

Apps

netdev or DPDK

OVS CLI Callable API

1 Configuration via controller, CLI, or Callable API

2

2 OVS userspace agent populates kernel cache

(Nova, Neutron)

Execute Action

Open vSwitch Datapath

Execute Action (e.g. Entunnel, Deliver to VM, Send to Port)

3 Offload datapath: copy match tables, sync stats

3

OVS Kernel DP Match/Act

Datapath Extension or Plugin

P4 / C in Sandbox

DP Ext.

4 Datapath extension software

4

4

OVS Kernel DP Match/Act

Miss

Miss

©2016 Open-NFP 13

Plugin in Datapath: Characteristics

• Some flexibility – Easy to implement custom actions - e.g. filter / modify packet – Difficult to implement custom classification (combined with OVS) – Can implement inner protocols (OVS must support outer protocols)

• Integration effort varies – Can model as custom port » Similar to VNF in network (on dedicated interface or attached to tunnel)

from controller perspective => least intrusive – Can model as custom action » Requires OpenFlow protocol extension (controller side, OVS side) » May require changes to OVN, OpenStack ML2 plugin / net driver, etc.

©2016 Open-NFP 14

OVS “on” SmartNIC P4 Datapath

Open vSwitch Subsystem

OVS Agent

OpenFlow

Virtual MachineVirtual Machine

Virtual Machine

x86 Kernel

x86 Userspace

PCIe

Virtual Machine

SR-IOV / VirtIO VFs

SR-IOV / VirtIO VFs

Agilio™SmartNIC

Apps

Apps

1

netdev or DPDK

netdev or DPDK

Apps

netdev or DPDK

Apps

netdev or DPDK

Apps

netdev or DPDK

OVS CLI Callable API (Nova, Neutron)

Execute Action

P4 Generated Datapath

Execute P4

Action

OVS Kernel DP Match/Act

P4 Matching

Fallback

Fallback

©2016 Open-NFP 15

P4 Datapath on SmartNIC: Characteristics

• Some flexibility – In theory easy to implement offloaded behavior on SmartNIC

(new protocols, new actions etc.) - using high productivity language (P4)

– Note however OpenFlow matching is more flexible - canaugment any table at any time with any field / action

– Furthermore limited to what OVS on host supports (C code) • Integration effort modest

– Already done if offloading existing OVS code – Must extend OpenFlow+OVSDB or OVN if enhancing OVS

©2016 Open-NFP 16

P4 “into” OVS Datapath

Open vSwitch Subsystem

OVS Agent

OpenFlow

Virtual MachineVirtual Machine

Virtual Machine

x86 Kernel

x86 Userspace

PCIe

Virtual Machine

SR-IOV / VirtIO VFs

SR-IOV / VirtIO VFs

Agilio™SmartNIC

Apps

Apps

1

netdev or DPDK

netdev or DPDK

Apps

netdev or DPDK

Apps

netdev or DPDK

Apps

netdev or DPDK

OVS CLI Callable API (Nova, Neutron)

P4 Generated Datapath

Execute P4 / OVS Action

Execute P4 / OVS Action

P4 / OVS Matching

P4 / OVS Matching

Fallback

Fallback

©2016 Open-NFP 17

P4 Datapath in Kernel: Characteristics

• Mixed flexibility – Easy to implement behavior (new protocols, new actions etc.) by

using high productivity language (P4) – Note however OpenFlow matching is more flexible - can

augment any table at any time with any field / action » Regenerate program on demand to correspond to OpenFlow tables » Implement program based on assumed model

• Integration effort considerable – Need to re-implement OVS on P4

(TBD how compatible userspace will be with regular OVS) – Offloading easier once infrastructure in place

©2016 Open-NFP 18

P4 “instead of” OVS (Datapath and Control)

Host Code

P4 Agent

Virtual MachineVirtual Machine

Virtual Machine

x86 Kernel

x86 Userspace

PCIe

Virtual Machine

SR-IOV / VirtIO VFs

SR-IOV / VirtIO VFs

Agilio™SmartNIC

Apps

Apps

1

netdev or DPDK

netdev or DPDK

Apps

netdev or DPDK

Apps

netdev or DPDK

Apps

netdev or DPDK

(Nova, Neutron)

P4 Generated Datapath

Execute Action

Execute Action

P4 or eBPF or XDP

P4 Matching or eBPF…

Fallback

Fallback

1 Protocol(s) to be discussed (could become callable API)

Other open issues: - Downloading programs via OpenStack or other systems- Scheduling VMs to run on nodes with acceleration hardware (Nova)Acceleration discussion started at OpenStack Spring 2016 (Austin)

or offloaded eBPF / XDP…

©2016 Open-NFP 19

New Datapath/Control: Characteristics

• Full flexibility – Easy to implement behavior (new protocols, new actions etc.)

by using high productivity language (P4) – Can deploy completely different control plane

• Integration effort considerable – New infrastructure required in OpenStack - new ML2 plugin /

driver, new southbound protocol or API, etc.

©2016 Open-NFP

SmartNIC Firmware (P4/C)

20

Example of Fully Customized Datapath (P4 / C)

Run-Time Interface

Server (x86 - Linux)

PCIe

Agilio™ SmartNIC

Virtual Machine 1

VNF Kernel Mode

(C)

netdev

P4 / C Development Environment

Edit - Debug

Control App

Populate tables, display

statistics

Security µVNF (C)

Timestamp µVNF (C)

Latency Stats µVNF (C)

Virtual Machine 2

VNF User Mode

(C)

DPDK

Timestamp µVNF (C)

Latency Stats µVNF (C)

Match Protocol

Meter

Other

TCP

Concepts:

• P4 and C running on SmartNIC implements datapath - e.g.defines protocols, match / actionbehavior

• Datapath steers traffic to VNFsrunning on x86 server and on SmartNIC

©2016 Open-NFP 21

OVS Throughput vs. Number of Rules

5

10

15

20

25

30

OVS in Kernel Space

OVS in User Space on DPDK

100 Wildcard Rules

1000 Wildcard Rules

10000 Wildcard Rules

64000 Wildcard Rules

Mill

ions

of P

acke

ts p

er S

econ

d

12 CPU Cores

12 CPU Cores

OVS Offloaded to Agilio™ CX-4000

1 CPU Core

5X Throughput Improvement + 90% CPU Savings

OVS L2/L3 Forwarding to 8 VMs with 64K Flows

©2016 Open-NFP 22

Efficiency: Throughput Per Server CPU Core

Throughput with single server CPU corededicated to network related processing

Mill

ions

of P

acke

ts p

er S

econ

d • 50x Efficiency vs. Kernel OVS

• 20x Efficiency vs. User Mode (DPDK) OVS=> Replace 3-6 racks with 1 rack!

Throughput / efficiency similar for Agilio™ Contrail vRouter

Benefit for your use case: search for

“netronome.com roi calculator”

©2016 Open-NFP 23

Integration / Open Sourcing ActivitiesArea Activities

Linux driversVF netdev (kernel device driver) upstreamed in kernel 4.5

Patches available for older kernels, or use VirtIO

Representative netdev (for fallback processing of traffic) proposals imminent

FreeBSD drivers Kernel device driver implemented

DPDK driversPoll mode driver upstreamed in DPDK 2.2

Patches available for older DPDK versions

Open vSwitch accelerationintegration

Hooks for acceleration proposed in October 2014

Iteration in progress

OpenStack integration

Proposing plugins and agents to support virtual switching acceleration via RFE process

Integration for OVS in process - Mirantis, Ericsson CEE etc.

Integration for Contrail vRouter in process - Juniper etc.

Also: P4 / protocol independent forwarding and OpenFlow run-time API, eBPF / XDP acceleration…

Participation appreciated — join us at Linux, Open vSwitch, Contrail vRouter, OpenStack, p4.org, OpenSourceSDN.org, …

©2016 Open-NFP 24

Next Steps

• Use Agilio™ SmartNICs with existing dataplanes • Use Agilio™ OVS (with / without Conntrack) • Use Agilio™ vRouter • Future: Agilio™ eBPF/XDP, etc.

• Program Agilio™ SmartNICs • Use APIs (on x86 servers) - with above dataplanes • Program in P4 and/or C (on SmartNIC / on x86)

• Improve performance + free up server resources!

©2016 Open-NFP 25

Thank You!

More information: netronome.com and, of course: open-nfp.org