OpenFlow Software Switch & Intel DPDK - EWSDN 2016 2013/IS1_1_Removi… · Slide title 70 pt CAPITALS Slide subtitle minimum 30 pt OpenFlow Software Switch & Intel® DPDK performance

Slide title

70 pt

CAPITALS

Slide subtitle

minimum 30 pt

OpenFlow Software Switch & Intel® DPDK performance analysis

Slide title

44 pt

Text and bullet level 1

minimum 24 pt

Bullets level 2-5

minimum 20 pt

Characters for Embedded font: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻżŽžƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ–—‘’‚“”„†‡•…‰‹›⁄€™ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș−≤≥fifl

ΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨΩΪΫΌΎΏ

ЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐәǽẀẁẂẃẄẅỲỳ№

Do not add objects or

text in the footer area OF 1.3 prototype measurement - EWSDN | Public | © Ericsson AB 2013 | 2013-10-07 | Page 2

› Background

– Intel® DPDK

– OpenFlow 1.3 implementation sketch

› Prototype design and setup

› Results

› Future work, optimization ideas

Agenda

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt






› Why Intel® DPDK? (www.intel.com/go/dpdk) – kernel space implementation

› is more restricted

› harder to develop and debug

› interrupts are still needed performance issues

– user space implementation over normal Linux kernel is slow

› user – kernel memory separation, copy is slow

- some workarounds exist (e.g. pcap mmap), but they are still not fast enough

– a similar, but less widespread solution: http://info.iet.unipi.it/~luigi/netmap/

› Main features – poll mode driver: avoid using interrupts and scheduling

– direct I/O: packet – or first X bytes – is copied to L1 cache directly

– some details from the Intel® DPDK tutorial will follow

Intel® dpdk basics

http://www.intel.com/go/dpdk

http://info.iet.unipi.it/~luigi/netmap/

http://info.iet.unipi.it/~luigi/netmap/

4

Designed to run on any Intel® architecture CPU

• Intel® Atom™ to client cores to Sandy Bridge

• Essential to the IA value proposition

PThread to bind h/w thread to s/w task

• Literally no scheduler overhead

User Level Polled Mode Driver

• No Kernel Context / Interrupt Context Switching Overhead

Huge Pages To Improve Performance

• 1Gig Huge as well as 2 Meg Page support

• Co-exists with Linux’s 4 K Page

Low Latency Cache and Memory Access

• DDIO - Cache

• Prefetch and rte_cache_aligned - memory

Intel® DPDK Basic Design

5

Understanding the Choices & Performance Setting the Direction for the Intel® DPDK

Scheduler (or why not)

• Hardware threads only

• No scheduler/task switcher - typical task switch time is between 200+ processor cycles (varies depending on processor architecture)

Process bunch of packets at a time

• Cores process a bunch of packets at a time to amortize some latencies

Prefetch

• Critical to latency hiding since we don’t have software threads. Stalls on hardware threads are costly

• The queue-based model is key to making prefetch effective

Locks

• Generally lockless implementations where-ever possible. A spinlock-unlock pair costs between 60-90 cycles.

• Queues are lockless (single producer & multi-producer, single consumer)

6

Scheduler … or Why not?

Primary reason was performance:

• Task-switch overhead is typically a “few” hundred cycles

• FXSAVE/FXRSTOR are 100 and 150 cycles respectively (on Intel NetBurst®)

• Faster on recent processors, but not significantly

• Need to add cost of interrupt if pre-emptive

To put that in perspective in a 10 GbE environment

• On a 3 GHz processor, for small (64B) packets, a packet arrives every 67.2 ns = 201 cycles

For lower bandwidth environments, an essential thing to think about is the added CPU bandwidth consumed …

7

Packet Bunching …

Done on the NIC today

• NIC Receive descriptors are bunched four to a cache line

• Writing back “partial” descriptors has a severe performance penalty

– Conflicts between CPU and I/O device on the same cache line

– Increases memory & PCI-E bandwidth usage

– Needed to overcome PCI-Express latencies

• All Intel Ethernet* controllers have settings that can be tweaked to control descriptor write-back

– Coalesce as many descriptors as possible on Receive

– Transmit side coalescing done as well (software controlled)

– Timer values can be set to control latency (EITR)

Took the paradigm to the next level in having the fast-path process bunches of packets

• Facilitated by the queue abstraction

* Other names and brands may be claimed as the property of others.

8

Prefetch

Two types of prefetch – hardware & software

• Hardware prefetch is issued by the core – L1 DCU prefetcher: Streaming prefetcher – triggered by ascending access to recently

loaded data

– L1 IP-based strided prefetcher: triggered on individual loads with a stride

– L2 DPL: Prefetches data into L2 cache based on DCU requests

– Adjacent cache line (n, n+2, prefetch n+2)

– Strided prefetcher (e.g. skipped cache lines)

• Software prefetch needs to be issued “appropriately” ahead of time to be effective – Too early could cause eviction before use

– Multiple types of software prefetch

9

Paging

• 1GB super-pages & 2 Meg Huge Page Support

Performance implications …

• Primarily due to D-TLB thrashing/replacement

• Paging performance drop is difficult to gauge … really dependent on application

• Gets significantly worse as memory footprint increases –

• Varies by architecture, but initial measurements suggested ~30% on L3 forwarding

– Quite often 2-3 D-TLB replacements per packet

10

Intel® Data Direct I/O Technology (Intel® DDIO)

1x SNB-EP 8C B0, 2.0GHz

10

11

Intel® DPDK Performance

IPv4 Layer 3 Forwarding on an

IA Server Platform

0.00

50.00

100.00

150.00

200.00

250.00

300.00

20092S Intel® Xeon

processor E5645(2x6C Westmere)

2.4GHz

20092S Intel® Xeon®processor E5540(2x4C Nehalem)

2.53 GHz

20102S Intel® Xeon®processor E5645

(2x6C Westmere-EP)2.40 GHz

20121S Intel® Xeon® E5-2658 processors C1

Stepping(1x8C Sandy bridge-EP)

2.1 GHz8 x 10GbE PCIe Gen2

20122S Intel® Xeon® E5-2658 processors C1

Stepping(2x8C Sandy bridge-EP)


20132S Intel® Xeon® E5-

2658v2(2x10C Ivy bridge-EP)


255 MPPS R1.4

255 MPPS R1.4

Mpps

Introduction of Integrated Memory Controller

+ Intel® DPDK

Introduction of Integrated PCIe* Controller

Massive IA Performance Improvements since 2009, PCIe Gen3 will Offer Even Better Performance….!

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

110 MPPS R1.3

110 MPPS R1.3

250 MPPS R1.3

250 MPPS R1.3

Intel® DPDK Release 1.3

• SNB @2.1 GHz 1C/1T = 18.6 Mpps 1C/2T = 24 Mpps

• SNB @2.7 GHz 1C/1T = 23.7 Mpps 1C/2T = 28.8 Mpps

Intel® DPDK Release 1.4

• IVB @2.4 GHz 1C/1T = 23.9 Mpps 1C/2T = 28.5 Mpps

64B Throughput

55 MPPS R1.0

55 MPPS R1.0

42 MPPS Intel DPDK R0.7

42 MPPS Intel DPDK R0.7

12 MPPS Native Linux

12 MPPS Native Linux

Native Linux Kernel Performance

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt






› Main idea: programmable networking – flexibility, programmability – together with high performance

› Problem: OF is either flexible OR fast today – flexible rules with many tuples: use TCAM or slow lookup

› TCAM is expensive and uses a lot of power

– complex instructions and actions: high overhead for software implementations

– some solutions limit flexibility to increase performance (e.g. TTP)

› In theory performance should only depend on the data plane functions the node is implementing in the given scenario

– it should be irrelevant whether the device is executing a native implementation of the use case, or is executing OF rules programmed by a controller for the same purpose

openflow basics

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt






Flow Table

Action Set

Execution

Meter

Actions

Group Entry

Buckets

Action Set

Flow Entry

Instructions

Apply Actions

wildcard/prio

lookup

table ID

meter ID

port ID

group ID

add/rem/mod

group entry

add/rem/mod

flow entry

group ID

data access per

packet processing

data access

by control plane

data access

by internal control

remove dependent

flows on removal

Port

Queue

queue ID

port ID

liveliness

propagation

group ID meter ID

port liveliness

bucket

liveliness remove assoc.

instructions

remove assoc.

flow entry

add/rem/mod

meter

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt






› Background

– OpenFlow 1.3 implementation sketch

– Intel DPDK

› Prototype design and setup

› Results

› Future work, optimization ideas

Agenda

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt






› Software prototypes investigated and not selected – OVS: well-established, open source

› mainly for virtual environment, performance issues

- OVS on Intel® DPDK (OVDK) is an ongoing activity

– CPqD softswitch: used by ONF for prototyping new features, open source

› serious performance limitations

– Linc: Erlang based softswitch, open source

› runs in a VM environment, while we are primarily interested in close to the hardware solutions

› Hardware based prototypes / products – they have serious limitations in terms of number of rules

› usually OF implementations use TCAM which has limited capacity

– usually hard to program / modify / add new features

why new prototype

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt






› Simple MAC based forwarding

– 1, 10, 100, 1000, 2000 and 5000 DMAC rules

› currently with linear search

› always the last rule will match caching is not easy…

– instruction = write action

› action set = Output (egress port)

› Intel® DPDK based generator station (tgen)

– generates 15 Mpps (@ 64 Bytes / pkt) on one core

configuration

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt






Flow Table

Action Set

Execution

Meter

Actions

Group Entry

Buckets

Action Set

Flow Entry

Instructions

Apply Actions

wildcard/prio

lookup

table ID

meter ID

port ID

group ID

add/rem/mod

group entry

add/rem/mod

flow entry

group ID

data access per

packet processing

data access

by control plane

data access

by internal control

remove dependent

flows on removal

Port

Queue

queue ID

port ID

liveliness

propagation

group ID meter ID

port liveliness

bucket

liveliness remove assoc.

instructions

remove assoc.

flow entry

add/rem/mod

meter

1

2

3

4

5

6

7

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt






measurement setup

GENERATOR

Intel Xeon E5-2630

2x6 cores @ 2.3 GHz

8x4 GB DDR3 SDRAM

OF-SW

Intel Xeon E5-2630

2x6 cores @ 2.3 GHz

8x4 GB DDR3 SDRAM

3Com (mgmt) 172.31.32.0/24

10G

Intel Niantic

(82599EB)

2x10 GbE

10G

1G 1G

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt






ETH2 (10G) ETH3 (10G)

GENERATOR

OF-SW

ETH2 (10G) ETH3 (10G)

core0

core1

core2

ETH0 (1G) ETH1 (1G)

Linux

Linux driver Intel® DPDK driver

core3

rx tx

core4

rx tx

core5

rx tx

q42 q52 q43 q53

OF code OF code

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt






› Main results: 25% overhead vs. L2FWD (Intel’s example)

– it was more without highly optimizing the software

– linear with nr. of rules

(not surprisingly)

› So we began some investigation…

results

Pkt size L2FWD OF 1.3 1 rule

Mpps Gbps Mpps Gbps

64 13.82 7.08 10.27 5.26

128 8.45 8.65 8.45 8.65

256 4.53 9.28 4.53 9.28

512 2.35 9.63 2.35 9.63

10

100

1000

10000

100000

1 10 100 500 1000 2000

Performance (kpps) Processing time (ns)

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt






› Processing time per number of rules

– at small number of rules cache(s) are effectively used

– note that real traffic would behave better

some details

4

5

6

7

8

9

10

11

12

13

14

15

10 100 500 1000 2000 5000

time per rule (ns)

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt






improving further

› Preliminary results of current code: overhead was

completely removed

8

9

10

11

12

13

14

15

16

L2FWD

OF 1.3 1 rule

OF 1.3 ++

Slide title

44 pt


minimum 24 pt

Bullets level 2-5

minimum 20 pt






and further…

› Current status: basically removed “static” OF overhead

› It’s time for improving rule processing speed and implement control plane

› Basic ideas under discussion – high-performance southbound interface

› minimize the need for locking, timeouts, etc.

– fast data plane execution

› flow caching

› lookup algorithm selection, “selective TTP”

› usage prediction

Documents

OpenFlow Software Switch & Intel DPDK - EWSDN 2016 2013/IS1_1_Removi… · Slide title 70 pt CAPITALS Slide subtitle minimum 30 pt OpenFlow Software Switch & Intel® DPDK performance