Embed Size (px)
Citation preview
MTS: Bringing Multi-Tenancy to Virtual Networking
Kashyap Thimmaraju, Saad Hermak, Gábor Rétvári and Stefan Schmid
USENIX Annual Technical Conference 2019July 11, Renton, Washington, USA
Virtual Networks Using Virtual Switches
Host OS
VM$_
VM$_
Host OS
VM$_
VM$_
VM$_
VM$_
2
Virtual Networks Using Virtual Switches
Host OS
VM$_
VM$_
Host OS
VM$_
VM$_
VM$_
VM$_
Virtual Switch
3
Virtual Networks Using Virtual Switches
Host OS
VM$_
VM$_
Host OS
VM$_
VM$_
VM$_
VM$_
4
Virtual Networks Using Virtual Switches
Host OS
VM$_
VM$_
Host OS
VM$_
VM$_
VM$_
VM$_
Broadcast | Multicast | Unicast | Tunnel
1. Red2. Blue3. Green
5
More Than 20 Virtual Switches
Most emphasis has been on performance and flexibility
6
Security Weaknesses ofVirtual Switches
7
Processes Untrusted Data
A malicious VM can send arbitrary packets to the virtual switch
Host OS
VM$_
Host OS
VM$_
VM$_
8
Privileged Packet Processing
Oftentimes runs in the kernel for performance
9
Host OS
VM$_
VM$_
Host OS
VM$_
VM$_
UserKernel
Single Point of Failure
Virtual network configurations are complex
10
Screenshot from Karim Elatov’s blog: https://elatov.github.io/2018/01/openstack-ansible-and-kolla-on-ubuntu-1604/#5-packet-goes-from-ovs-inte
gration-bridge-br-int-to-ovs-tunnel-bridge-br-tun
Single Point of Failure
Mis-configurations could lead to security issues
Host OS
VM$_
Host OS
VM$_
VM$_
11
Co-Located with the Host OS
The consequence of a compromise can be severe, e.g., break out of VM isolation
Host OS
VM$_
VM$_
Host OS
VM$_
VM$_
12
Exploiting Virtual Switches in the Cloud
SOSR’18: Remote-Code ExectionOvS Con’19: Cross Tenant DoS
Host OS
VM$_
Host OS
VM$_
VM$_
13
Outline ● Motivation
● MTS
● Evaluation
● Scalability
● Pros and Cons
● Conclusion
14
MTS: Multi-Tenant Switch
15
Least Privilege Virtual Switch
16
1. Processes untrusted data
2. Privileged packet processing
3. Single point of failure
4. Co-located with the Host OS
Host$_
VM$_
VM$_
Least Common Mechanism
17
1. Processes untrusted data
2. Privileged packet processing
3. Single point of failure
4. Co-located with the Host OS
Host$_
VM$_
VM$_
Extra Security Boundary
18
1. Processes untrusted data
2. Privileged packet processing
3. Single point of failure
4. Co-located with the Host OS
Host$_
VM$_
VM$_
Complete Mediation
19
1. Processes untrusted data
2. Privileged packet processing
3. Single point of failure
4. Co-located with the Host OS
Host$_
VM$_
SR-IOV NIC
PFIn/Out
VFGw VF
TVF
VM$_
In/Out VF
Gw VF
TVF
L2 Switch in NIC
Evaluation
20
Experimental Setup & Factors
Mellanox ConnectX4, Open vSwitch, DPDK, QEMU, KVMMore details in the paper
● Resources● Traffic Patterns
21
Shared Resources
CPU● Host OS pinned to 1 core● All vswitch-VMs pinned to 1
core● Each Tenant VM got
dedicated cores (not shown here)
Host OS
22
Traffic Patterns
VM
NICIn Out NICIn Out
VM
NICIn Out
VM
p2p p2v v2v23
Baseline vs MTSPacket Processing Throughput Comparison
64 byte UDP packetsRoughly the same in p2pMTS is ~2x Baseline in p2v and v2v
24
BASELINE
1
VS-VM
2
VS-VM
4
VS-VM
Baseline vs MTSPacket Processing Throughput Comparison
64 byte UDP packetsRoughly the same in p2pMTS is ~2x Baseline in p2v and v2v
25
Baseline vs MTSNetwork Application Throughput
MTS beats Baseline inApache and Memcached
26
1+ Physical Core4x Network Isolation1.5-2x Throughput
27
Scaling MTS
28
Containers in VMs
Real cloud systems can host more than just 4 tenants on a server
● Work in progress
● The packets per second throughput is
the same as running it in a VM for 4
containers
● Can run 12 vswitches spread across 4
VMs
● Faced an issue with libvirt when
adding 40 VFs to 16 vswitches spread
across 4 VMs. The interfaces do not
appear in the VM although the
configuration is present.
29
Pros and Cons
30
Limitations ● PCIe bus could become a bottleneck
which our evaluation did not reveal
● The number of VFs on the NIC
● No clean solution for live migration of
VMs with VFs
31
Pricing State-of-the-art MTS
Charge for CPU cycles used by the tenant-specific virtual switch
Broadcast | Multicast | Unicast
1. Red2. Blue
Broadcast | Multicast | Unicast
Broadcast | Multicast | Unicast
$$ $
32
Tenant Specific Virtual Switch Software
Broadcast | Multicast | Unicast
1. Red2. Blue
Broadcast | Multicast | Unicast
Broadcast | Multicast | Unicast
State-of-the-art MTS
1. Reduce parsing logic2. Support tenant-specific
features
33
Conclusion
34
Key Takeaways 1. Many virtual switches can be
exploited to compromise Host and
Network isolation
2. MTS is based on secure design
principles that addresses security
weakness of existing designs
3. MTS with SR-IOV offers security and
performance for modest resources
Security Performance Resource
HighHigh Mid
Our scripts and data are on githubwww.github.com/securedataplane
Backup
36
Protocol Growth for OvS
37
Complex & Manual Protocol Parsers
Virtual switches have to support an increasing number of protocols over time
38
Vswitch Table Analysis
39
So Many Virtual Switches
More than 20
40
So Many Virtual Switches
More than 20
41
So Many Virtual Switches
More than 20
42
Ingress Traffic Flow Example
43
VM$_
HOST$_
L2 Switch in NIC
TVFPF GW
VF
IN/OUTVF
VM$_
TVF
GWVF
IN/OUTVF
44
VM$_
HOST$_
L2 Switch in NIC
VM$_
Packet destined to VM $_
TVFPF GW
VF
IN/OUTVF
TVF
GWVF
IN/OUTVF
45
L2 Switch in NIC
VM$_
HOST$_
VM$_
MAC address of the
vswitch VF
IP address of VM $_
TVFPF GW
VF
IN/OUTVF
TVF
GWVF
IN/OUTVF
46
L2 Switch in NIC
VM$_
HOST$_
VM$_
TVFPF GW
VF
IN/OUTVF
TVF
GWVF
IN/OUTVF
47
L2 Switch in NIC
VM$_
HOST$_
VM$_
TVFPF GW
VF
IN/OUTVF
TVF
GWVF
IN/OUTVF
48
L2 Switch in NIC
VM$_
HOST$_
VM$_
TVFPF GW
VF
IN/OUTVF
TVF
GWVF
IN/OUTVF
49
L2 Switch in NIC
VM$_
HOST$_
VM$_
TVFPF GW
VF
IN/OUTVF
TVF
GWVF
IN/OUTVF
50
L2 Switch in NIC
HOST$_
VM$_
TVFPF GW
VF
IN/OUTVF
TVF
GWVF
IN/OUTVF
51
Pricing
52
How it Helps Pricing
Can charge for compute and memory used by the vswitch
53
Latency
54
Baseline vs MTSLatency Comparison
64 byte UDP packetsBaseline is faster than MTS in p2pMTS is faster than Baseline in p2v and v2v
55
Baseline vs MTSLatency Comparison
Baseline is faster than MTS in p2pMTS is faster than Baseline in p2v and v2v
56
Baseline vs MTSLatency Comparison
Baseline is faster than MTS in p2pMTS is faster than Baseline in p2v and v2v
57
Baseline vs MTSLatency Comparison
Baseline is faster than MTS in p2pMTS is faster than Baseline in p2v and v2v
58
