Networking in Docker Containers

Page 1 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Networking in ContainersAttila Kanto


Agenda

• How networking works in Docker

• Container Network Model

• Networking plugin


Containers

• Isolate and package applications• Resources (CPU, memory, IO)• Namespaces (pid, users, network, uts, mnt )• Storage (device mapper, overlayfs, aufs, btrfs)• Security (capabilities)


Network

• UTS namespace• isolate hostname

• Network namespace• network interface(s)• loopback device• routing table• iptable rules


Basic networking overview

5


Networking without Docker

eth0

iptables

route



ifconfig

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.1.100 netmask 255.255.255.0 broadcast 192.168.1.255

ether 33:83:5a:44:50:ff txqueuelen 0 (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536

inet 127.0.0.1 netmask 255.0.0.0



ifconfig

eth0:

inet 192.168.1.100

ether 33:83:5a:44:50:ff

OSI Layers (1 – 4)



route -n

Destination Gateway Genmask Iface

0.0.0.0 192.168.1.1 0.0.0.0 eth0

192.168.1.0 0.0.0.0 255.255.255.0 eth0

iptables -t nat -L

target prot opt source destination



eth0

iptables

route192.168.1.0/24 -> eth00.0.0.0 -> 192.168.1.1 (eth0)

192.168.1.100


Networking with Docker

11


Install Docker

eth0

iptablesMASQUERADE 172.17.0.0/16

route192.168.1.0/24 -> eth00.0.0.0 -> 192.168.1.1 (eth0)172.17.0.0/16 -> docker0

192.168.1.100 172.17.0.1

docker0


Run container / bridged networking

• Docker0 bridge• already there, created during install

• Network namespace• container netns needs to be created

• Veth pair• created during the creation of container• connects two network namespaces

• External communication• Only through Network Address Translation (NAT)


Run container / bridged networking / 8080 -> 9090

eth0

iptablesMASQUERADE 172.17.0.0/16DNAT dpt:9090 to:172.17.0.2:8080


192.168.1.100 172.17.0.1

docker0

container1ns

eth0vxxveth

172.17.0.2

route

SRC DSTClient Port 9090Client IP 192.168.1.100Client MAC MAC of eth0

SRC DSTClient Port 8080Client IP 172.17.0.2

SRC DSTClient Port 8080Client IP 172.17.0.2

MAC of docker0 MAC of eth0


Overlay networking with Docker

15


Run container / overlay networking• Bridges

• docker_gwbridge created if does not exist• br0 in a “hidden” namespace associated with the overlay network

• Network namespace• container netns needs to be created

• Veth pairs• connects br0 and and eth0 of container• connects docker_gwbridge and eth1 of container

• External communication• Through Network Address Translation (NAT)• Through VXLAN (other container using the same overlay network)


Install Docker (again)

eth0

iptablesMASQUERADE 172.17.0.0/16


192.168.1.100 172.17.0.1

docker0


Run container / overlay networking

eth0

iptables

route

192.168.1.100

172.18.0.1

docker_gw

container1ns

eth1vxx veth172.18.0.2

172.17.0.1

docker0

ns

br0 eth0vyyveth

10.10.10.210.10.10.1

VXLAN

route


Software-defined networking (SDN)

• Separation control and data plane of network

• Control plane• makes decisions about where traffic is sent

• Data plane• forward traffic to the selected destination


Data Plane (in Docker overlay)

• Virtual Extensible LAN (VXLAN)• overlay technology• encapsulates L2 frames as UDP packets

• VTEP – VXLAN Tunnel End Point• originator and/or terminator of VXLAN tunnel

• VNI – VXLAN Network Identifier• part of the VXLAN Header• similar to VLAN ID


Data Plane (in Docker overlay)

• Container sends a packet• ARP (neighbor) table is checked for destination container IP -> MAC

interface mapping• L2 FDB (forwarding database) is checked to determine IP of destination

VTEP for destination MAC on source VTEP• packet is encapsulated for destination VTEP with configured VNI and sent

to destination• destination VTEP de-capsulates the packet • inner packet is received by the destination container


Network Control Plane (in Docker overlay)


Container Network Model

23


Container Network Model (CNM)• Sandbox

• holds the config of a container's network stack (DNS, routing, etc.)• multiple endpoints from multiple networks• Linux Network Namespace / FreeBSD Jail

• Network• Group Endpoints that are able to communicate with each-other directly• Linux Bridge / VXLAN

• Endpoint• joins Sandbox to Network• veth pair / ovs patch port


Docker libnetwork

• Docker’s networking library• Implements CNM• Built-in drivers (in process)• Network drivers (bridge, overlay)• IPAM drivers

• Plugin mechanism (off process)• External Network drivers (Calico, Midonet, my own driver) • External IPAM drivers


Libnetwork plugins

• Implemented using libnetwork’s remote driver• Running off-process (not in Docker daemon)• HTTP POSTs with JSON payload• KV store API not exposed• can be implemented in any programming language

• KV store• KV url / credentials needs to be passed in init time

• Can be deployed as container


Network plugin API (Network)

• CreateNetwork

• DeleteNetwork


Network plugin API (Endpoint)

• CreateEndpoint

• DeleteEndpoint


Network plugin API (Join)

• Join

• Join (resp)


Floating IP network driver

• Containers on same L2 network• Connected with Open vSwitch• IP Address Management• libnetwork built-in IPAM driver is used

• Externally addressable IP / container• no Network Address Translation• no port collision• extremely fast• scalability


Run container / floating driver

iptables

route

192.168.1.100

container1ns

172.17.0.1

docker0

floating_bridge

eth0

192.168.10.2

eth1 veth2veth1

container2ns

eth0

192.168.10.3

veth veth

eth0


Demo

32


How to use it in Hadoop world

• Using multiple networks• overlay to create internal network• floating for exposing servers

Data Node

Data Node

Data Node

Ambari

Master Node

Data Node

Data Node

Data Node

Master Node

Edge Node

OverlayFloating


Takeaways

• Since 1.9 Docker networking has improved• Easy to write a plugin that does certain things better• Multiple networks can be used by the same container• Not everybody is happy with it

• Kubernetes http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-use-libnetwork.html• Mesos https://issues.apache.org/jira/browse/MESOS-3828


We are hiring!

35