19
Kubernetes on OpenStack By Naveen Joy Cloud Architect

Kubernetes on open stack

Embed Size (px)

Citation preview

Page 1: Kubernetes on open stack

Kubernetes on OpenStack

ByNaveen Joy

Cloud Architect

Page 2: Kubernetes on open stack

Why Kubernetes?Provides a higher level abstraction to a lower level docker interfaceOrganize applications running in docker containers into PODs

PODs form the basic unit of operation POD == set{ one or more containers }

Users declare end state using a POD manifestScheduling mechanism for PODsContainers in a POD are tightly coupled i.e. co-located on a host and share network namespace, volumes and hostname

Page 3: Kubernetes on open stack

Why Kubernetes?Ability to group PODs using labelsEnable access to the POD group using a service abstraction (provides a stable service VIP)

The service will keep track of its PODs - endpoints of a serviceWhen traffic hits the service virtual IP, it will be proxied to one of the backend PODs

POD ManagementRestart a failed container in a POD automaticallySelf healing - ability to replace PODs when the machine failsHorizontal scaling

Page 4: Kubernetes on open stack

Architecture Overview

Page 5: Kubernetes on open stack

Networking Model

Page 6: Kubernetes on open stack

Default Networking Model in Docker• Host-Private Networking• Creates a virtual bridge named docker0 on each host• Allocates a private subnet (e.g. 172.17.0.0/16) from RFC 1918

for that bridge • Attaches each container to docker0 using a virtual ethernet

device • Assigns an IP from the private subnet to the container and sets

the bridge IP address is set as the gateway for the container

Page 7: Kubernetes on open stack

Default Docker Networking Model

Page 8: Kubernetes on open stack

Container reachability across hostsDocker may allocate the same IP addresses to containers across hosts

Containers can talk to each other on the same machine Containers cannot route traffic directly across hosts using their private IP address

Containers communicate across hosts by using DNAT

Host IP:Port To Container IP:Port

Page 9: Kubernetes on open stack

Default Networking model in docker can pose issues to Apps

Coordinating static port allocations to containers is very difficult in practice across multiple developers and groups that share hostsIf using dynamic port allocation, there are still complications

service discovery, application configuration etc.

NAT is hard to troubleshootApplication running in a container does not know its actual IP address – so some apps will break

apps that need to register their actual IP addressapps that perform IP based access control/authentication

Page 10: Kubernetes on open stack

Networking in Kubernetes

Containers communicate directly over a routed IP network without using NAT

A container sees the real IP of another containerThe host sees the real IP of the container

The default networking model of docker must be modified for Kubernetes to work

Page 11: Kubernetes on open stack

Networking in Kubernetes

• A routable IP address is assigned per POD

• All containers within a POD share the network namespace including the IP address and port• Implemented by creating a docker container

for the POD• This “pod-container” is wired to the POD IP• All other containers are configured to share

the network stack of the POD container using the --net=container:<name | id > function in docker

Page 12: Kubernetes on open stack

POD networking

• Each VM is assigned a subnet for POD networking (Note: This is in addition to the main neutron subnet used by the VM)

• The default docker bridge docker0 is replaced with a linux bridge say “cbr0”

• cbr0 is configured on the POD subnet• Docker daemon is started with this

bridge using --bridge=cbr0 in its options

• Docker allocates IPs to the containers from the POD subnet block

Page 13: Kubernetes on open stack

Routing POD SubnetsOption 1:

Create routable POD networks1. Configure instances to forward IP packets to the bridged POD

network by enabling IP forwarding in the kernel • sudo sysctl net.ipv4.ip_forward = 1

2. Add static routes on the L3 neutron gateway to route traffic to the instance• neutron router-update --routes type=dict list=true \

destination=NODE_X_POD_CIDR, \nexthop=NODE_X_INTERFACE_IP_ADDR

Page 14: Kubernetes on open stack

Routing POD SubnetsOption 1 :

3. When neutron security-groups is enabled, traffic is restricted to/from the instance IP address by neutron

• Add iptables FORWARD chain rules on the host to allow incoming and outgoing traffic to/from the POD CIDR

POD_CIDR=10.5.0.0/16

sudo iptables -I FORWARD 1 -p all -s $POD_CIDR –d $POD_CIDR -j ACCEPT

Page 15: Kubernetes on open stack

Option 1: Network Diagram

Page 16: Kubernetes on open stack

Routing POD SubnetsOption2:Build an overlay network to route POD networks• Proceed with caution for production deployment • These technologies are still in experimental stage

• Creates a layered virtual network architecture • Create POD virtual network overlay using the neutron virtual

networks as the underlay

• Open source options: • Flannel, Weave, Calico

Page 17: Kubernetes on open stack

Flannel• Designed for Kubernetes• Creates a POD subnet on each instance• Uses etcd to maintain the subnet to real host IP mapping• Builds an overlay mesh network between instances using

UDP tunneling to connect the subnets• Requires UDP port 8285 opened in the instance security

groups• Adjust the MTU size for performance

Page 18: Kubernetes on open stack

Option 2: Network Diagram

Page 19: Kubernetes on open stack

ConclusionCheckout the Kubernetes github repo

Latest docsContains several deployment examplesSaltStack scripts to automate a cluster deployment across multiple providers