Kubernetes in kubernetes 搭建高可用環境

Kubernetes in Kubernetes：搭建高可用環境

資訊與通訊研究所蔣是文 Mac Chiang

交通大學資工所鄭偉聖 Sam Zheng

Copyright 2017 ITRI 工業技術研究院

Agenda

• Cluster Management Issues

• Self-hosted Kubernetes

• Service with High Availability

• Summary

2


Microservices Challenges

3

• A lots of microservices/components

• Zero downtime deployment

• Incremental roll out of features faster

• Improves the ability scale efficiently


Kubernetes Cluster

4

• Scheduling

• Deployment

• Healing

• Discovery/Load balancing

• Scaling

But how about operating Kubernetes?


Kubernetes Operation Tasks

5

• Deployment

• Upgrade / rollback

• Scaling

• Monitoring

• Healing

• Security

• …

• A lot of manual/semi-manual work

• No standard way to approach all the problems

Problems!

http://www.infoq.com/cn/presentations/self-hosted-infrastructure-take-kubernetes-as-an-example


What’s Self-hosted Kubernetes?

6

• Running all required and optional components of a Kubernetes cluster on top of Kubernetes itself

• Kubernetes manages own core components

• Core component deployed as native API objects

https://www.youtube.com/watch?v=EbNxGK9MwN4


Why Self-hosted Kubernetes?

7

• Small Dependencies▪ Reduce the number of components required (Kubelet and Docker)

• Deployment consistency▪ Reduce the number of moving parts relying on the host OS

• Introspection▪ Can be debugged and inspected by users using existing Kubernetes APIs

• Cluster Upgrades▪ Upgrade the components via Kubernetes APIs

• Easier Highly-Available Configurations▪ Easier to scale up and monitor an HA environment without complex external tooling

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/self-hosted-kubernetes.md


Launching a Self-hosted Cluster

8

• Need an initial control plane to bootstrap a self-hosted cluster

• Bootkube▪ Provides a temporary control plane to run a full blown self-hosted control

plane

▪ Run only on very first node, then not needed again

https://www.youtube.com/watch?v=EbNxGK9MwN4


Kubernetes Architecture

9

Master node

etcd

SchedulerController

Manager

API Server

Worker node

Kubelet Kube-proxy

dockerPod Pod

Worker node

Kubelet Kube-proxy

dockerPod Pod


Spectrum of Self-hosted Ways

10

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/self-hosted-kubernetes.md

• 0-4 self-hosted cluster

• kubelet: daemon set

• API server: pod / service


• system installed Kubelet


• everything except etcd self-hosted


How bootkube works?

11

etcd

Bootkube

API Server

Scheduler

Controller

Manager

Kubelet


How bootkube works? (cont.)

12

etcd

Bootkube

API Server

Scheduler

Controller

Manager

Kubelet

Create

• Deployment

• Daemonset

• Service

• Secret

kubectl



13

etcd

Bootkube

API Server

Scheduler

Controller

Manager

Kubelet

Pods

API Server

Scheduler

Controller

Manager

create



14

etcd

Bootkube

API Server

Scheduler

Controller

Manager

Kubelet

Pods

API Server

Scheduler

Controller

Manager

create


Pods


15

etcd

Kubelet

API Server

Scheduler

Controller

Manager


Self-hosted Control Plane

16

[root@centos7 ~]# kubectl get pod -n kube-system

NAME READY STATUS RESTARTS AGE

kube-apiserver-kkxq9 1/1 Running 0 1d

kube-controller-manager-2953862963-t7m1q 1/1 Running 0 1d

kube-controller-manager-2953862963-wlsjp 1/1 Running 0 1d

kube-dns-2431531914-gqnnd 3/3 Running 0 1d

kube-flannel-wnk1j 2/2 Running 0 1d

kube-flannel-xcsx2 2/2 Running 0 1d

kube-flannel-xrksj 2/2 Running 1 1d

kube-proxy-04x11 1/1 Running 0 1d

kube-proxy-11n6t 1/1 Running 0 1d

kube-proxy-1zlgz 1/1 Running 0 1d

kube-scheduler-1873817829-4c7mm 1/1 Running 1 1d

kube-scheduler-1873817829-pmp0n 1/1 Running 0 1d

pod-checkpointer-11q7g 1/1 Running 0 1d

pod-checkpointer-11q7g-10.201.3.6 1/1 Running 0 1d


Self-hosted Control Plane (cont.)

17

[root@centos7 ~]# kubectl get deployment -n kube-system

NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE

kube-controller-manager 2 2 2 2 2d

kube-dns 1 1 1 1 2d

kube-scheduler 2 2 2 2 2d

[root@centos7 ~]# kubectl get svc -n kube-system

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kube-dns 10.3.0.10 <none> 53/UDP,53/TCP 2d

[root@centos7 ~]# kubectl get ds -n kube-system

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE

kube-apiserver 1 1 1 1 1 node-role.kubernetes.io/master= 2d

kube-flannel 3 3 3 3 3 <none> 2d

kube-proxy 3 3 3 3 3 <none> 2d

pod-checkpointer 1 1 1 1 1 node-role.kubernetes.io/master= 2d

[root@centos7 ~]# kubectl get secret -n kube-system

NAME TYPE DATA AGE

kube-apiserver Opaque 7 2d

kube-controller-manager Opaque 2 2d


Disaster Recovery

18

• Node failure in HA deployments (Kubernetes)

• Partial loss of control plane components (Kubernetes)

• Power cycling the entire control plane (Kubernetes)

• Permanent loss of control plane (External tool)

http://www.infoq.com/cn/presentations/self-hosted-infrastructure-take-kubernetes-as-an-example


Service with High Availability

19


Kubernetes Networking

20

Copyright 2017 ITRI 工業技術研究院 21



How do we to export the service IP to public network on bare metal?

• In kubernetes we have some existed solution

NodePort

CloudProvider Load Balancer

Ingress

• But it is enough?

23


NodePort

24

kubeProxy kubeProxy kubeProxy

NodePort NodePort NodePort

Pod Pod Pod


NodePort

25



Pod Pod Pod


Load Balancer

• Cloud Provider e.g. AWS , GCP, OpenStack

• Load Balancer is created by Cloud Provider, and provide the external IP to for service

• But it is only for Cloud Provider, the bare metal cannot do this

26


Load Balancer

27



Cloud

LoadBalancer

Pod Pod Pod



Ingress

29

• HTTP Load Balancing

• SSL Termination

• Content-base Routing

• Not fully for external network


Ingress

30

Ingress

nginx.com echoheaders.com

ServiceA ServiceB

10.0.0.1:80


Ingress

31

Ingress

Controller

Ingress

ResourceLoad Balancer

watches configure


Ingress

32

kubeProxykubeProxy kubeProxy

Pod PodPodPod PodPod

Momo.foo.com

Jojo.foo.comyoyo.foo.com

hostnetwork


Ingress

33

kubeProxykubeProxy kubeProxy

Pod PodPodPod PodPod

DNS



Keep-Alived VIP

• Real High-Availability

• Virtual IP Address

• IP to Service

• Configmap

• DaemonSet

35

Ref :

- https://github.com/kubernetes/contrib/tree/master/keepalived-vip


Keep-Alived VIP

36


Pod PodPod

Keepalived Keepalived Keepalived

140.113.1.1 140.113.1.2 140.113.1.3

Vip: 140.113.1.50


Keep-Alived VIP

37


PodPodPod


140.113.1.1 140.113.1.2 140.113.1.3

Vip: 140.113.1.50

Pod


Keep-Alived VIP

38


PodPodPod


140.113.1.1 140.113.1.2 140.113.1.3

Vip: 140.113.1.50



Keep-Alived VIP + Ingress

40


PodPod Pod


140.113.1.1 140.113.1.2 140.113.1.3

Vip: 140.113.1.50

Momo.foo.com

Jojo.foo.com


Summary

• Self-hosted K8S ▪ Make K8S operations more easier

▪ Bootkube is still a incubator project

▪ Support disaster recovery

• Service with High Availability▪ Using DNS to provide your service – Ingress

▪ More vip – keepalived-VIP

▪ Using cloud to build your kubernetes – cloud Load Balancer

▪ Test – NodePort

41

Thank [email protected]

[email protected]

Kubernetes Taiwan User Group

Technology

Kubernetes in kubernetes 搭建高可用環境