Kubernetes HAMontreal Kubernetes Meetup
October 12
Hello, my name is Alexandre
@alex_gervais
alexgervais
AppDirect background
- Chef provisioning
- Centos 7
- Multiple deployments
- AWS
- On-premise
- Automation, automation, automation!
- Packer
- Terraform
- CI
- In production since k8s 1.1
- Rolling upgrade every minor release
- Experimenting since early beta
Although it is easy to deploy and make your applications and micro-services highly available within a Kubernetes cluster, Kubernetes masters are not HA in typical
setups.
It requires a little more work, but not that much…
Here’s the 3-step program.
0. Single master
1.etcd clustering
$ curl https://discovery.etcd.io/new?size=3
2. Master election
podmaster and hyperkube
On every master node:
/etc/kubernetes/manifests/podmaster.yamlgcr.io/google_containers/podmaster:1.1
/srv/kubernetes/kube-controller-manager.yamlgcr.io/google_containers/hyperkube:1.4.0
/srv/kubernetes/kube-scheduler.yamlgcr.io/google_containers/hyperkube:1.4.0
On the elected node:
The podmaster will copy kube-controller-manager.yaml and kube-scheduler.yaml to /etc/kubernetes/manifests and kubelet picks them up!
Disclaimer
Since kubernetes 1.2
--leader-elect
--apiserver-count=3
3. API load balancing
🎉
$ kubectl get po --namespace=kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODEkube-addon-manager-ip-172-31-29-97.ec2.internal 1/1 Running 1 40d 172.31.29.97 ip-172-31-29-97.ec2.internalkube-controller-manager-ip-172-31-29-97.ec2.internal 1/1 Running 1 40d 172.31.29.97 ip-172-31-29-97.ec2.internalkube-dns-v19-5ut0y 3/3 Running 3 40d 10.0.55.2 ip-172-31-51-130.ec2.internalkube-dns-v19-srphp 3/3 Running 0 13d 10.0.50.5 ip-172-31-46-232.ec2.internalkube-dns-v19-tf5u6 3/3 Running 1 33d 10.0.20.3 ip-172-31-29-97.ec2.internalkube-scheduler-ip-172-31-29-97.ec2.internal 1/1 Running 1 40d 172.31.29.97 ip-172-31-29-97.ec2.internalkubernetes-dashboard-v1.1.0-zta4y 1/1 Running 0 40d 10.0.55.5 ip-172-31-51-130.ec2.internalpodmaster-ip-172-31-29-97.ec2.internal 3/3 Running 3 40d 172.31.29.97 ip-172-31-29-97.ec2.internalpodmaster-ip-172-31-52-169.ec2.internal 3/3 Running 6 33d 172.31.52.169 ip-172-31-52-169.ec2.internalpodmaster-ip-172-31-7-176.ec2.internal 3/3 Running 3 40d 172.31.7.176 ip-172-31-7-176.ec2.internal
$ kubectl get ep
NAME ENDPOINTS AGEkubernetes 172.31.29.97:6443,172.31.52.169:6443,172.31.7.176:6443 40d
Cluster-wide upgrades
- Chef(ing)
- Rolling upgrades of existing nodes
- Terraform(ing)
- Replace nodes, one-by-one
- Datadog monitoring
References
- etcd clusteringhttps://coreos.com/etcd/docs/latest/clustering.html
- hyperkubehttps://github.com/kubernetes/kubernetes/tree/master/cluster/images/hyperkube
- Master node deploymentshttps://coreos.com/kubernetes/docs/latest/deploy-master.html
- Kubernetes HA recipehttp://kubernetes.io/docs/admin/high-availability/
AppDirect Shameless Plug