Kubernetes Configuration & BestPractices

2020-01-28

Table of ContentsInstallation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  2

Google Kubernetes Engine (GKE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  2AWS / EKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  2AWS / Kubespray / Terraform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  12

Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  17Kubectl autocompletion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  17Kubectl aliases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  18Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  18Useful commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  21ClusterIP vs NodePort vs Ingress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  21Restart deployment pods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  21Getting logged in to ECR for image pulling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  22Force updating image with same tag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  22Example application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  24

Helm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  24Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  24Initialization for current cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  24Chart installation example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  25Chart uninstallation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  25Multi-tenant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  25

NGINX Ingress Controller. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  25Cert manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  25

Istio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  26Prerequisites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  26Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  26Activation for a namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  27UIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  27

Powerfulseal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  28Troobleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  29

User "system:anonymous" cannot get resource. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  29Full project example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  30

Target architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  30Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  30Deployments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  30

Table 1. History

Date Author Detail

2020-01-28 bcouetil Ὅ� (all) refactor detail tag to new collapsible asciidoc feature

2020-01-28 bcouetil Ὅ� (k8s) kubectl-debug

2020-01-28 bcouetil Ὅ� (core) update theme

2020-01-28 bcouetil Ὅ� (core) remove sample and build with docker

2020-01-21 BenoîtCOUETIL

Ὅ� add microk8s and git branch clean

2019-12-15 BenoîtCOUETIL

Ὅ� (doc) add asciidoc docker generation sample on gitlab + add pipeline sh scripts

2019-10-24 bcouetil Ὅ� added asciidoctor generation using docker + gitlab deploy to k8s / aws

2019-07-23 bcouetil k8s updates + gitlab mono-repo yarn + openiconic

2019-05-09 bcouetil eks terraform+ hyperledger + k8s aliases + cert manager + k8s aws anonymous+ k8s tools + gerrit replication

2019-04-22 bcouetil eks dashboard & deployment scale + unselectable $ before shell commands +fully working spotlight

2019-04-11 bcouetil k8s & aws-eks merge + k8s aws dashboard + k8s istio + jdk8 in CI + gitlab &nexus following lacroix exp + new backgrounds

2019-03-12 bcouetil asciidoc/gradle + new backgrounds + meme extension + k8s/aws/kubespraydoc following lacroix experience + aws/eks/users/ci

2018-12-18 bcouetil Moved Gerrit plugin configuration from Jenkins to Gerrit page + EnhancedHTML CSS + Cropped some images + Improved PlantUML skin

2018-11-13 bcouetil - Updated sample project with Reveal.js generation - Duplicated Reveal.jsexecution to have multiple styles - Compromised layout between 4/3 and 16/9- Minor changes in Reveal.js css - Added some web comics

2018-09-19 bcouetil - Sample asciidoctor maven project published on Github - Github & LinkedInlinks - Sample project tree - new images + resizing and positioning

2018-08-29 bcouetil Asciidoc HTML look & feel changes

2018-08-24 bcouetil Icones added for download + favicon added for webpage

1

InstallationGoogle Kubernetes Engine (GKE)Create an account on GCP and follow any tutorial, for example this video workshop.

AWS / EKSHave an admin account on AWS.

Manual installation

• Good to do once to understand every steps

• You can switch menu language at the bottom left of any page. Select english.

Follow Getting Started with Amazon EKS of Official documentation to :

• Create your Amazon EKS Service Role

• Create your Amazon EKS Cluster VPC

• Install and Configure kubectl for Amazon EKS

• Download and Install the Latest AWS CLI

• Create Your Amazon EKS Cluster

• Configure kubectl for Amazon EKS

• Launch and Configure Amazon EKS Worker Nodes

Terraform installation

To destroy the cluster : terraform destroy --force

2

https://www.youtube.com/watch?v=ZpbXSdzp_vohttps://docs.aws.amazon.com/eks/latest/userguide/getting-started.html

If needed, install Terraform using this guide :

cd /tmp/wget https://releases.hashicorp.com/terraform/0.12.6/terraform_0.12.6_linux_amd64.zipunzip terraform_0.12.6_linux_amd64.zipsudo mv terraform /usr/local/bin/terraform --version

Official repository

Using the official terraform/eks repo as a module, without cloning it.

• Create the main config file

3

https://askubuntu.com/questions/983351/how-to-install-terraform-in-ubuntuhttps://github.com/terraform-aws-modules/terraform-aws-eks

Example 1. main.tf

terraform {  required_version = ">= 0.11.8"}

provider "aws" {  version = ">= 2.0.0"  region = "${local.region}"}

# data "aws_region" "current" {}data "aws_availability_zones" "available" {}

locals {  cluster_name = "my-sweet-cluster"  region = "eu-north-1"

  worker_groups = [  {  instance_type = "t3.small" # 2CPU, 2GO RAM. t2 does not exist in Stockholm  asg_desired_capacity = "1" # Desired worker capacity in the autoscaling group.  asg_max_size = "5" # Maximum worker capacity in the autoscaling group.  asg_min_size = "1" # Minimum worker capacity in the autoscaling group.  autoscaling_enabled = true # Sets whether policy and matching tags will be added to allow autoscaling.  # spot_price = "" # "0.01" or any value to use "spot" (cheap but can leave) instances  },  ]

  map_users = [  {  user_arn = "arn:aws:iam:::user/my.user.one"  username = "my.user.one"  group = "system:masters"  },  {  user_arn = "arn:aws:iam:::user/my.user.one"  username = "my.user.two"  group = "system:masters"  },  ]

  map_users_count = 6

  tags = {  Environment = "POC"  creation-date = "${timestamp()}"  }}

module "vpc" {  source = "terraform-aws-modules/vpc/aws"  name = "my-sweet-cluster-vpc"  cidr = "10.0.0.0/16"  # azs = ["${local.region}a", "${local.region}b", "${local.region}c"]  azs = ["${data.aws_availability_zones.available.names[0]}","${data.aws_availability_zones.available.names[1]}", "${data.aws_availability_zones.available.names[2]}"]  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]  public_subnets = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]  enable_nat_gateway = true  single_nat_gateway = true  tags = "${merge(local.tags, map("kubernetes.io/cluster/${local.cluster_name}", "shared"))}"}

module "eks" {  source = "terraform-aws-modules/eks/aws"  cluster_name = "${local.cluster_name}"  subnets = ["${module.vpc.private_subnets}"]

4

  tags = "${local.tags}"  vpc_id = "${module.vpc.vpc_id}"  worker_groups = "${local.worker_groups}"  map_users = "${local.map_users}"  map_users_count = "${local.map_users_count}"  worker_sg_ingress_from_port = "0" # default 1025, which means no POD port exposed below 1024}

• Create a minimal output file

Example 2. outputs.tf

output "cluster_endpoint" {  description = "Endpoint for EKS control plane."  value = "${module.eks.cluster_endpoint}"}

output "cluster_security_group_id" {  description = "Security group ids attached to the cluster control plane."  value = "${module.eks.cluster_security_group_id}"}

output "kubectl_config" {  description = "kubectl config as generated by the module."  value = "${module.eks.kubeconfig}"}

output "config_map_aws_auth" {  description = ""  value = "${module.eks.config_map_aws_auth}"}

• If you configured autoscaling usage, create the associated config file

Example 3. autoscaler.yml

## Config values specific to AWS/EKS# see https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/autoscaling.md#

rbac:  create: true

sslCertPath: /etc/ssl/certs/ca-bundle.crt

cloudProvider: awsawsRegion: eu-north-1

autoDiscovery:  clusterName: my-sweet-cluster  enabled: true

• If not already done, configure AWS CLI

aws configure

• Create the cluster. It should take about 10min.

5

terraform initterraform apply

• Configure kubeconfig

◦ Either with terraform output

terraform output kubeconfig > ~/.kube/my-sweet-clusterexport KUBECONFIG=~/.kube/my-sweet-cluster

• Or with aws eks

aws eks update-kubeconfig --name my-sweet-cluster

• If you configured it to use the autoscaler

◦ Install and initialize helm as described in [helm].

◦ Apply the helm chart

helm install stable/cluster-autoscaler --values=autoscaler.yml --name cas --namespace kube-system

• Test the autoscaler

◦ Scale up

kubectl run example --image=nginx --port=80 --replicas=50kubectl logs -l "app.kubernetes.io/instance=cas" -fkubectl get nodes -w

• Scale down

kubectl delete deployment examplekubectl logs -l "app.kubernetes.io/instance=cas" -f

After 10 minutes (by default), the cluster should scale down and you should see

I0423 12:18:52.539729 1 scale_down.go:600] ip-10-0-3-163.eu-north-1.compute.internal was unneeded for10m8.928762095sI0423 12:18:52.539815 1 scale_down.go:600] ip-10-0-3-149.eu-north-1.compute.internal was unneeded for10m8.928762095sI0423 12:18:52.539884 1 scale_down.go:600] ip-10-0-1-206.eu-north-1.compute.internal was unneeded for10m8.928762095sI0423 12:18:52.539947 1 scale_down.go:600] ip-10-0-1-222.eu-north-1.compute.internal was unneeded for10m8.928762095sI0423 12:18:52.540077 1 scale_down.go:819] Scale-down: removing empty node ip-10-0-3-163.eu-north-1.compute.internalI0423 12:18:52.540190 1 scale_down.go:819] Scale-down: removing empty node ip-10-0-3-149.eu-north-1.compute.internalI0423 12:18:52.540261 1 scale_down.go:819] Scale-down: removing empty node ip-10-0-1-206.eu-north-1.compute.internalI0423 12:18:52.540331 1 scale_down.go:819] Scale-down: removing empty node ip-10-0-1-222.eu-north-1.compute.internal

6

Alternative repository

Smaller option list, easier to understand

• clone this working repo

git clone https://github.com/WesleyCharlesBlake/terraform-aws-eks.git

• check that this issue is merged, else apply the changes locally

• create a configuration file at root and change values if needed. See default values in variables.tf or onGithub page.

terraform.tfvars

cluster-name = "my-sweet-cluster"k8s-version = "1.12"aws-region = "eu-west-1"node-instance-type = "t2.medium"desired-capacity = "1"max-size = "5"min-size = "0"

• Configure AWS to the right account

pip install --upgrade awscliaws configure

• Create the cluster. It should take about 10min.

terraform apply

• Configure kubeconfig

terraform output kubeconfig > ~/.kube/trekea-clusterexport KUBECONFIG=~/.kube/trekea-cluster

or

aws eks update-kubeconfig --name my-sweet-cluster

• Make the workers join the cluster and watch

terraform output config-map > config-map-aws-auth.yamlkubectl apply -f config-map-aws-auth.yamlkubectl get nodes --watch

Administration

Initial admin

For a user / maintainer of the cluster, here are the pre-requisites :

7

https://github.com/WesleyCharlesBlake/terraform-aws-ekshttps://github.com/WesleyCharlesBlake/terraform-aws-eks/pull/16

• Install kubectl

• Install aws-iam-authenticator

• Install AWS CLI

• Check installation

kubectl versionaws-iam-authenticator helpaws --version

• Configure AWS and kubectl

create-access-key --user-name aws configureaws eks --region update-kubeconfig --name

Additional admins

To grant cluster rights to other admins than the cluster creator, do the following.

IAM

Go to AWS console / IAM to give EKS/ECR admin rights to the user(s).

Policies

Default policies do not cover EKS and ECR admin usage (!), so we create some custom policies.

• Create EKS policies with [Policies] → [Create policy]

◦ Service = EKS

◦ Action = All EKS actions

◦ Resources = All resources

• Click [Review policy]

◦ Name = EKS-admin

• Create EKS policies with [Policies] → [Create policy]

◦ Service = ECR

◦ Action = All ECR actions

◦ Resources = All resources

◦ Name = ECR-admin

Groups

• Create a group with [Policies] → [Create policy]

◦ Group Name = MyDevTeam

◦ Choose policies :

▪ IAMSelfManageServiceSpecificCredentials

▪ IAMFullAccess

▪ IAMUserChangePassword

8

https://kubernetes.io/docs/tasks/tools/install-kubectl/https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.htmlhttps://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

▪ IAMUserSSHKeys

▪ EKS-admin

▪ ECR-admin

◦ Click [Next] then [Create Group]

Users

If needed, create the user in AWS console / IAM. Copy the userarn for next step.

Attach the group to the user by clicking on the user → [Groups] → [Add user to groups] → selectMyDevTeam → [Add to Groups].

Configmap

The actual admin has to update the configmap with the new user.

kubectl edit -n kube-system configmap/aws-auth

apiVersion: v1data:  mapRoles: |  - rolearn: arn:aws:iam:::role/adx-worker-nodes-NodeInstanceRole-XXXXX  username: system:node:{{EC2PrivateDNSName}}  groups:  - system:bootstrappers  - system:nodes  mapUsers: |  - userarn: arn:aws:iam:::user/my-user  username: my-user  groups:  - system:masters  - userarn: arn:aws:iam:::user/another-user ①  username: another-user  groups:  - system:masterskind: ConfigMap[...]

① Add user(s) to the list in mapUsers

Modification is asynchronous, you may have to wait a few minutes for this to be takeninto account

Tools configuration

You now have the rights, you can then perform operations described in Initial admin.

Kubernetes dashboard

Follow official guide :

9

https://docs.aws.amazon.com/eks/latest/userguide/dashboard-tutorial.html

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yamlkubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yamlkubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yamlkubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yamlkubectl apply -f https://raw.githubusercontent.com/ericdahl/hello-eks/master/k8s/dashboard/eks-admin-service-account.yamlkubectl apply -f https://raw.githubusercontent.com/ericdahl/hello-eks/master/k8s/dashboard/eks-admin-cluster-role-binding.yamlkubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}')

This gets you a token for connecting to a local redirection of the dashboard when typing

kubectl proxy

CPU / RAM

You may not have CPU and RAM on EKS as of late 2018 / early 2019, see this issue for explanation.

• check heapster logs

kubectl get allkubectl logs heapster-***

E1228 12:13:05.074233 1 manager.go:101] Error in scraping containers from kubelet:10.0.30.39:10255: failed to getall container stats from Kubelet URL "http://10.0.30.39:10255/stats/container/": Posthttp://10.0.30.39:10255/stats/container/: dial tcp 10.0.30.39:10255: getsockopt: connection refused

• Fix the deployment by editing the yml to add some extra parameters to --source

kubectl edit deployment heapster

10

http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/loginhttps://github.com/awslabs/amazon-eks-ami/issues/128

spec:  template:  spec:  containers:  - command:  - /heapster  ---source=kubernetes:kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true ①  - --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086  name: heapster

• Add some ClusterRole & ClusterRoleBinding by creating this file

heapster-node-stats.yml

kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata:  name: node-stats-fullrules:- apiGroups: [""]  resources: ["nodes/stats"]  verbs: ["get", "watch", "list", "create"]---kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata:  name: heapster-node-statssubjects:- kind: ServiceAccount  name: heapster  namespace: kube-systemroleRef:  kind: ClusterRole  name: node-stats-full  apiGroup: rbac.authorization.k8s.io

• And applying it

kubectl apply -f heapster-node-stats.yml

Error logs should stop. After a few minutes, with enough data, the dashboard should show CPU / RAM inAll namespace list. Else delete the kubernetes-dashboard-* to restart it.

CI pipelines

Full automation requires a CI technical user being able to interract with the cluster. For this :

• Create a technical user in IAM

• Install, configure and use AWS CLI, aws-iam-authenticator and kubectl in your pipeline

Bitbucket

For Bitbucket, we use an AWS docker image. First add some configuration by going to your repository →[Settings] → [Repository variables]

11

AWS_ACCOUNT_ID 777777777777AWS_DEFAULT_REGION eu-west-3AWS_ACCESS_KEY_ID XXXXXXXAWS_SECRET_ACCESS_KEY tPbfRc/wx3JmPp6XXXXXXXty2yFJ6wl4rZ0B/Q

Then define the pipeline

bitbucket-pipelines.yml

- step: &deploy-to-develop-k8s  name: Deploy to Develop on Kubernetes cluster  image: atlassian/pipelines-awscli  max-time: 5  services:  - docker  script:  - export REPOSITORY_URL=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com  # Download the necessary tools to deploy to kubernetes  - apk add --no-cache curl  - curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -shttps://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl  - chmod +x ./kubectl  - mv ./kubectl /usr/local/bin/kubectl  # Download aws-iam-authenticator  - curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator  - chmod +x ./aws-iam-authenticator  - mkdir $HOME/bin && cp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator && export PATH=$HOME/bin:$PATH  - echo 'export PATH=$HOME/bin:$PATH' >> ~/.bashrc  - aws eks update-kubeconfig --name ${KUBERNETES_CLUSTER_NAME}  - kubectl set image deployment/api-dpl api=${REPOSITORY_URL}/adx/adx-api:develop  - kubectl set image deployment/client-dpl client=${REPOSITORY_URL}/adx/adx-client:develop

AWS / Kubespray / Terraform

Prerequisites

• Install pip3 (Python 3) or pip (Python 2)

sudo apt updatesudo apt install python3-pippip3 --version

• clone kubespray repository

git clone https://github.com/kubernetes-sigs/kubespray.git

• Install the requirements using pip3

pip3 install -r requirements.txt

• This installs Ansible but not Terraform, which we will use to generate the hosts.ini file used by Ansible.Let’s install it following this guide.

12

https://askubuntu.com/questions/983351/how-to-install-terraform-in-ubuntu

cd /tmp/wget https://releases.hashicorp.com/terraform/0.11.11/terraform_0.11.11_linux_amd64.zipunzip terraform_0.11.11_linux_amd64.zipsudo mv terraform /usr/local/bin/terraform --version

• Create a key pair in AWS Console

◦ [Services] → [Network & Security] → [Key Pairs] → [Create Key Pair]

◦ Save it and change rights

cp my-private-key.pem ~/.sshchmod 700 ~/.ssh/my-private-key.pem

Terraform : EC2 servers and host.ini creation

We will use Terraform to generate the host.ini file.

• Go to your cloned kubespray project

• Create the file credentials.tfvar

contrib/terraform/aws/credentials.tfvar

#AWS Access KeyAWS_ACCESS_KEY_ID = "XXXXXXXXX"#AWS Secret KeyAWS_SECRET_ACCESS_KEY = "YYYYYYYYYYYYYYYYYYY"#EC2 SSH Key NameAWS_SSH_KEY_NAME = "my-key-pair-name"#AWS RegionAWS_DEFAULT_REGION = "eu-west-3"

• Copy the file terraform.tfvars.example to create one that suits your needs

13

contrib/terraform/aws/terraform.tfvars

#Global Varsaws_cluster_name = "mycluster"

#VPC Varsaws_vpc_cidr_block = "10.250.192.0/18"aws_cidr_subnets_private = ["10.250.192.0/20","10.250.208.0/20"]aws_cidr_subnets_public = ["10.250.224.0/20","10.250.240.0/20"]

#Bastion Hostaws_bastion_size = "t2.medium"

#Kubernetes Cluster

aws_kube_master_num = 1aws_kube_master_size = "t2.medium"

aws_etcd_num = 1aws_etcd_size = "t2.medium"

aws_kube_worker_num = 1aws_kube_worker_size = "t2.medium"

#Settings AWS ELB

aws_elb_api_port = 6443k8s_secure_api_port = 6443kube_insecure_apiserver_address = "0.0.0.0"

default_tags = {  Env = "mycluster"  App = "mycluster-my-app"  Product = "kubernetes"}

inventory_file = "../../../inventory/mycluster/hosts.ini"

• Optional : if you want Ubuntu/Debian images for your cluster (instead of CoreOS), change this invariables.tf

contrib/terraform/aws/variables.tf

data "aws_ami" "distro" {  most_recent = true

  filter {  name = "name"  values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-*"]  # values = ["debian-stretch-hvm-x86_64-gp2-2018-11-10-63975-572488bb-fc09-4638-8628-e1e1d26436f4-ami-0f4768a55eaaac3d7.4"]  }

  filter {  name = "virtualization-type"  values = ["hvm"]  }

  owners = ["099720109477"] #ubuntu  # owners = ["679593333241"] #debian}

• Initialize inventory folder (where host.ini will be written in next step)

mv inventory/sample inventory/mycluster

14

• Initialize Terraform and apply

cd contrib/terraform/awsterraform initterraform plan --var-file=credentials.tfvarsterraform apply --var-file=credentials.tfvars

• Edit the generated host.ini file to put the internal DNS instead of generated names (see names in AWSconsole)

inventory/mycluster/host.ini

[all]ip-10-250-206-126.eu-west-3.compute.internal ansible_host=10.250.206.126ip-10-250-201-250.eu-west-3.compute.internal ansible_host=10.250.201.250ip-10-250-204-239.eu-west-3.compute.internal ansible_host=10.250.204.239bastion ansible_host=35.180.250.230bastion ansible_host=35.180.55.194

[bastion]bastion ansible_host=35.180.250.230bastion ansible_host=35.180.55.194

[kube-master]ip-10-250-206-126.eu-west-3.compute.internal

[kube-node]ip-10-250-201-250.eu-west-3.compute.internal

[etcd]ip-10-250-204-239.eu-west-3.compute.internal

[k8s-cluster:children]kube-nodekube-master

[k8s-cluster:vars]apiserver_loadbalancer_domain_name="kubernetes-elb-mycluster-*****.eu-west-3.elb.amazonaws.com"

Yes, there are 2 bastions with the same name, you can change names if you want both tobe configured

• Optionnal : to restart from scrach (helpfull when you messed a lot with ansible)

terraform destroy --var-file=credentials.tfvarsterraform apply --var-file=credentials.tfvars

Ansible : Cluster creation and configuration

• Change some configuration files

inventory/mycluster/group_vars/all/all.yml

cloud_provider: aws

15

inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml

cluster_name: mycluster

# Make a copy of kubeconfig on the host that runs Ansible in {{ inventory_dir }}/artifactskubeconfig_localhost: true# Download kubectl onto the host that runs Ansible in {{ bin_dir }}kubectl_localhost: true

• Go to project root directry and launch Ansible

cd [..]/kubesprayansible-playbook -vvvv -i ./inventory/mycluster/hosts.ini ./cluster.yml -e ansible_user=core -b --become-user=root--flush-cache --private-key=~/.ssh/my-private-key.pem -e ansible_ssh_private_key_file=~/.ssh/my-private-key.pem 2>&1 |tee "ansible_$(date +"%Y-%m-%d_%I-%M-%p").log"

• When Ubuntu image, change this parameter : -e ansible_user=ubuntu

• You can drop the end (starting with 2>&) if you don’t want to save output to file

• With the minimal configuration we used before, the script should take 10min. Use this time to checkoperations on the master

ssh -F ./ssh-bastion.conf core@ -i ~/.ssh/my-private-key.pemjournalctl -f

• When finished, check the cluster connecting on master

ssh -F ./ssh-bastion.conf core@ -i ~/.ssh/my-private-key.pemsudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes

Kubectl configuration behind a bastion

• Get Master and Bastion IPs from ./inventory/my-cluster/hosts.ini or ./ssh-bastion.conf

• Forward cluster API port to your local machine

ssh -L 6443::6443 ubuntu@ -N -i ~/.ssh/traffic_staging_k8s_fr.pem

• Use admin.conf generated by Ansible playbook as a kube config file, replacing master ip by localhost.Suggested approach : copy it and link it to default kube config path. Then it won’t be replaced by aplaybook run.

cp ./inventory/my-cluster/artifacts/admin.conf ./inventory/my-cluster/artifacts/kubeconfigmv ~/.kube/config ~/.kube/config.oldsudo ln -s ./inventory/my-cluster/artifacts/kubeconfig ~/.kube/config

a non updated kubeconfig could lead to the following error on kubectl usage : Unable toconnect to the server: x509: certificate signed by unknown authority (possiblybecause of "crypto/rsa: verification error" while trying to verify candidateauthority certificate "kubernetes")`

16

Prepare bastion for CI usage

You can use the bastion as a server for the CI to interract with the cluster.

• connect to bastion

ssh-add ~/.ssh/cluster-ssh-key.pemssh BASTION_USER@BASTION_IP

• install docker

sudo apt-get remove docker docker-engine docker.io containerd runcsudo apt-get updatesudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-commoncurl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"sudo apt-get updatesudo apt-get install -y docker-ce docker-ce-cli containerd.io

• install kubectl

sudo apt-get update && sudo apt-get install -y apt-transport-httpscurl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.listsudo apt-get updatesudo apt-get install -y kubectl

• create kubeconfig with content of admin.conf

mkdir .kubevi .kube/config

• add CI runner public ssh key to authorized_keys

vi ~/.ssh/authorized_keys

• download/create the key file to be used to connect to a cluster node

vi ~/.ssh/cluster-key.pemchmod 400 ~/.ssh/cluster-key.pem

• test the ssh connection to the node (and then accept new host)

ssh admin@$MASTER_IP -i ~/.ssh/cluster-key.pem

TipsKubectl autocompletionYou can add autocompletion to kubectl commands.

17

sudo apt-get install bash-completionsudo kubectl completion bash >/etc/bash_completion.d/kubectl

Kubectl aliasesThis article mentions this repo for alias list. Let’s use this fork with more aliases.

• Clone the alias file

curl -o ~/.kubectl_aliases https://raw.githubusercontent.com/jessestuart/kubectl-aliases/master/.kubectl_aliases

• replace kubectl with kctl, for use together with print command

sed -i 's/kubectl/kctl/g' ~/.kubectl_aliases

• Add it to .bashrc, with command print and some custom aliases

cat ~/.bashrc

[ -f ~/.kubectl_aliases ] && source ~/.kubectl_aliases

# k8s change namespacealias kn='kctl config set-context $(kubectl config current-context) --namespace'

# k8s get events sorted by last timealias kgel='kubectl get events --sort-by=.lastTimestamp'

# k8s get events sorted by creation timealias kgec='kubectl get events --sort-by=.metadata.creationTimestamp'

# print kubectl command and then execute itfunction kctl() {  echo "+ kubectl $@";  command kubectl $@;}EOT

Tools

K9S : terminal UI

18

https://containerized.me/600-kubectl-aliases-for-devops-ninjas/https://github.com/ahmetb/kubectl-aliases/blob/master/.kubectl_aliaseshttps://github.com/jessestuart/kubectl-aliases

• Installation

cd /tmp/curl -L https://github.com/derailed/k9s/releases/download/0.5.2/k9s_0.5.2_Linux_x86_64.tar.gz | tar zxsudo mv k9s /usr/bin/

Kubectx : context switching

• Installation

sudo curl -o /usr/bin/kubectx https://raw.githubusercontent.com/ahmetb/kubectx/master/kubectxsudo chmod +x /usr/bin/kubectx

• Autocompletion

bash

sudo apt-get install bash-completionsudo curl -o /etc/bash_completion.d/kubectxhttps://raw.githubusercontent.com/ahmetb/kubectx/master/completion/kubectx.bash

Kubens : namespace switching

• Installation

sudo curl -o /usr/bin/kubens https://raw.githubusercontent.com/ahmetb/kubectx/master/kubenssudo chmod +x /usr/bin/kubens

• Autocompletion

not working under AWS

19

sudo apt-get install bash-completionsudo curl -o /etc/bash_completion.d/kubenshttps://raw.githubusercontent.com/ahmetb/kubectx/master/completion/kubens.bash

Kubeval : manifest validator

kubeval is a tool for validating a Kubernetes YAML or JSON configuration file.

cd /tmpwget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gztar xf kubeval-linux-amd64.tar.gzsudo cp kubeval /usr/bin

Stern : multi-pods tailing

• Installation

sudo curl -L -o /usr/bin/stern https://github.com/wercker/stern/releases/download/1.10.0/stern_linux_amd64sudo chmod +x /usr/bin/stern

• Example

stern --all-namespaces . --since 1m

Kubectl-debug : sidecar debugging pod

kubectl-debug is an out-of-tree solution for troubleshooting running pods, which allows you to run a newcontainer in running pods for debugging purpose. Follow official instructions to install it.

Microk8s

Microk8s is like Minikube but without VM, so lighter.

Multiple nodes on a single host is possible, yet a bit tricky, see this issue.

• Installation

sudo snap install microk8s --classicsudo usermod -a -G microk8s $USERsu - $USER

• To use it like a regular cluster among others, add this to your .bashrc :

export PATH=/snap/bin:$PATHmicrok8s.kubectl config view --raw > $HOME/.kube/microk8s.configexport KUBECONFIG=$HOME/.kube/configexport KUBECONFIG=$KUBECONFIG:$HOME/.kube/microk8s.config

• To install dns, storage and dashboard, and get the token :

20

https://github.com/instrumenta/kubevalhttps://github.com/aylei/kubectl-debughttps://github.com/aylei/kubectl-debug#install-the-kubectl-debug-pluginhttps://microk8s.io/https://kubernetes.io/docs/setup/learning-environment/minikube/https://github.com/ubuntu/microk8s/issues/732

microk8s.enable dns storage ingress dashboardtoken=$(microk8s.kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)microk8s.kubectl -n kube-system describe secret $tokenkubectl proxy

Now you can serve the dashboard with kubectl proxy and then browse the dashboard using the token.

• To list plugins

microk8s.kubectl status

• To delete all

microk8s.kubectl reset

Useful commands• Switch namespace

kubectl config set-context $(kubectl config current-context) --namespace=kube-system

• View an applied yaml

kubectl apply view-last-applied svc/client

• Get pod’s name

kubectl get pods --show-labelskubectl get pod -l app=sonarqube -o jsonpath="{.items[0].metadata.name}"

• Decode a secret

kubectl get secrets/my-secret --template={{.data.password}} | base64 --decode

ClusterIP vs NodePort vs IngressGreat article on the subject.

Restart deployment podsTo have fresh pods, you can kill them. But you can also scale down and up again the deployment.

This induces a service interruption

kubectl scale deployment kubernetes-dashboard --replicas=0

21

http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/loginhttps://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0

deployment.extensions "kubernetes-dashboard" scaled

kubectl scale deployment kubernetes-dashboard --replicas=1

deployment.extensions "kubernetes-dashboard" scaled

Getting logged in to ECR for image pullingaws ecr get-login

docker login -u AWS -p -e none https://.dkr.ecr..amazonaws.com

docker login -u AWS -p https://.dkr.ecr..amazonaws.com

Force updating image with same tag

Manually

The ECR secret has to be updated in the last 12H

To force update an image, if the deployment is configured as imagePullPolicy: Always, deleting the podwill pull the new image.

kubectl get pods -w

NAME READY STATUS RESTARTS AGEapi-dpl-6b5698b848-vddc8 1/1 Running 0 2mclient-dpl-69786bdd8f-5zcnd 1/1 Running 0 13ddb-dpl-6874657d-w6mzb 1/1 Running 0 27mkibana-dpl-55fdf8776f-k45pm 1/1 Running 0 26m

kubectl delete pod api-dpl-6b5698b848-vddc8

pod "api-dpl-6b5698b848-bthbs" deleted

kubectl get pods -w

22

NAME READY STATUS RESTARTS AGEapi-dpl-6b5698b848-bthbs 1/1 Running 0 13dclient-dpl-69786bdd8f-5zcnd 1/1 Running 0 13ddb-dpl-6874657d-w6mzb 1/1 Running 0 54skibana-dpl-55fdf8776f-k45pm 0/1 ContainerCreating 0 3skibana-dpl-8d76c6dd8-cmrvz 0/1 Terminating 0 14dkibana-dpl-8d76c6dd8-cmrvz 0/1 Terminating 0 14dkibana-dpl-8d76c6dd8-cmrvz 0/1 Terminating 0 14dkibana-dpl-55fdf8776f-k45pm 1/1 Running 0 13s

kubectl describe pod api-dpl-6b5698b848-vddc8

Events:Type Reason Age From Message---- ------ ---- ---- -------Normal Scheduled 38s default-scheduler Successfully assigned develop/api-dpl-6b5698b848-vddc8 to ***.compute.internalNormal Pulling 37s kubelet, ***.compute.internal pulling image "694430786501.dkr.ecr.eu-west-3.amazonaws.com/adx/adx-api:develop"Normal Pulled 14s kubelet, ***.compute.internal Successfully pulled image "694430786501.dkr.ecr.eu-west-3.amazonaws.com/adx/adx-api:develop"Normal Created 13s kubelet, ***.compute.internal Created containerNormal Started 13s kubelet, ***.compute.internal Started container

kubectl logs api-dpl-6b5698b848-vddc8

2019-02-08 11:04:50.939 INFO 1 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : InitializingExecutorService 'applicationTaskExecutor'2019-02-08 11:04:51.578 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started onport(s): XXX (http) with context path ''2019-02-08 11:04:51.585 INFO 1 --- [ main] com.biomerieux.adxapi.AdxApiApplication : StartedAdxApiApplication in 9.462 seconds (JVM running for 10.442)

In CI with deployment file

• Add a variable in deployment file

my-app.svc-dpl.yml

spec:  replicas: 1  selector:  matchLabels:  app: sltback  template:  metadata:  labels:  app: sltback  # to force rollout on apply  commit: ${CI_COMMIT_SHA} ①

① Unique value tied to CI, for example commit ID, here using Gitlab CI variables

• During CI, replace the value

23

apk --update add gettext # or apt-get, depending on the OSenvsubst < "my-app-with-variable.svc-dpl.yml" > "my-app.svc-dpl.yml"

• Apply the deployment file

Example applicationHere is a sample application, k8s-workshop, backed by a youtube video.

It’s a scalable webapp with a Redis cluster. To have some replicas at start, edit auto scaling files.

git clone https://github.com/reactiveops/k8s-workshop.gitkubectl create namespace k8s-workshopkubectl label namespace k8s-workshop istio-injection=enabledcd k8s-workshop/completekubectl apply -f 01_redis/kubectl apply -f 02_webapp/kubectl get pods --watch

NAME READY STATUS RESTARTS AGEredis-primary-7566957b9c-6rzb6 1/1 Running 0 4h10mredis-replica-7fd949b9d-db2rf 1/1 Running 0 4h10mredis-replica-7fd949b9d-rz7nb 1/1 Running 0 108mwebapp-5498668448-8hcgq 1/1 Running 0 4h10mwebapp-5498668448-ghv22 1/1 Running 0 107mwebapp-5498668448-jdx9j 1/1 Running 0 107m

HelmHelm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade eventhe most complex Kubernetes application.

Installationcurl -s https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash

Initialization for current clusterkubectl create serviceaccount tiller -n kube-systemkubectl create clusterrolebinding tiller-is-admin --clusterrole=cluster-admin --serviceaccount=kube-system:tillerhelm init --service-account tiller --upgrade

if already initialized by mistake, use this command :

helm reset --force

Incubator repository

24

https://github.com/reactiveops/k8s-workshophttps://www.youtube.com/watch?v=H-FKBoWTVws

helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubatorhelm repo update

Chart installation examplehelm install stable/my-chart --name mc

if no name provided, an auto-generated name is used, like wonderful-rabbit. It is prefixedto all objects, so better shorten it by providing something.

Chart uninstallationhelm delete --purge mc

Multi-tenantBy default, Helm cannot install multiple charts with the same release name, which can become a problemwhen having multiple environment in the same cluster. The solution is having multiple tillers, one in eachnamespace/environment.

NGINX Ingress Controllerhelm install stable/nginx-ingress --name ngc --namespace kube-system

Then get the public DNS with :

kubectl get svc

Works out of the box with AWS/EKS

Cert managerFor TLS, a Certificate manager is needed. Steps are taken from official install documentations.

• Install the manager

kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yamlkubectl create namespace cmkubectl label namespace cm certmanager.k8s.io/disable-validation=truehelm repo add jetstack https://charts.jetstack.iohelm repo updatehelm install --name cm --namespace cm --version v0.7.0 jetstack/cert-manager

• Check installation using official procedure.

25

https://gist.github.com/venezia/69e9bd25d79fec9834fe1c5b589d4206https://cert-manager.readthedocs.io/en/latest/getting-started/install.html#stepshttps://cert-manager.readthedocs.io/en/latest/getting-started/install.html#verifying-the-installation

IstioConnect, secure, control, and observe services in a Kubernetes Cluster.

• Installation inspired from Official documentation.

• Default Istio installation does not install all modules. Istio is installed here in demomode to have all. See profiles coverage.

PrerequisitesHave [helm] installed, locally and cluster side.

Installation• Download and initialise Istio using Helm

curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.1.0 sh -cd istio-1.1.0export PATH=$PWD/bin:$PATHhelm install install/kubernetes/helm/istio-init --name istio-init --namespace istio

• Check that bellow command returns 53 (since cert-manager is not enabled, else it would be 58)

kubectl get crds | grep 'istio.io\|certmanager.k8s.io' | wc -l

• Finish installation with Helm

helm template install/kubernetes/helm/istio --name istio --namespace istio --valuesinstall/kubernetes/helm/istio/values-istio-demo.yaml | kubectl apply -f -

• Check Istio pods startup

kubectl get pods -n istio -w

26

https://istio.io/docs/setup/kubernetes/install/helm/https://istio.io/docs/setup/kubernetes/additional-setup/config-profiles/

NAME READY STATUS RESTARTS AGEgrafana-57586c685b-m67t6 1/1 Running 0 2d19histio-citadel-645ffc4999-7g9rl 1/1 Running 0 2d19histio-cleanup-secrets-1.1.0-ks4fb 0/1 Completed 0 2d19histio-egressgateway-5c7fd57fdb-spwlp 1/1 Running 0 2d19histio-galley-978f9447f-zj8pd 1/1 Running 0 2d19histio-grafana-post-install-1.1.0-wjn4m 0/1 Completed 0 2d19histio-ingressgateway-8ccdc79bc-p67np 1/1 Running 0 2d19histio-init-crd-10-t6pwq 0/1 Completed 0 2d19histio-init-crd-11-j788x 0/1 Completed 0 2d19histio-pilot-679c6b45b8-5fbw7 2/2 Running 0 2d19histio-policy-fccd56fd-8qtlb 2/2 Running 2 2d19histio-security-post-install-1.1.0-p9k29 0/1 Completed 0 2d19histio-sidecar-injector-6dcc9d5c64-szrcw 1/1 Running 0 2d19histio-telemetry-9bcfc78bd-mfwsh 2/2 Running 1 2d19histio-tracing-656f9fc99c-r9n7d 1/1 Running 0 2d19hkiali-69d6978b45-t7tdn 1/1 Running 0 2d19hprometheus-66c9f5694-c548z 1/1 Running 0 2d19h

Activation for a namespaceNamespaces have to be explicitely labelled to be monitored by Istio

kubectl label namespace k8s-workshop istio-injection=enabled

UIsIf your cluster is behind a bastion, forward cluster API port to your local machine

ssh -L 6443:10.250.202.142:6443 ubuntu@35.180.231.227 -N -i ~/.ssh/traffic_staging_k8s_fr.pem

Kiali

Kiali project provides answers to the questions: What microservices are part of my Istio service mesh andhow are they connected.

• forward the port

kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=kiali -ojsonpath='{.items[0].metadata.name}') 20001:20001 &

• access UI at http://localhost:20001

27

http://localhost:20001

Jaeger / OpenTracing

Open source, end-to-end distributed tracing. Monitor and troubleshoot transactions in complex distributedsystems.

• forward the port

kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=jaeger -ojsonpath='{.items[0].metadata.name}') 16686:16686 &

• access UI at http://localhost:16686

Grafana & Prometheus

Grafana in front of Prometheus, for analytics and monitoring the cluster.

• forward the port

kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -ojsonpath='{.items[0].metadata.name}') 3000:3000 &

• access UI at http://localhost:3000/dashboard/db/istio-mesh-dashboard

PowerfulsealPowerfulSeal adds chaos to your Kubernetes clusters, so that you can detect problems in your systems asearly as possible. It kills targeted pods and takes VMs up and down.

28

http://localhost:16686http://localhost:3000/dashboard/db/istio-mesh-dashboardhttps://github.com/bloomberg/powerfulseal

Powerfulseal uses collected cluster IPs, so if the cluster is behind a bastion, installationand usage must be on the bastion.

• Install Powerfulseal

sudo pip install powerfulseal

• Create a policy file. Here we focus on pods failure

~/seal-policy.yml

config:  minSecondsBetweenRuns: 1  maxSecondsBetweenRuns: 30

nodeScenarios: []podScenarios:  - name: "delete random pods in default namespace"

  match:  - namespace:  name: "k8s-workshop"

  filters:  - randomSample:  size: 1

  actions:  - kill:  probability: 0.77  force: true

• Launch powerfulseal

seal autonomous --kubeconfig ~/.kube/config --no-cloud --inventory-kubernetes --ssh-allow-missing-host-keys --remote-user ubuntu --ssh-path-to-private-key ~/.ssh/traffic_staging_k8s_fr.pem --policy-file ~/seal-policy.yml --host 0.0.0.0--port 30100

• watch your pods failing and restarting

kubectl get pods --watch

NAME READY STATUS RESTARTS AGEredis-primary-7566957b9c-6rzb6 1/1 Running 1 4h10mredis-replica-7fd949b9d-db2rf 1/1 Running 0 4h10mredis-replica-7fd949b9d-rz7nb 1/1 Running 0 108mwebapp-5498668448-8hcgq 1/1 Running 3 4h10mwebapp-5498668448-ghv22 1/1 Running 1 107mwebapp-5498668448-jdx9j 1/1 Running 1 107m

TroobleshootingUser "system:anonymous" cannot get resourceWith some script under AWS, you can have this error

29

User \"system:anonymous\" cannot get resource

This command grants admin rights to anonymous users :

kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous

temporary when installing, and only if your cluster is not accessible from internet

Full project exampleTarget architecture

cluster

external

client(front-end)

api(back-end)

elasticsearch(database)

kibana(database UI)

auth0

user

admin

10080:80

10080:80

by port forwarding :5601

:8080

:9200

:9200

EnvironmentsThe AWS cluster will host multiple environments, so we first create and use a develop namespace :

kubectl create namespace developkubectl config current-context

arn:aws:eks:::cluster/

kubectl config set-context arn:aws:eks:::cluster/ --namespace=develop

DeploymentsKubernetes deployments and services are stored in the same file for each module.

Elasticsearch

We start with the elasticsearch database.

Some explanation :

• This is the OSS image, simpler, no need for X-Pack

• Note the system command in initContainers section

30

Deployment file

Example 4. db.svc-dpl.yml

## database (elasticsearch) service and deployement#

apiVersion: v1kind: Servicemetadata:  name: elasticsearch  labels:  app: db  tier: backend  group: adxspec:  ports:  - port: 9200 # External port  targetPort: http # Port exposed by the pod/container from the deployment  selector:  app: db  group: adx---apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: db-dpl  labels:  app: db  tier: backend  group: adxspec:  replicas: 1  template:  metadata:  labels:  app: db  tier: backend  group: adx  spec:  initContainers:  - name: "sysctl"  image: "busybox"  imagePullPolicy: "IfNotPresent"  command: ["sysctl", "-w", "vm.max_map_count=262144"]  securityContext:  privileged: true  containers:  - name: elasticsearch  image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.0  imagePullPolicy: "IfNotPresent"  ports:  - containerPort: 9200  name: http  env:  - name: ES_JAVA_OPTS  value: "-Xms512m -Xmx512m"  - name: discovery.type  value: single-node  resources:  limits:  memory: 1024Mi  requests:  memory: 512Mi

31

Commands

Launch (or update) the deployment :

kubectl apply -f adx.db.svc-dpl.yml

service/api createddeployment.extensions/api-dpl created

kubectl get rs

NAME DESIRED CURRENT READY AGEdb-dpl-5c767f46c7 1 1 1 32m

kubectl get pods

NAME READY STATUS RESTARTS AGEdb-dpl-5c767f46c7-tkqkv 1/1 Running 0 32m

Kibana

Kibana is included, only for elasticsearch administration in test environments.

Some explanation :

• This is the OSS image, simpler, no need for X-Pack

• This will not be accessible from external network, for security reasons

Deployment file

32

Example 5. kibana.svc-dpl.yml

## kibana (elastic admin) service and deployement#

apiVersion: v1kind: Servicemetadata:  name: kibana  labels:  app: kibana  tier: backend  group: adxspec:  # pour protéger, pas de type et pas de port + proxy forward  # type: NodePort # Make the service externally visible via the node  ports:  - port: 5601 # External port  targetPort: http # Port exposed by the pod/container from the deployment  selector:  app: kibana  group: adx---apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: kibana-dpl  labels:  app: kibana  tier: backend  group: adxspec:  replicas: 1  template:  metadata:  labels:  app: kibana  tier: backend  group: adx  spec:  containers:  - name: kibana  image: docker.elastic.co/kibana/kibana-oss:6.6.0  env:  - name: ELASTICSEARCH_URL  value: http://elasticsearch:9200  resources:  limits:  memory: 512Mi  requests:  memory: 256Mi  ports:  - containerPort: 5601  name: http

Commands

Launch (or update) the deployment :

kubectl apply -f adx.kibana.svc-dpl.yml

To access the UI, we use port forwarding in a dedicated shell :

33

kubectl port-forward svc/kibana 5601:5601

The Kibana UI is now available at http://localhost:5601

Api / backend

Some explanation :

• The backend is pulled from AWS/ECR registry

Prerequisites

• Get the full image name in EKR

◦ Got to AWS Admin UI

◦ Choose the zone containing your registry

◦ [Services] → [ECR] → api repository

◦ Get the Image URI

• get the registry password

aws ecr get-logindocker login -u AWS -p -e none https://.dkr.ecr..amazonaws.com

• create/update the secret using it

kubectl delete secret ecr-registry-secret

kubectl create secret docker-registry ecr-registry-secret --docker-username="AWS" --docker-password=""--docker-server=".dkr.ecr..amazonaws.com" --docker-email="my.email@my-provider.com"

It is valid for 12 hours.

Now we can update the file and deploy it.

Deployment file

34

http://localhost:5601

Example 6. api.svc-dpl.yml

## api (back-end) service and deployement#

apiVersion: v1kind: Servicemetadata:  name: api  labels:  app: api  tier: backend  group: adxspec:  ports:  - port: 8080 # External port  targetPort: http # Port exposed by the pod/container from the deployment  selector:  app: api  group: adx---apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: api-dpl  labels:  app: api  tier: backend  group: adxspec:  replicas: 1  template:  metadata:  labels:  app: api  tier: backend  group: adx  spec:  # initContainers:  # - name: "sysctl"  # image: "busybox"  # imagePullPolicy: "IfNotPresent"  # command: ["curl", "-XGET", "http://elasticsearch:9200/_cluster/health?pretty=true"]  containers:  - name: api  image: ***.dkr.ecr.eu-west-3.amazonaws.com/adx/adx-api:develop  ports:  - containerPort: 8080  name: http  env:  - name: ELASTICSEARCH_REST_URIS  value: http://elasticsearch:9200  imagePullPolicy: Always  # imagePullSecrets:  # - name: ecr-registry-secret

Commands

Launch (or update) the deployment :

kubectl apply -f adx.api.svc-dpl.yml

35

Client / frontend

Prerequisites

Same as Api module.

Deployment file

36

Example 7. client.ing-svc-dpl.yml

## client (front-end) ingress service and deployement#apiVersion: extensions/v1beta1kind: Ingressmetadata:  name: client-demospec:  # tls:  # - hosts:  # - cafe.example.com  # secretName: cafe-secret  rules:  - host: demo.afq.link  http:  paths:  - path: /  backend:  serviceName: client  servicePort: 10080---apiVersion: v1kind: Servicemetadata:  name: client  labels:  app: client  tier: frontend  group: adxspec:  type: LoadBalancer # Make the service visible to the world  ports:  - port: 10080 # External port  targetPort: http # Port exposed by the pod/container from the deployment  selector:  app: client  group: adx---apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: client-dpl  labels:  app: client  tier: frontend  group: adxspec:  replicas: 1  template:  metadata:  labels:  app: client  tier: frontend  group: adx  spec:  containers:  - name: client  image: 694430786501.dkr.ecr.eu-west-3.amazonaws.com/adx/adx-client:develop  ports:  - containerPort: 80  name: http  imagePullPolicy: Always  # imagePullSecrets:  # - name: ecr-registry-secret

37

Commands

Launch (or update) the deployment :

kubectl apply -f adx.client.svc-dpl.yml

Access the frontend in a browser

• Get the host/port

get services -o wideNAME TYPE CLUSTER-IP EXTERNAL-IPPORT(S) AGE SELECTORadx-api ClusterIP 10.100.78.159 8080/TCP 2h app=api,group=adxclient LoadBalancer 10.100.145.183 .us-east-2.elb.amazonaws.com 10080:30587/TCP 2happ=client,group=adxelasticsearch ClusterIP 10.100.15.82 9200/TCP 23h app=db,group=adxkibana ClusterIP 10.100.114.147 5601/TCP 23h app=kibana,group=adx

• Go to http://..elb.amazonaws.com:10080

38

http://..elb.amazonaws.com:10080

39

Kubernetes Configuration & Best PracticesTable of ContentsInstallationGoogle Kubernetes Engine (GKE)AWS / EKSAWS / Kubespray / Terraform

TipsKubectl autocompletionKubectl aliasesToolsUseful commandsClusterIP vs NodePort vs IngressRestart deployment podsGetting logged in to ECR for image pullingForce updating image with same tagExample application

HelmInstallationInitialization for current clusterChart installation exampleChart uninstallationMulti-tenant

NGINX Ingress ControllerCert manager

IstioPrerequisitesInstallationActivation for a namespaceUIs

PowerfulsealTroobleshootingUser "system:anonymous" cannot get resource

Full project exampleTarget architectureEnvironmentsDeployments