Deploying WSO2 Middleware on Kubernetes

Imesh GunaratneSenior Technical Lead, WSO2

Agenda● PaaS Solutions for WSO2 Middleware● Deploying WSO2 Middleware on Kubernetes

○ Kubernetes Architecture○ Kubernetes Features Used○ WSO2 Docker Images○ WSO2 Carbon Reference Architectures for Kubernetes○ Deployment Workflow○ WSO2 Kubernetes Artifacts○ Monitoring

PaaS Solutions forMiddleware

PaaS Solutions for WSO2 Middleware

Core PaaS Features Needed for a Production Deployment

● Auto healing● Autoscaling● Automatic cluster discovery● Dynamic load balancing● VM/Container support● Multi-tenancy● Configuration orchestration● Artifact distribution● Multi-region/cloud deployments

● Centralized logging● Monitoring● Metering

Deploying WSO2 Middleware on Kubernetes

An Introduction to Kubernetes

● An open source container cluster management system● Started by Google with decade and half experience on

running containers at scale (Borg)● Written in Go● Can be deployed on any Infrastructure/IaaS● Over 600 contributors

http://kubernetes.io/v1.0/docs/whatisk8s.html

Kubernetes High Level Architecture

Node1 Node2 Node n

Physical Network

Master

Overlay Network (Flannel/OpenVSwitch/Weave)A

PI S

erve

rSc

hedu

ler

Con

trol

ler M

anag

eret

cd

Kubernetes Component Architecture

Kubernetes Features used by WSO2 Middleware

● Pods● Replication Controllers (RC)● Horizontal Pod Autoscalers (HPA)● Services● Ingress/Service Load Balancer● Overlay Network● Service Accounts & Secrets● Namespaces

Kubernetes Pods

● A group of containers which can share resources and context

● Shared namespaces:○ PID namespace (processes visible globally)○ network namespace (same IP and port

space)○ IPC namespace (SystemV IPC or POSIX)○ UTS namespace (share a hostname)

http://kubernetes.io/v1.0/docs/user-guide/pods.html

Pod

C1 C2 C3

Kubernetes Replication Controllers

● Provides features for replicating pods:○ Auto-healing○ Manual scaling○ Rolling updates○ Multiple release tracks

http://kubernetes.io/v1.0/docs/user-guide/replication-controller.html

Pod TemplateReplicas = n

Pod 1 Pod 2 Pod n

Replication Controller

Kubernetes Horizontal Pod Autoscaler

● Provides features for horizontally autoscaling pods○ CPU consumption based○ Currently in Alpha

http://kubernetes.io/v1.1/docs/user-guide/horizontal-pod-autoscaler.html

Pod TemplateReplicas = n

Pod 1 Pod 2 Pod n

Replication Controller

Horizontal Pod Autoscaler

● Service types:○ ClusterIP (default)○ NodePort○ LoadBalancer

● Service discovery:○ DNS○ Environment variables

● Client IP based Session Affinityhttp://kubernetes.io/v1.0/docs/user-guide/services.html

Kubernetes ServicesNode IP: 172.17.8.102

Port: 9443

Domain Name: service1IP: 10.2.10.20

Port: 9443NodePort: 32001

Protocol: TCP

Pod 1 Pod 2 Pod n

Service

L1

L1 L1 L1

Node

● Unique IP addresses● Provides pod to pod networking● Options available:

○ Flannel (default)○ OpenVSwitch○ Weave○ Calico

http://kubernetes.io/v1.1/docs/admin/networking.html

Kubernetes Overlay Network

WSO2 Docker Images

https://github.com/wso2/dockerfiles

WSO2 Carbon Reference Architecture 1

Pod 1 Pod 2

Manager Cluster Services

Pod 3 Pod 4 Pod n

Worker Cluster Services

Manager Replication Controller

Worker Replication Controller

Client

Gov Reg

Conf Reg

User Store

WSO2 Carbon Reference Architecture 2

Pod 1

Gateway Mgr Services

Pod 2 Pod 3

Gateway Worker Services

Client

Pod 4 Pod 5

Key Manager Services

Pod 6 Pod 7

Store Services

Gateway Mgr RC

Gateway Worker RC

Key Manager RC

Store RCGov Reg

Conf Reg

User Store

API-M DB

A volume mount for synchronizing API artifacts

Pod 8 Pod 9

Publisher Services

Publisher RC

Carbon Cluster Discovery on Kubernetes

Pod 1 Pod 2 Pod n

Service

Server 1 Server 2 Server n

Pod 1

Service

Server 1

Pod 1 Pod 2

Service

Server 1 Server 2

Hazelcast member initialization

Hazelcast member initialization

Hazelcast member initialization

1 2 n

Deployment Workflow

3

1 2Build Docker

ImageImport Docker Image

to Docker Registry

Application Artifacts

Kubernetes Membership

Scheme

Product Distribution

Docker Image (DI)

Puppet Module + Hiera FilesJDK Deploy Kubernetes

ArtifactsKubernetes

Artifacts

WSO2 Kubernetes Artifacts

● WSO2 Kubernetes Artifacts Git Repository○ Kubernetes Membership Scheme○ Kubernetes Services○ Kubernetes Replication Controllers

https://github.com/wso2/kubernetes-artifacts

Monitoring Kubernetes Cluster Health

Monitoring Kubernetes Cluster/Node/Pod Health

cAdvisorcAdvisor

Node 1

cAdvisor

Heapster

Node 2 Node n

Grafana

InfluxDB

Thank you!