Kubernetes at Spreadshirt - First steps to production

Spreadshirt

Kubernetes at SpreadshirtFirst steps to production

Jens Hadlich, Chief Architect

Spreadshirt

SpreadshirtWE PRINT IDEAS

Spreadshirt

Not all that long ago …We want to build a

service with Ruby …Why?

… and the other team was allowed to build a service with Node.js …

…

Sure, I already forgot. Hmm … OK. Let‘s try.

Ruby is the right tool for the job ...

We could build it in Java but that would

be too much …

Other companies use it for years already …

It‘s a very simple service …

…

…

…

3

Spreadshirt

Some time later …To run it, you just need some to install some

gems.

Didn‘t work.Sure not. You need Ruby in Version X.

X? You can only have Y.

What‘s a gem?

…

How about using Docker?

…

…

Ecosystem man.

…

4

Spreadshirt 5

Docker

Spreadshirt

Docker is nice, but … *

Number ofRunningContainers

1 5 10 50 100

* without container orchestration or additional tooling

6

Spreadshirt

Docker is nice, but … *

Number ofRunningContainers

1 5 10 50 100

* without container orchestration or additional tooling

7

Spreadshirt

Docker is nice, but … *

Number ofRunningContainers

1 5 10 50 100

* without container orchestration or additional tooling

8

Spreadshirt

Docker is nice, but … *

Number ofRunningContainers

1 5 10 50 100

* without container orchestration or additional tooling

9

Spreadshirt

Docker is nice, but … *

Number ofRunningContainers

1 5 10 50 100

* without container orchestration or additional tooling

10

Spreadshirt 11

Docker is nice, but … *

Number ofRunningContainers

1 5 10 50 100

* without container orchestration or additional tooling

Spreadshirt 12

Kubernetes

Spreadshirt

Kubernetes for Container Orchestration

The promise ...• Easily bring up new components and services

• Easily manage hundreds or thousands of containers

• Easily add capacity

• Better resource utilization

• (Auto-)scale

13

Spreadshirt

Preparation

Think about …• Monitoring

• Logging

• Metrics

• Configuration management

• Build & Deployment pipelines

• Your application stack

14

Spreadshirt

Tool Ecosystem at Spreadshirt

15

Spreadshirt

Tool Ecosystem at Spreadshirt (cont’d)

16

VxLAN (and even more …)

Spreadshirt 17

Spreadshirt

Build

git commitgit push

.gitlab-ci.yml

* io.fabric8/docker-maven-plugin

*

18

Spreadshirt

Deploy

Manifest files

kubectl

Service IP

19

Spreadshirt

Kuh (Kubernetes hiera)

• Homegrown preprocessor for k8s manifest files

• Why? Avoid yet another big project (change configuration management)

• Bridge between Puppet/hiera (our “old” world) and Kubernetes (the “new” world) – peaceful coexistence

20

Spreadshirt

Puppet/hiera

21

Spreadshirt

Kuh input structure

├── deployment

│ ├── data

│ │ ├── common.yaml

│ │ ├── ops.yaml

│ │ ├── qa.yaml

│ │ ├── <team>.yaml

│ │ └── ...

│ ├── src

│ │ ├── config

│ │ │ └── application-config.yaml

│ │ └── infra

│ │ └── k8s.yaml

hiera “data” files like with Puppet

add to your git project

22

k8s manifest templates

Spreadshirt

Kubernetes manifest templates with Kuh

apiVersion: v1

kind: Service

metadata:

name: my-service

labels:

app: my-service

spec:

type: NodePort

clusterIP: 192.168.X.Y

ports:

- port: <%= scope().call_function('hiera',['my_service::http_port']) %>

name: http

...

Retrieve the value from hiera*

23

* yeah, we know this looks ugly …

Spreadshirt

Kuh output

├── deployment

│ ├── data

│ ├── src

│ └── target

│ ├── config

│ │ └── application-config.yaml

│ └── k8s

│ ├── configmap

│ │ └── application-config.yaml

│ └── infra

│ └── k8s.yaml

24

Final k8s manifest files

Spreadshirt

Kubernetes Manifests

• Deployment

• ConfigMap

• Secret

• Service

• (more to come)

25

Spreadshirt

Docker registry & images

• Nexus 3 Java / Maven repository Own Docker registry

- Nice WebUI- Traffic*

• Docker images latest tag, override Other tags:

- ${git.commit.time}.${git.commit.id}e.g. 2017-03-

29.a5344e3501d591e4c7908d97a044d3d0f38ddb43

to be able to clean up old images and better readability

26

* pushing and pulling Docker images can generate significant network traffic

Spreadshirt

Stateful Applications

• Not at the moment!

• Databases, Elasticsearch, … are “external”

• Avoid local storage

• Use Web-APIs (e.g. S3)

27

Spreadshirt

Open issues

We are still in learning mode!

• Improve high availability for Kubernetes 2 separate Kubernetes clusters

- Single master: if it fails – restart- Clustered etcd

Puppetized => spin up a new cluster within minutes

• How to manage storage / stateful services? E.g. DB, Elasticsearch

28

Spreadshirt

Open issues (cont’d)

• Security patches / lifecycle of Docker images / containers within k8s What’s running inside my container? Manage re-build / re-deploy after a certain time? Trust 3rd party base images?

• Reduce complexity

• Revisit custom HAProxy vs. built-in Ingress & Co.

29

Spreadshirt

Nice tools

• sysdig / csysdig

30

Spreadshirt 31

Thank Youjns@spreadshirt.com

Spreadshirt

Links

• https://sysdig.com/blog/digging-into-kubernetes-with-sysdig/