Embed Size (px)
Citation preview
Spreadshirt
Kubernetes at SpreadshirtFirst steps to production
Jens Hadlich, Chief Architect
Spreadshirt
SpreadshirtWE PRINT IDEAS
Spreadshirt
Not all that long ago …We want to build a
service with Ruby …Why?
… and the other team was allowed to build a service with Node.js …
…
Sure, I already forgot. Hmm … OK. Let‘s try.
Ruby is the right tool for the job ...
We could build it in Java but that would
be too much …
Other companies use it for years already …
It‘s a very simple service …
…
…
…
3
Spreadshirt
Some time later …To run it, you just need some to install some
gems.
Didn‘t work.Sure not. You need Ruby in Version X.
X? You can only have Y.
What‘s a gem?
…
How about using Docker?
…
…
Ecosystem man.
…
4
Spreadshirt 5
Docker
Spreadshirt
Docker is nice, but … *
Number ofRunningContainers
1 5 10 50 100
* without container orchestration or additional tooling
6
Spreadshirt
Docker is nice, but … *
Number ofRunningContainers
1 5 10 50 100
* without container orchestration or additional tooling
7
Spreadshirt
Docker is nice, but … *
Number ofRunningContainers
1 5 10 50 100
* without container orchestration or additional tooling
8
Spreadshirt
Docker is nice, but … *
Number ofRunningContainers
1 5 10 50 100
* without container orchestration or additional tooling
9
Spreadshirt
Docker is nice, but … *
Number ofRunningContainers
1 5 10 50 100
* without container orchestration or additional tooling
10
Spreadshirt 11
Docker is nice, but … *
Number ofRunningContainers
1 5 10 50 100
* without container orchestration or additional tooling
Spreadshirt 12
Kubernetes
Spreadshirt
Kubernetes for Container Orchestration
The promise ...• Easily bring up new components and services
• Easily manage hundreds or thousands of containers
• Easily add capacity
• Better resource utilization
• (Auto-)scale
13
Spreadshirt
Preparation
Think about …• Monitoring
• Logging
• Metrics
• Configuration management
• Build & Deployment pipelines
• Your application stack
14
Spreadshirt
Tool Ecosystem at Spreadshirt
15
Spreadshirt
Tool Ecosystem at Spreadshirt (cont’d)
16
VxLAN (and even more …)
Spreadshirt 17
Spreadshirt
Build
git commitgit push
.gitlab-ci.yml
* io.fabric8/docker-maven-plugin
*
18
Spreadshirt
Deploy
Manifest files
kubectl
Service IP
19
Spreadshirt
Kuh (Kubernetes hiera)
• Homegrown preprocessor for k8s manifest files
• Why? Avoid yet another big project (change configuration management)
• Bridge between Puppet/hiera (our “old” world) and Kubernetes (the “new” world) – peaceful coexistence
20
Spreadshirt
Puppet/hiera
21
Spreadshirt
Kuh input structure
├── deployment
│ ├── data
│ │ ├── common.yaml
│ │ ├── ops.yaml
│ │ ├── qa.yaml
│ │ ├── <team>.yaml
│ │ └── ...
│ ├── src
│ │ ├── config
│ │ │ └── application-config.yaml
│ │ └── infra
│ │ └── k8s.yaml
hiera “data” files like with Puppet
add to your git project
22
k8s manifest templates
Spreadshirt
Kubernetes manifest templates with Kuh
apiVersion: v1
kind: Service
metadata:
name: my-service
labels:
app: my-service
spec:
type: NodePort
clusterIP: 192.168.X.Y
ports:
- port: <%= scope().call_function('hiera',['my_service::http_port']) %>
name: http
...
Retrieve the value from hiera*
23
* yeah, we know this looks ugly …
Spreadshirt
Kuh output
├── deployment
│ ├── data
│ ├── src
│ └── target
│ ├── config
│ │ └── application-config.yaml
│ └── k8s
│ ├── configmap
│ │ └── application-config.yaml
│ └── infra
│ └── k8s.yaml
24
Final k8s manifest files
Spreadshirt
Kubernetes Manifests
• Deployment
• ConfigMap
• Secret
• Service
• (more to come)
25
Spreadshirt
Docker registry & images
• Nexus 3 Java / Maven repository Own Docker registry
- Nice WebUI- Traffic*
• Docker images latest tag, override Other tags:
- ${git.commit.time}.${git.commit.id}e.g. 2017-03-
29.a5344e3501d591e4c7908d97a044d3d0f38ddb43
to be able to clean up old images and better readability
26
* pushing and pulling Docker images can generate significant network traffic
Spreadshirt
Stateful Applications
• Not at the moment!
• Databases, Elasticsearch, … are “external”
• Avoid local storage
• Use Web-APIs (e.g. S3)
27
Spreadshirt
Open issues
We are still in learning mode!
• Improve high availability for Kubernetes 2 separate Kubernetes clusters
- Single master: if it fails – restart- Clustered etcd
Puppetized => spin up a new cluster within minutes
• How to manage storage / stateful services? E.g. DB, Elasticsearch
28
Spreadshirt
Open issues (cont’d)
• Security patches / lifecycle of Docker images / containers within k8s What’s running inside my container? Manage re-build / re-deploy after a certain time? Trust 3rd party base images?
• Reduce complexity
• Revisit custom HAProxy vs. built-in Ingress & Co.
29
Spreadshirt
Nice tools
• sysdig / csysdig
30
Spreadshirt
Links
• https://sysdig.com/blog/digging-into-kubernetes-with-sysdig/
