Adventures with Laravel and Kubernetesat superbalist.com

William Stewart zatech zoidbergwill basically everywhere

1 / 28

IntroWho am I?What do I do?

2 / 28

OverviewDocker

What is Docker?Why do we use it?

KubernetesWhat is Kubernetes?Why do we use it?

Some extra toolsA quick demoWar stories

LoggingSignal propagationLong running processesSchema changes

3 / 28

Docker

4 / 28

What is Docker?

5 / 28

How does it help?Language agnostic build artifact

FROM composer:alpineCOPY composer.json composer.lock /appRUN composer install --ansi --no-interaction --no-scripts --no-dev --no-autoloaderCOPY . /appRUN composer install

or

FROM node:alpineCOPY package.json /appRUN npm installCOPY . /app

6 / 28

How does it help?No more "works on my machine", because they're the same images locally and on production

Smaller shippable artifacts than full machine images, and easier to independently scale components

A lot of useful services already in the Docker registry:

MySQLRedisMemcachedElastic SearchRabbitMQ

7 / 28

Running Docker in production

8 / 28

How does it help?- Container hosting- Con�g changes- Supervision- Monitoring- Rolling deployments- Networking / Service discovery- and more...

9 / 28

Automate the boring stu�f“Automation is a force multiplier, not a panacea”

Value of automationConsistencyExtensibilityMTTR (Mean Time To Repair)Faster non-repair actionsTime savings

Dan Luu's Notes on Google's Site Reliability Engineering book

10 / 28

What is Kubernetes?

an open source containercluster manager

11 / 28

How does it help?Faster/smarter builds than a .tar.gzReliable rolling updates (safer than a symlink swap)Similar build/deploy scripts for all languages and servicesAutoscaling (based on CPU usage)Fine-grained healthcheckingAccurate resource monitoring and better usage

12 / 28

Helps us to easily move towards SOA,Easily start new projects in any language, that can rely on the same service discovery (using DNS)

Discovery via DNS (curl http://elasticsearch/) or automatically set Kubernetes environmentvariables (env('ELASTICSEARCH_SERVICE_HOST')), that balances load over a collection of Elastic Searchclients, all automatically updated by Kubernetes.

A huge collection of base Docker�les on hub.docker.com and quay.io

13 / 28

Avoid "works on my machine" issuesAccurately replicate most of our production environment locally (other than a mocked Google Loadbalancer)

Using:

minikube, which behaves like a single-node Kubernetes clusterdocker, moving towards rocker, to better utilise global Composer/NPM cacheNFS, to allow faster feedback when developingsome bash glue

14 / 28

Easily implement monitoring, logging, and LetsEncryptSimply implement metrics monitoring and LetsEncrypt using Kubernetes annotations:

annotations: prometheus.io/port: "9150" prometheus.io/scrape: "true"

apiVersion: extensions/v1beta1kind: Ingressmetadata: annotations: kubernetes.io/ingress.class: gce kubernetes.io/tls-acme: "true" name: general-ingressspec: rules: - host: tasks.zoidbergwill.com http: paths: - backend: serviceName: tasks-app servicePort: 80 path: / tls: - hosts: - tasks.zoidbergwill.com secretName: tasks-tls

15 / 28

Demo toolsMinikube

Rocker

Kube-lego

16 / 28

DEMO

DEMO

17 / 28

War stories

18 / 28

LoggingLogging to a �le worked great on compute engine with �uentd

Gcloud Stackdriver Logging is pretty amazing, but with container engine it wants stdout

± ag log tasks/config/app.php112: | Logging Configuration115: | Here you may configure the log settings for your application. Out of116: | the box, Laravel uses the Monolog PHP logging library. This gives117: | you a variety of powerful log handlers / formatters to utilize.119: | Available Settings: "single", "daily", "syslog", "errorlog"123: 'log' => env('APP_LOG', 'single'),125: 'log_level' => env('APP_LOG_LEVEL', 'debug'),212: 'Log' => Illuminate\Support\Facades\Log::class,

19 / 28

Logging Monolog/Laravel optionssingle/daily �le:

doesn't go to stdout and get consumed by fluentd-cloud-logging

syslog:

another dependency, and we'd need to install syslog and tail it, or talk to an external syslog daemon

errorlog:

php-fpm logs have a pre�x you can't get rid ofBubbling them to the nginx container also has a gross pre�x and means you can't separate the logs for each container

20 / 28

Upstream php images also do some gross logging stu�fFROM php:7-fpm

[global]error_log = /proc/self/fd/2

[www]access.log = /proc/self/fd/2

Access logs are junk for Laravel:

:: - 07/Feb/2017:13:57:57 +0000 "GET /index.php" 200 (x 1337)

error_log can be pretty crap too:

[07-Feb-2015 11:46:44] WARNING: [pool www] child 32465 said into stderr: "NOTICE: PHP message: Array"[07-Feb-2015 11:46:44] WARNING: [pool www] child 32465 said into stderr: "("[07-Feb-2015 11:46:44] WARNING: [pool www] child 32465 said into stderr: " [6] => 363"[07-Feb-2015 11:46:44] WARNING: [pool www] child 32465 said into stderr: ")"[07-Feb-2015 11:46:44] WARNING: [pool www] child 32465 said into stderr: ""

21 / 28

Going full circleLet's try single again:

> cat entrypoint.sh# Tail a log file, and background that processtail -f /var/www/html/storage/logs/laravel.log &# Now run the more important piecephp-fpm7

22 / 28

Signal propagationLinux relies on PID 1 doing important stuff, that an init system does

Most Docker entrypoints use exec to become the main process, but this kills anything running in the script.

Shells (like bash) don't propagate signals, like SIGTERM or SIGQUIT, which messes with graceful shutting down.

When a pod dies:

PID 1 in the container gets a SIGTERM and given a grace period (default: 30s) to shutdown by defaultOnce the grace period has expired then a SIGKILL signal is sent

dumb-init and lifecycle hooks to the rescue!

dumb-init wraps your ENTRYPOINT shell script and makes sure signals propagate to child processes

lifecycle: preStop: exec: # SIGTERM triggers a quick exit; gracefully terminate instead command: ["/usr/sbin/nginx","-s","quit"] # or php artisan queue:restart

23 / 28

CronIf you run plain old cron it runs processes in a separate env, so we dump our Kubernetes environment for phpdotenv toread:

# Cron is a bastard that has sub processes with different env vars.env > .envsed -i -e "s/=\(.*\)$/=\"\1\"/g" .env

24 / 28

Long running processesThe max "grace period" for a pod is 5 minutes.

Easy steps in the right direction:

Break tasks up as small as possiblePut most tasks on queuesUse Kubernetes "Jobs" when forced to

25 / 28

Database migrationsThree step DB Migrations

Migrating before starting a rolling update

26 / 28

Superbalist.com and Takealot.com are hiring!!!

Sourcehttps://github.com/zoidbergwill/kubernetes-php-examples

Slides: zoidergwill.github.io/presentations/2017/kubernetes-php/

Links on the next slide!

27 / 28

LinksO'Reilly's Site Reliability Engineering: How Google Runs Production Systems Amazon

Dan Luu's notes are good too

Borg, Omega, and Kubernetes Lessons learned from three container-management systems over a decade. Essay

kubernetes-dashboard

Cloud Native Computing Foundation

Kubernetes-Anywhere: An of�cial Kubernetes repo with some documentation on running Kubernetes with Dockerfor Mac beta

minikube: The of�cial Go binary for running a simpler local cluster.

awesome-kubernetes list on GitHub, cuz it has some neat things.

Docker and the PID 1 reaping problem

Containers really are the future

28 / 28