Building Docker Containers @ Scale

  • Published on
    23-Jan-2018

  • View
    496

  • Download
    4

Embed Size (px)

Transcript

<ol><li> 1. Building containers Go-Toronto Nov/2016 @lxfontes </li><li> 2. #bfcm </li><li> 3. Continuous Delivery Pipeline 1-1 BuildGit Push Test Deploy </li><li> 4. Your company keeps growing </li><li> 5. Growing </li><li> 6. Growing MOAR... </li><li> 7. Clone repository docker build - Install RubyGems - Install NPM Packages - Compile static assets - Seal Docker image docker push - Upload to Docker Registry Building a Rails Container </li><li> 8. Inertia You get used to it Your build time only goes up Number of workers only goes up Our build is complex We already use Docker, the Cloud. This is as fast as it gets! It is automated, we dont care it takes 30 minutes The XYZ team should fix it </li><li> 9. it was a reality check 20 minutes per build Flakiness Resource Starvation Really expensive * where do you put secrets?????? </li><li> 10. Why is it slow? push webhook worker bootstrap docker build cache not reused between builds cold asset precompilation 2nd build </li><li> 11. peed </li><li> 12. Where are we: Daily stats +200 Devs +700 Builds +42k Tests per build ~3 Images per build </li><li> 13. Commitment Container Build Git Push Automated Tests Deploy 5s 90s 200s 300s From Git Push to Deployed in ~10 minutes Every developer can deploy to production Goals: Testing Rails at Scale by Emil How We Deploy Shopify by Kat This right here </li><li> 14. push 30 sec POLL Pull / Build Coordinator + API zookeeper workers registry branch affinity </li><li> 15. ohai bootstrap </li><li> 16. Clone repository docker build - Install RubyGems - Install NPM Packages - Compile static assets - Seal Docker image docker push - Upload to Docker Registry Building a Rails Container </li><li> 17. Dockerfile </li><li> 18. mo layers, mo problems * apparently docker 1.13 can squash these </li><li> 19. Look ma, no Dockerfiles prepare Install OS packages precompile Load (ejson) secrets Populate artifact cache compile Combine app + artifact cache Seal image Daily seed 20min Builds 2min Inherit Cache Inherit Cache </li><li> 20. Docker Layers Base Ubuntu-ish Prepare OS packages Intermediate (Daily seed) App + Libraries Final Delta between Intermediate Current </li><li> 21. Locutus trade offs FAST Local caches Secure Stable But needs love We need to maintain: Infrastructure Orchestration Web UI Scripts Copy &amp; Pasta Dockerfiles / Buildpacks :( Troubleshooting :( </li><li> 22. Pipa * Kite in Portuguese (and sounds cool) </li><li> 23. Provider agnostic Disposable Secure Sandboxed Docker Daemon (dind) Pre-made recipes Buildpack Dockerfiles Locutus Assembly Pipa: Requirements Build whatever you want Cache whatever you want Repeatable Run locally if needed As fast as Locutus </li><li> 24. Is there an app fo dat? Orchestration / UI Docker first On-premise Worker fully sandboxed Concurrency Primitives Only 3 builds for app X at same time Parallelism Send job to 30 workers </li><li> 25. Buildkite On-premise Orchestration *only* Github Hooks UI / Reporting API Agent is a single binary (Go) Also distributed as docker container Concurrency Primitives Parallelism Per Build artifacts Per Build shared key-value store Switch workers!! No batteries included on purpose </li><li> 26. Buildkite Pipeline Sample - buildkite/sample-pipelines Waits for previous step to complete Requires Interaction UI/API/slack Branch filtering </li><li> 27. Can it scale? </li><li> 28. BuildkiteGitHub BK Pipa Docker Registry BK Tests Webhook Build Container Switch Worker Start Tests Fetch Image Tests Done Ship It! Tests Upload Image 100 cores 8820 cores Pipa: Goal </li><li> 29. Pipa: Architecture goals Everything exposed via environment variables PIPA_SSH_KEY / PIPA_APP_COMMIT_ID Chainable (simplify IF/ELSE ops) pipa build --unless v1 -- docker build -t myimage:v1 . Configurable via environment, config file, argument switches PIPA_APP_NAME=myapp pipa build ... pipa build --app-name myapp Isolated Docker Daemon (docker-in-docker) Reset after each build Kubernetes: external SSD / Local: give it a directory </li><li> 30. Environment / Process Tree pipa wrapper AWS / GCS / SSH Keys Kubernetes secrets env vars with paths APP Name Buildkite / Jenkins / etc GIT (local) APP SHA Buildkite / Jenkins / etc GIT (local) Your code Signals </li><li> 31. Prototype: Im going to write this all in bash </li><li> 32. Single Binary Shell files as Assets using go-bindata docker / aws-cli / rsync </li><li> 33. LOL </li><li> 34. BUT, normalized environment is solid! </li><li> 35. Provider agnostic Disposable Secure Sandboxed Docker Daemon (dind) Pre-made recipes Buildpack Dockerfiles Locutus Assembly Pipa: Requirements Build whatever you want Cache whatever you want Repeatable Run locally if needed As fast as Locutus </li><li> 36. Toolbox: spf13/cobra CLI handling docker, kubernetes, rkt, etcd ABUSE: func init() </li><li> 37. Toolbox: kelseyhightower/envconfig *spf13/cobra deficiency Fills struct with env vars </li><li> 38. Why both? spf13/cobra kelseyhightower/envconfig </li><li> 39. Result Google Cloud Storage Amazon S3 JSON Asymmetric Encryption Shopify/ejson Registry Interaction push / pull / exists mmap diff 2 dirs burke/treediff Environment normalizer (entrypoint) </li><li> 40. Using it Everything is namespaced! pipa cache pull /tmp/artifacts pipa build --if base -- docker build -t $PIPA_IMAGE_FULL_NAME . pipa image exists --local --tag base pipa image exists --remote --tag base Downloads s3://artifacts/app_group/app_env/app_name/cache.tar to /tmp/artifacts Only runs command if registry/app_group/app_env/app_name:base exists Checks if registry/app_group/app_env/app_name:base exists (locally/remotely) </li><li> 41. Pipeline Inference How should pipa build your app? Customize it in your repo; or Select a template; or Let pipa figure it out! </li><li> 42. Built-in pipelines </li><li> 43. Pipeline Output </li><li> 44. Pipeline Parallelization 3 parallel builds </li><li> 45. Pipeline Selection Custom script in repo Custom pipeline in repo Default pipeline in repo User requested specific pipeline Check for common file locations repo/Dockerfile? -&gt; docker repo/borg? -&gt; borg Default to Heroku Buildpack PIPA_PIPELINE_CMD PIPA_PIPELINE_FILE PIPA_PIPELINE_TEMPLATE repo/.buildkite/pipeline.yml </li><li> 46. High Level push webhook Agent poll SSL docker images cache artifacts Queue API User Interface </li><li> 47. ssh-key gcloud-key buildkite-token docker-auth Sec: Kubernetes Layout EJSON priv/pub keypairs /builder-secrets /app-secrets /var/lib/docker (dind) /builds (git clone) /cache (artifacts) 6x nodes 24 pods /mnt/disks/ssd0/builder1/docker </li><li> 48. Provider agnostic Disposable Secure Sandboxed Docker Daemon (dind) Pre-made recipes Buildpack Dockerfiles Locutus Assembly Pipa: Requirements Build whatever you want Cache whatever you want Repeatable Run locally if needed As fast as Locutus? </li><li> 49. Gems </li><li> 50. Tar + Gzip </li><li> 51. Go + Docker = fsouza/go-dockerclient Respect environment variables Reads $HOME/.docker/config.json or $HOME/.dockercfg </li><li> 52. Rocker - grammarly/rocker Dockerfile on steroids </li><li> 53. Herokuish - gliderlabs/herokuish If you like Heroku, you will want this Packaged as docker container or single binary </li><li> 54. alpinelinux.org based on musl libc and busybox Lots of *up-to-date* packages (yes debian, looking at you) </li><li> 55. Demo Time </li></ol>