Kubernetes in Docker

Alex MavrogiannisDaniel HiltgenDocker EE Engineering

Agenda1. Recap EE demo from Keynote2. General CE/EE Architectures3. EE: Topics on mixed workloads4. EE: AuthN/AuthZ5. Q&A

Demo Recap

General CE/EE Architecture

Linuxkit VM

Kubernetes CLI

(Swarm-Mode) Kubernetes

etcd

Docker CLI

kubeadm

Docker CE to include Kubernetes (Windows and Mac)

Stacks CRD

Single Docker Engine

vpnkitHost fs mounts hyperkit / hyperv

Docker EE to include Kubernetes

DockerEnterpriseEdition

ProductionReadyWindowsandIBMP/ZSupport Pods,batchjobs,blue-greendeployments,horizontalpodauto-scaling

DockerSwarm Swarm-Mode Kubernetes

Private Image Registry

Secure Access and User Management App and Cluster Management

Image Security Scanning Content Trust and Verification

Policy Management

Orchestrator: Docker Swarm

● github.com/docker/swarm

● Cluster-wide imperative API based on the Single-node API of the Docker Engine

● High Availability and peer discovery managed through a pluggable discovery backend:

etcd, consul

● Leader caches cluster state: containers, volumes, networks etc.

● Scheduling decisions based on the reservations and limits of all cached Docker Containers.

Orchestrator: Docker Engine with Swarm-Mode Enabled

● github.com/docker/swarmkit

● Declarative State through the “Service” construct

● Built-in Routing Mesh & Overlay networking

● Scheduling decisions based on all the reservations of all swarm services across all nodes.

● Built-in in-memory Raft Store for all state (persisted to disk)

● Built in CA, per-node cryptographic node identity, mTLS between all endpoints

Orchestrator: Kubernetes

● github.com/kubernetes/kubernetes

● Scheduling: Pods

● Declarative State through “Controllers”: Deployment, ReplicaSet, DaemonSet …

● Flat Networking model delegated to plugins

● Scheduling decisions based on usage, reservations and limits of all kubernetes workloads.

○ Usage monitored through “cadvisor”, a cgroup monitoring tool

GUI

Universal Control Plane

Trusted Registry Kubernetes CLI

Docker Engine

Swarm-Mode Docker Swarm Kubernetes

etcd

CA OIDC Provider

Docker CLI

Agent Reconciler

Docker EE to include Kubernetes

Docker EE Architectural Highlights

● Unmodified Kubernetes components run as Docker containers

● Swarm Managers are Kubernetes Masters

● Swarmkit node inventory is source of truth

● Cryptographic Node Identity and mTLS used throughout

● UCP Agent/Reconciler manages component lifecycle

○ Manager / Worker states

○ Certificate validity

Plugin Interfaces

● General: Native API extensibility supported

○ API server and kubelet flags not modifiable

● Networking:

○ Support for CNI plugin during install

○ Ingress

● Storage: Docker Volume Plugins supported via built-in flexvolume driver, CSI in future

● Metrics: Heapster Storage Backends or Prometheus

Topics on Mixed Workloads

Resource Contention

● Allocatable Resources: The set of CPU and Memory resources available for scheduling by

an orchestrator

● Multiple orchestrators = Different definitions of allocatable resources○ Docker Swarm: Respectful of CPU/Memory limits, but container cache may be stale○ Docker Engine with Swarm-Mode: Only aware of its own reservations○ Kubernetes: Effective handling of out-of-resource situations, but only for kubernetes

workloads● When a node is at/near capacity:

○ All CPU shares throttled equally○ The OS’s OOM killer kills processes○ All orchestrators will reschedule on OOM, but potential workload interruption

Resource Contention (cont.)

For production workloads

● For now we recommended one orchestrator per node

● Working on UX to provide simple orchestrator selection per node

Future:

● Working to address shortcomings to better support mixed orchestration

Workload Interoperability

● Networking

○ Layer 3 not connected between kubernetes & swarm

○ Batteries-included kubernetes ingress controller

○ Layer 7 routing for swarm workloads

○ Configure external DNS

● Storage: Kubernetes workloads with docker volumes via flexvolume

AuthN / AuthZ

AuthN

AuthZ

In Summary...● Docker will include an unmodified Kubernetes

distribution.

● Resource Contention mitigated via workload separation

● In EE, Authentication and Authorization integrated via standard plugin interfaces.

Thank You!

alexmavrdhiltgen