MULTITENANCY IN KUBERNETES

WHAT COMPANIES CARE ABOUT

Velocity

2

Cost

Hello!

I AM KATHARINA PROBST

I’m a Senior Engineering Manager at Google.You can find me at www.linkedin.com/in/katharina.probst

3

WHY MULTITENANCY

4

KUBERNETES AT A GLANCE

5

masteruser CLI/API/UI

node

node

node

NODES

ONE USER, ONE CLUSTER

6

masteruser CLI/API/UI

node

node

node

NODES

MULTIPLE USERS, MULTIPLE CLUSTERS

7

masteruser CLI/API/UI

node

node

node

NODES

masteruser CLI/API/UI

node

node

node

NODES

SPRAWL OF MANY CLUSTERS

8

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

HOW DOES THIS SCALE FINANCIALLY?

9

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

HOW DOES THIS SCALE OPERATIONALLY?

10

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

masteruser CLI/API/UI

kubelet

kubelet

kubelet

NODES

...Namespace 2

Namespace 1

...

MANY USERS, ONE CLUSTER

11

master

User 2

CLI/API/UIUser 3

User 1

User n

Namespace n

MULTITENANCY USERS

12

CLUSTER ADMIN

◦ Set up cluster and namespaces. Set up resource limits.

◦ Ensure consistency across namespaces in the cluster.

◦ Operate the clusters (e.g., respond to incidents).

CLUSTER USER

◦ Start/stop/manage their own app(s) in their own namespace(s).

◦ Understand their namespace(s)’ resource limits.

◦ Don’t trample on other tenants.

NAMESPACE ADMIN

Admin rights to specific namespace(s)

MODELS OF MULTITENANCY

13

HARD MULTITENANCY

◦ Zero-trust tenants

◦ Not yet widely used in production

◦ Ongoing work in kubernetes community to strengthen

14

WHAT WILL HARD MULTITENANCY TAKE?

15

WHAT WILL HARD MULTITENANCY TAKE?

◦ Tenants can’t

▫ DoS/impact access to others’ resources

▫ See each others’ stuff (e.g., by intercepting network traffic or

accessing stored data)

16

WHAT WILL HARD MULTITENANCY TAKE?

◦ Tenants can’t

▫ DoS/impact access to others’ resources

▫ See each others’ stuff (e.g., by intercepting network traffic or

accessing stored data)

◦ Resource objects don’t collide, e.g., custom controllers/CRDs

17

WHAT WILL HARD MULTITENANCY TAKE?

◦ Tenants can’t

▫ DoS/impact access to others’ resources

▫ See each others’ stuff (e.g., by intercepting network traffic or

accessing stored data)

◦ Resource objects don’t collide, e.g., custom controllers/CRDs

◦ Control plane (master) resources are shared fairly

MODELS OF MULTITENANCY

18

SOFT MULTITENANCY

◦ Tenants are more trusted◦ Often used within one enterprise◦ Different teams → different namespaces

◦ Used in practice and in production◦ Often used in combination with infrastructure/platform built on

top by centralized team for consistent security, networking, etc.

MULTITENANCY PRIMITIVES

19

ACCESS CONTROL

Use policies to ensure that tenants can access only what they should have access to

FAIR SHARING

Enforce limits per tenant[Better developed on data plane; control plane WIP]

ISOLATION

Ensure tenants cannot access each others’ workloads, secrets, etc. (security isolation)

ACCESS CONTROL - RBAC

20

ClusterRole A pre-set of capabilities, cluster-wide

Role Like ClusterRole, but namespace-scoped

ClusterRoleBinding Give permissions defined in a ClusterRole

RoleBinding Like ClusterRoleBinding, but namespace-scoped

◦ Use RBAC for controlling access to Secrets etc.

◦ Pod Security Policy (e.g., access to volume types, privileged): enable

fine-grained authorization of pod creation and update

◦ Network Policy (ingress, egress): control which pods can talk to each other

◦ Make Custom Resource Definitions namespace-scoped

◦ Sandboxes, e.g., gVisor

▫ Ensure security isolation of pods

▫ Ensure that information is not leaked between untrusted tenants

21

ISOLATION

◦ Resource Quotas (with Limit Range defaults) for CPU, memory, object

counts

◦ Pod Priority, Quality of Service Classes

◦ Node

▫ Taints & Tolerations: allow a node to repel a set of pods

▫ Labels and node selectors (less flexible than Affinity)

◦ Pod

▫ Affinity

▫ Pod Anti-affinity

22

FAIR SHARING - DATA PLANE (NODES)

23

DATA PLANE VS. CONTROL PLANE MULTITENANCY

Much of what we’ve talked about is sharing the data plane (nodes)

24

CONTROL PLANE MULTITENANCY

...Namespace 2

Namespace 1

...

MANY USERS, ONE CLUSTER

25

master

User 2

CLI/API/UIUser 3

User 1

User n

Namespace n

Not able to manage multiple clusters →

1:1 mapping between master and cluster, though cluster may have multiple

tenants

26

MULTITENANT API SERVER?

...Namespace 2

Namespace 1

...

MANY USERS, ONE CLUSTER

27

master

User 2

CLI/API/UIUser 3

User 1

User n

Namespace n

All tenants share master (incl. Secrets, ConfigMap), but RBAC helps

Little protection against individual tenants DoSing each other

28

MULTITENANT API SERVER?

Max inflight requests: mechanism for protecting API server against CPU and

memory overloads

Current problem to address: Tenants can crowd each other out (accidentally or

on purpose)

29

BETTER API SERVER FAIRNESS (UNDERWAY)

Proposal generalizes … max-in-flight request handler in the apiserver to make

more distinctions among requests and provide prioritization and fairness among

the categories of requests.”

30

BETTER API SERVER FAIRNESS (UNDERWAY)

BETTER API SERVER FAIRNESS (UNDERWAY)

Multiple priority levels, each has queues

Within each priority level, queues compete evenly

31

WHAT COMPANIES CARE ABOUT

Velocity

32

Cost

◦ Use multitenancy for improved resource efficiency, cost, and operations

◦ Different models of multitenancy

▫ Soft

▫ Hard

◦ Hard multitenancy is still work in progress

◦ Soft multitenancy is already used in production by various companies

▫ But is often coupled with a shared internal platform to gain

consistency of networking, security, etc. across teams

33

KEY TAKE-AWAYS

QUESTIONS?

◦ Project plan for multitenancy

▫ Building CRD for more automatic management of namespaces

◦ API Machinery KEP for improved resource sharing

◦ Cluster multi-tenancy for a good overview

35

Links for additional details