of Running Kubernetes for publication Deep Dive: The Value · 2018-09-05 · Deep Dive: The Value of Running Kubernetes on vSphere Frank Denneman, VMware, Inc. @FrankDenneman Michael

#vmworld

Deep Dive: The Valueof Running Kubernetes

on vSphereFrank Denneman, VMware, Inc.

@FrankDennemanMichael Gasch, VMware Global Inc.

@embano1

CNA1553BU

#CNA1553BU

VMworld 2018 Content: Not for publication or distribution

Disclaimer

2©2018 VMware, Inc.

This presentation may contain product features orfunctionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.


Agenda


Kubernetes Primer

Customer Scenario – Making the Case for Bare Metal

Experience Report – Kubernetes on Bare Metal vs. vSphere

QnA



Kubernetes Primer


#CNA1553BU 5©2018 VMware, Inc.

Google Search(late 1990s)

The Origin of Kubernetes



Revolutionizing the Way we build Distributed(cloud-native) Applications today.

Google Search Pillars:• Commodity• Fault-Tolerant Software• Fraction of the Cost from High-End Servers

The Origin of KubernetesGoogle Search

Source: https://ai.google/research/pubs/pub49VMworld 2018 Content: Not for publication or distribution

https://ai.google/research/pubs/pub49


Platform Engineering Responsibilities


#CNA1553BU 8©2018 VMware, Inc. CONFIDENTIAL



“We must treat the Datacenter itself as one massive Warehouse-scale Computer.”

The Origin of KubernetesThe Datacenter as a Computer

SSH




Borg(~2003)





Borg(~2003)

Cgroups(2007)





Borg(~2003)

Cgroups(2007)

Omega(~2012)




The Origin of KubernetesContainers become Mainstream

In Search for a Common Language



The Origin of KubernetesSo what is a Container, really?

Kernel Mode

Cgroups

Namespaces

Security Capabilities

Scheduler

Syscall

task_struct

…

Scheduling Entity (se)

“running”

syscall.Exec(ENTRYPOINT/CMD)*

A Structure in Kernel Memory. The Kernel has no Notion of a “Container”. It’s yet another Executable.

User Mode

Docker Engine

ContainerCreate()

* After Container Sandbox Initialization(nsenter.go/nsexec.c)

sched_classfair.c (CFS)




Borg(~2003)

Cgroups(2007)

Omega(~2012)

Docker(2013)





Borg(~2003)

Cgroups(2007)

Omega(~2012)

Docker(2013)


Kubernetes(2014)



„Kubernetes is an open-source System for automating Deployment, Scaling, and

Management of containerized Applications.”

The Origin of KubernetesContainer Orchestration



Kubernetes Cluster

KubernetesHigh-Level Architecture

Infrastructure(Compute, Storage, Networking)

Control Plane Worker

Pod Pod Pod Pod PodAPI

Kubernetes Cloud Provider



Customer ScenarioMaking the Case for Bare Metal



ABC Inc. is the Leader in Manufacturing Wayback Machines

Enterprise IT Organization with separate Infrastructure, Linux/Middleware and Development Teams (Silos)

>90% standardized and virtualized on VMware vSphere

Going through Digital Transformation to become more Customer and Feedback driven• Need to develop (iterate) faster with an agile Approach

Technical Vehicle: Containers and “cloud-native” Application Architectures• Kubernetes as the Framework to build and run these new Applications• Embrace and contribute to Open Source Software

Meet ABC Inc.



Linux Team at ABC Inc. decided to deploy Kubernetes on Bare Metal

Justification:• New (cloud-native) Applications don’t need vSphere Features like HA and vMotion• Containers are more lightweight, replacing VM’s and the Hypervisor• Kubernetes provides Hypervisor Functionality, e.g. Resource Management and HA• Virtualization reduces Performance of containerized Applications• Reduce Complexity and Costs by eliminating the Hypervisor from the Stack• IT Infrastructure not agile enough (no Self-Service)

ABC Inc.’s vSphere Team reached out to VMware for Help

ABC Inc.’s Decision to go Bare Metal



Back to 2005

merchoid.com



Experience ReportKubernetes on Bare Metal vs. vSphere



Day 0 Planning and “Green Lights”

Day 1 Experiences with first Deployments

Day 2 Container and Cluster Sprawl

Day 3 Maintenance & Availability

Terminology



Day 0Planning and “Green Lights”



Day 0Planning and “Green Lights”

Kubernetes Cluster

Infrastructure(Compute, Storage, Networking)

Cloud Provider(Custom)

External Dependencies

DNS DBs IPAM

Images CA Auth

Secrets Monitoring Logging

CustomIntegrations

Label: AZ=AZ-1VMworld 2018 Content: Not for publication or distribution


Day 0Realization: Managing Bare Metal Systems is hard



How vSphere Can Help



Average Time to get HW in DC – Unpredictable Process



Average Time to get Hardware in Data Center

86 Days

#CNA1553BU



Hardware CompatibilityDecoupling the OS from the Hardware reduces operational Overhead

a simple NIC revision change can directly impact the Kubernetes host

Virtualized hardware decouples the OS from the underlying hardware.Hardware abstraction reduces operational overhead for supported firmware versions of components.

Configuration management done at the physical layer (firmware, drivers, etc). (Drift?)(Supported?)



Non Disruptive PatchingvMotion Workload away for Hardware, Firmware or Driver Update


Need to Patch, vMotion workloadNo disruption

Need to patch – Kill workload



Kill Doesn’t matter for Stateless WorkloadsETCD isn’t stateless, and what about top 10 Workloads in Containers today?



Strong Security IsolationStrong Isolation between Workloads with efficient Resource Usage


VMs provides strong isolation between guest, allowing multi tenancy. Efficient use of resources

Containers are processes in Linux Kernel, security concerns can lead to reduced resource utilization

Tenant A Tenant B



Modern DCs operate various workloads in different packaging formats

vSphere provides unified platform for these workloads

Use your current tool and skillset to manage this workloads

Focus on creating value

Functional Use of HardwareGeneral vs Dedicated

general purpose allows mixed workloads and superior resource utilization

dedicated hardware to a particular function hinders resource optimization

#CNA1553BU



Day 1Experiences with first Deployments



Physical Host

Day 1In the old Bare Metal Days (Pre-Virtualization Era)

Kernel

App

M

Hardware

16 Cores 128GB RAID NIC-Teaming

G G W W W

NUMA sysctl nice IRQ-Balance

Bins/Libs

Almost exclusive Access to Host Resources for this App (1:1)

Best Practices for this Deployment Type were developed

App uses Host Information for Runtime Tuning

Downside: Utilization & Agility

M

G

W

OS Thread: main()

OS Thread: GC

OS Thread: Worker/PoolVMworld 2018 Content: Not for publication or distribution


Physical Host

Day 1Containers and Kubernetes on Bare Metal to the Rescue?

Kernel

Hardware

64 Cores(HT)

384GB RAID NIC-Teaming

NUMA sysctl Cgroups IRQ-Balance

Container Runtime

Kubelet

Not all Runtimes are Cgroup-aware!

How to tune per Workload?

Resource Contention and Workload Interference!

Isolation (Security)?

Utilization & Agility kube-scheduler

Node: BM001Capacity:

cpus: 64memory: 384GB

Allocatable:cpus: 60memory: 360GB

How much to reserve vs. Waste?






Hyperthreading



Hyperthreading in vSphere


‹#› 44©2018 VMware, Inc.

Consistent performance is obtained by avoiding NUMA boundaries



NUMA Architecture



NUMA Architecture



Day 2Container and Cluster Sprawl




Org

aniz

atio

n

EnvironmentVMworld 2018 Content: Not for publication or distribution



Org

aniz

atio

n

EnvironmentVMworld 2018 Content: Not for publication or distribution



Imb

alan

ce

CostVMworld 2018 Content: Not for publication or distribution





Multi-Tenancy


VM

vSphere Cluster

VM VM VM VM VM VM VMVM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Test

VM

K8S Test

VM

K8S Test

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod



Multi-TenancyIncrease in Utilization: Scale out & redistribute


VM

vSphere Cluster


K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Test

VM

K8S Test

VM

K8S Test

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod



Resource DistributionUse Per-VM Reservations or Resource Pool Structures



Day 3Maintenance & Availability



Day 3Maintenance & Availability (Control Plane)

Admission Control and Failover Capacity (MTTR)?Proactive HA?Impact of Host Maintenance/

Failure on Control Plane?



Day 3Maintenance & Availability (Workloads)

Kubernetes Control Plane

Controller Manager Scheduler

“QA”“Dev” “Prod” “QA”

4 CPUs 4 CPUs 4 CPUs

“Prod” “QA”“Dev”

2 CPUs 1 CPU 1 CPU 2 CPUs 1 CPU1 CPU 1 CPU

Only considering beta/stable and in-tree Kubernetes FeaturesDisruptive Pod Priority & Preemption, incl. Priority Queue, beta in v1.11

Example assumes shared File System for Persistent Volumes

Queue: “QA”“Dev”“Prod”pick

* Default, configurable** Fixed

pod-eviction-timeout

5min*ReconcilerMaxWaitForUnmountDuration

6min**

“Dev”

1 CPU

“QA”

1 CPU






Multi Cluster ConfigurationPriority

VM

vSphere Cluster


K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Test

VM

K8S Test

VM

K8S Test

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

Important Important Important Important

More Important More Important More Important More Important

Meh Meh Meh



HA Restart PriorityEnsure “Prod” Systems get restarted first



Restart Dependency

Works based on VM to VM rules

Only 1 level, so for A-B-C create two rules

• VM Group B depends on A• VM Group C depends on B• ETCD-Masters-Workers

Specify when to start next batch

• Resources allocated• Powered On• Guest Heartbeat• App Heartbeat

Or “HA Orchestrated Restart” as it is also called



Kubernetes Cluster Node RolesControl Plane (Masters) and Workers

VM

vSphere Cluster


K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Test

VM

K8S Test

VM

K8S Test

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

(Master) (Master) (Master) (Workers) (Worker)



Multi-TenancyDRS Affinity Rules

VM

vSphere Cluster


K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Test

VM

K8S Test

VM

K8S Test

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod




Multi-TenancyDRS Affinity Rules

VM

vSphere Cluster


K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Test

VM

K8S Test

VM

K8S Test

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod




Multiple Fault DomainsQuorum dictates Design

VM

Fault Domain A


K8S Prod

VM

K8S Prod

V

K8S Prod

VM

K8S Prod

VM

K8S Test

VM

K8S Test

VM

K8S Test

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

(Master) (Master)

Fault Domain B

(Worker)

VM

K8S Prod(Master) (Worker)

VM

K8S Prod

(Worker) (Worker)

(VM Anti-Affinity)

Host-VM Rules



DRS proactively avoid needing the use of HA• Integrates with server vendor’s monitoring software• Health states are passed to DRS• DRS reacts based on health state of hardware

None of DRS affinity/anti-affinity rules are violated

Quarantine Mode accepts workloads if performance degradation is imminent

Proactive HAMoving Workloads away at first Signs of Trouble



Admission Control ensures satisfying VM-level reservations

Percentage based works on the reservation per VM• Set aside a percentage for fail-over capacity• And the reservations (and overhead) will be removed

from the total percentage of available resources

Slots are based on the highest reservation + overhead

• If you have a 32GB mem reservation, your slot size is 32GB for memory + memory overhead

HA Admission ControlUse Per-VM Reservations to ensure Resource Availability during HA Events



VM Latency SensitivityCPU Core Isolation


THANK YOU!

#vmworld #CNA1553BU



Global Services: Reimagining Support Giving you a more proactive, personalized, effortless experience

Read All About ItSupport Insider Blog

https://blogs.vmware.com/kb/

Meet the Team Connect at the

VMVillage’s Listening Post and the Global

Services Meeting Center

Download VMware Skyline™

Visit the VMware Skyline station in the Solutions

Exchange VMworld 2018 Content: Not for publication or distribution

https://blogs.vmware.com/kb/

More Sessions on Kubernetes

Try HOL

VMware Cloud-Native Apps

Follow Us

https://blogs.vmware.com/cloudnativehttps://www.youtube.com/c/VMwareCloudNativeApps

@cloudnativeapps

Tuesday, August 28CNA2084BU Intro to VMware Kubernetes Engine—Managed K8s Service on Public CloudCNA1656BU Put a Lid on It: Securing Containers and Kubernetes on vSphereCNA3146BU Technical Deep Dive: Kubernetes Networking and Security with NSX-T on PKSCNA2755BU Architecting PKS for Production Lessons Learned from PKS Deployments

Wednesday, August 29CNA1075BU Operating and Managing Kubernetes on Day 2 with PKSCNA2009BU Run Stateful Apps on Kubernetes with PKS: Highlight WebLogic ServerCNA3124BU Deep Dive: VMware Kubernetes Engine – Kubernetes as a Service on Public Cloud

193201CNA_U VMware Kubernetes Engine – Getting Started

193101NET_U VMware Pivotal Container Service and Kubernetes – Getting Started193002CNA_U VMware vSphere Integrated Containers – Getting Started


PLEASE FILL OUTYOUR SURVEY.Take a survey and enter a drawingfor a VMware company store gift card.

#vmworld #CNA1553BU



Top 5 Best Practices for Kubernetes on vSphere



Kubernetes is Linux• Learn Components/Implementation• Don’t treat it as “Black Box” VMs which someone else will hopefully manage

vSphere Best Practices for HW and OS Configuration/Patching/Security still apply

Use a tested and certified (supported) Kubernetes Distribution • Optimal Integration• E2E Support Experience• Reduce Overhead on Customer Side (create Business Value vs. undifferentiated Plumbing)

#1The Basics (“80/20” Rule)



NUMA Rules and Sizing Best Practices still apply• Defer NUMA Handling from Guest OS to the Hypervisor

CPU/MEM Reservations for critical Components• Control Plane and PROD-Workers

DRS for intelligent Placement and X-K8s Cluster Balancing

Leverage vSphere Overcommit Capabilities, but wisely• VM “Flat Rate”, i.e. slice & dice the Host into VMs as needed• vSphere Host and Cluster Resource Controls allow Overcommit Flexibility for different

Workloads and Classes (Dev/Test vs. Prod)

#2Embrace vSphere Resource Management



Use DRS Host Groups and Affinity/Anti-Affinity Rules for increased Availability• Use “should” instead of “must” Rules for K8s whenever possible• Respected by HA

Set higher HA Restart Priorities for K8s Control Plane and Dependencies

vMotion to avoid planned Downtime, e.g. Host Maintenance

Coming soon: Automatic Kubernetes vSphere Topology & Availability Zone Awareness• https://github.com/kubernetes/kubernetes/issues/64021

#3Embrace vSphere High Availability Features


https://github.com/kubernetes/kubernetes/issues/64021


#4Embrace the vSphere Ecosystem



Read and understand Kubernetes Vendor Documentation to provide Guidance

Example:

“Lastly, set all of the VMs created to High VM Latency to ensure some additional tuningrecommended by VMware for latency sensitive workloads [...]” (Source: Red Hat OpenShift v3.9 Reference Architecture for vSphere)

#5Provide Guidance


Documents

of Running Kubernetes for publication Deep Dive: The Value · 2018-09-05 · Deep Dive: The Value of Running Kubernetes on vSphere Frank Denneman, VMware, Inc. @FrankDenneman Michael