CNA2080BU Deep Dive: How to Deploy and Operationalize or ...€¦ · Deep Dive: How to Deploy and Operationalize Kubernetes VMworld 2017 Content: Not ... –MongoDB, CouchDB, Couchbase,

Cornelia Davis, Pivotal

Nathan Ness

Technical Product Manager, CNABU

@nvpnathan

CNA2080BU

#VMworld #CNA2080BU

Deep Dive: How to Deploy and Operationalize Kubernetes

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

#CNA2080BU CONFIDENTIAL 2



bution

Agenda


1 What is the need?

2 Introducing the toolchain

3 Pivotal Container Service (PKS)

4 PKS Day 1

5 PKS Day 2VMworld 2017 Content: Not fo


bution

The Need for Operationalizing Kubernetes



bution

CONTAINERSEVENT-DRIVEN

FUNCTIONS

DATA SERVICES

MICROSERVICES

BATCHES

MONOLITHIC APPLICATIONS

Companies Have Many Ways to Package and Run Their Workloads in the Cloud




bution

Workloads that Might Be Suitable for Kubernetes

Those:

• Requiring Persistence

– MongoDB, CouchDB, Couchbase, Elastic Search, …

• Managed as a cluster

– nodes need to communicate with one another

– often with the help of service meshes such as Istio or Linkerd

– Spark, Elastic Search

• Needing new architectural primitives

• Misc things like multiple ports, etc.




bution

Workeretcd

etcd

Serving up Kubernetes Dial-tone

7

Kubernetes

etcd

kubectlRouting

MasterMaster

WorkerWorker

Responsible for the

workloads running

in K8s

Responsible for the

K8s cluster(s)

themselves

manage

#CNA2080BU CONFIDENTIAL



bution

Operational Challenges with Any Platform

Patches Patching platform components with thousands of apps running should feel normal.

Scaling Seamlessly scale platform components to accommodate changing demand.

Upgrades How do you roll out new versions of the platform with the lights on?

Operating Effort Operating a platform should require very few resources and minimum manual intervention. Otherwise, is it really providing operational benefits?

Multi-cloud Provide a reliable and smooth experience for any cloud.

Open APIs Allow platform operations from different toolsets and the creation of CD pipelines.

Consistency Provide a consistent setup experience, across different cloud environment configurations.

Setup time How long does it take to setup a real world working environment? Think hours, not weeks.

Day 1 - Build Day 2 - Operate




bution

Kubernetes - Especially Hard to Operationalize


High Availability. No out-of-the-box fault-tolerance for the cluster components themselves (masters, workers and etcd nodes).

Scaling. Kubernetes clusters handle scaling the pod/service within the Nodes, but doesn’t provide a mechanism to scale Workers, Masters & etcd VMs.

Health checks and healing. The Kubernetes cluster only does routine health checks for the health ofworkloads running on Nodes.

Upgrades. Rolling upgrades on a large fleet of clusters is hard. Who manages the system it runs on?



bution

Introducing BOSH



bution

Powered by BOSH


Pivotal container service ops

BOSH is an open source tool

for release engineering,

deployment, lifecycle

management, and monitoring

of distributed systems.

BOSHPackaging w/ embedded OS

Server provisioning on any IaaS

Software deployment across availability zones

Health monitoring (server AND processes)

Self-healing w/ Resurrector

Storage management

Rolling upgrades via canaries

Easy scaling of clusters



bution

Powered by BOSH


Pivotal container service ops

Packaging w/ embedded OS

Server provisioning on any IaaS

Software deployment across availability zones

Health monitoring (server AND processes)

Self-healing w/ Resurrector

Storage management

Rolling upgrades via canaries

Easy scaling of clusters

Workeretcdetcd

Kubernetes

etcd

MasterMaster

WorkerWorker

BOSHVMworld 2017 Content: Not fo


bution

Primary BOSH Entities


Workeretcdetcd

etcd

MasterMaster

WorkerWorker

BOSH

The definition of each of the nodes in the

cluster, including:

• The bits installed on a node (packages)

• The processes started on a node (jobs)


cluster, including:




cluster, including:



• Parameterized

BOSH release

A declaration of the desired state of the

cluster:

• Assembly of the components from BOSH

releases (relationships, dependencies)

• Parameter values

BOSH deployment

Relationship to the underlying infrastructure

BOSH cloud config

Kubernetes



bution

The Workflow


Workeretcdetcd

etcd

MasterMaster

WorkerWorker

BOSH


cluster, including:




cluster, including:




cluster, including:



• Parameterized

BOSH release


cluster:




BOSH deployment


BOSH cloud configSTEP 1: Install and configure BOSH

STEP 2: Install and Manage Kubernetes

Kubernetes



bution

Pivotal Container Service (PKS)



bution

Project Kubo


Uniform way to instantiate, deploy, and manage highly available Kubernetes clusters. On any cloud.

Launched by Pivotal & Google Feb 2017, Donated to Cloud Foundry Foundation June 2017

“Day 1” Build● Deploy Kubernetes cluster via BOSH

“Day 2” Operate● Self-healing VMs and monitoring via

BOSH● Elastic scaling for clusters● Rolling upgrades to latest Kubernetes

release● High-availability and multi-AZ supportVMworld 2017 Content: N

ot for publicatio

n or distribution

Workeretcdetcd

Kubernetes

etcd

MasterMaster

WorkerWorker

BOSH

This forms the

Open Core of

Pivotal Container Service(PKS)

Release

templates

Manifest

Kubo Release

bosh deploy

Kubo Provides Specification of K8S Components

#CNA2080BU CONFIDENTIAL

17



bution

Provides the control plane for provisioning and managing Kubo releases

Joint development effort between Pivotal, VMWare and Google

Kubernetes Dial Tone:

• Health management

• Aggregated Metrics and Logging

• Autoscaling

• Persistence interface

Control Plane:

• Provisioning Engine

• Self-service Clusters

• Software Update Automation

• Load balancing

• Networking

• Multi-tenancy




bution

PKS Leverages the Power of BOSH

19

PKS

Release

templates

Manifest

Kubo Release

BOSHVMworld 2017 Content: N

ot for publicatio

n or distribution

Kubernetes Cluster – Day 1Deploy



bution

Starting with a BOSH Deployment...


Workeretcdetcd

etcd

MasterMaster

WorkerWorker

BOSH


cluster, including:




cluster, including:




cluster, including:



• Parameterized

BOSH release


cluster:




BOSH deployment

Kubernetes



bution

Message Bus

vSphereBOSH

DB

BOSH Director

Blobs

Health Monitor

Deployment

• Packages

• Blobs

• Source

• Jobs

• Manifest

Deploy my

K8sWorker VMs

etcd

Target VMMaster

Target VMWorker

Target VM

Deploying a Kubernetes Cluster with Cloud Foundry BOSH




bution

Kubernetes Cluster – Day 2Operationalize



bution

1 Managing Health

2 Scaling

3 Upgrade

Day 2: Operationalize



bution

K8s Cluster Health: Processes are Monitored


vSphereBOSH

Master

AGENT

etcd

AGENT

Worker

AGENT

Message Bus

Health Monitor

Responses:

pager

email

monitoring

…



bution



vSphereBOSH

Master

AGENT

etcd

AGENT

Worker

AGENT

Message Bus

Health Monitor

Responses:

pager

email

monitoring

…



bution



vSphereBOSH

Master

AGENT

etcd

AGENT

Worker

AGENT

Message Bus

Health Monitor

Responses:

pager

email

monitoring

…



bution

K8s Cluster Health: VMs are Monitored


vSphereBOSH

Master

AGENT

etcd

AGENT

Worker

AGENT

Message Bus

Health Monitor

Responses:

pager

email

monitoring

ressurector

…

BOSH Director

Desired State Actual State



bution



vSphereBOSH

Master

AGENT

etcd

AGENT

Worker

AGENT

Message Bus

Health Monitor

Responses:

pager

email

monitoring

ressurector

…

BOSH Director




bution



vSphereBOSH

Master

AGENT

etcd

AGENT

Worker

AGENT

Message Bus

Health Monitor

Responses:

pager

email

monitoring

ressurector

…

BOSH Director


CPI



bution

1 Managing Health

2 Scaling

3 Upgrade





bution

Primary BOSH Entities


Workeretcdetcd

etcd

MasterMaster

WorkerWorker

BOSH


cluster, including:




cluster, including:




cluster, including:



• Parameterized

BOSH release


cluster:




BOSH deployment


BOSH cloud config

Kubernetes



bution

instance_groups:

- name: etcd

instances: 3

networks:

- name: &network-name ((deployments_network))

azs: [z1]

jobs:

- name: etcd

release: kubo-etcd

properties:

etcd:

require_ssl: false

peer_require_ssl: false

stemcell: trusty

vm_type: common

persistent_disk_type: 5120

- name: master

instances: 2

networks:

- name: *network-name

azs: [z1]

jobs:

- name: cloud-provider

release: kubo

properties: {}

- name: kubernetes-api

release: kubo

properties:

admin-username: admin

admin-password: ((kubo-admin-password))

...

- name: kubeconfig

release: kubo

properties:

...

...

stemcell: trusty

vm_type: master

- name: worker

instances: 3

networks:


azs: [z1]

jobs:

- name: docker

release: docker

properties:

...

- name: kubeconfig

release: kubo

properties:

...

- name: kubelet

release: kubo

properties:

...

- name: kubernetes-proxy

release: kubo

properties:

...

stemcell: trusty

vm_type: worker


Manifest

33



bution

instance_groups:

- name: etcd

instances: 3

networks:

- name: &network-name ((deployments_network))

azs: [z1]

jobs:

- name: etcd

release: kubo-etcd

properties:

etcd:

require_ssl: false

peer_require_ssl: false

stemcell: trusty

vm_type: common


- name: master

instances: 2

networks:


azs: [z1]

jobs:

- name: cloud-provider

release: kubo

properties: {}

- name: kubernetes-api

release: kubo

properties:

admin-username: admin

admin-password: ((kubo-admin-password))

...

- name: kubeconfig

release: kubo

properties:

...

...

stemcell: trusty

vm_type: master

- name: worker

instances: 3

networks:


azs: [z1]

jobs:

- name: docker

release: docker

properties:

...

- name: kubeconfig

release: kubo

properties:

...

- name: kubelet

release: kubo

properties:

...

- name: kubernetes-proxy

release: kubo

properties:

...

stemcell: trusty

vm_type: worker


Scaling is a matter of changing the number of

instances and telling BOSH to

“make it so”



“make it so”



“make it so”

34



bution

1 Managing Health

2 Scaling

3 Upgrade




bution

update:

canaries: 1

max_in_flight: 1

serial: true

canary_watch_time: 10000-300000

update_watch_time: 10000-300000

K8s Cluster Upgrade: Canary Deployments


Manifest VMworld 2017 Content: Not fo


bution



V1.0 V1.1

# OF CANARIES: 2

MAX IN FLIGHT: 2

EXAMPLE:

CANARIES



bution



# OF CANARIES: 2

MAX IN FLIGHT: 2

EXAMPLE:

V1.1 V1.2Once failed, Canary VMs are kept

for troubleshooting purposes.



bution

Operationalizing at Scale



bution

Supporting Kubernetes Needs at Scale

40

PKS Service Broker

Release

templates

Manifest

Kubo Release

BOSH



bution

41

PKS Service Broker

Release

templates

Manifest

Kubo Release

BOSH

create cluster(with upgrade policy)

Supporting Kubernetes Needs at Scale

manage

Thousands

Ones

https://thenewstack.io/comcast-1500-developers-working-cloud-foundry



bution

https://thenewstack.io/comcast-1500-developers-working-cloud-foundry

Let Us Show You…



bution

43#CNA2080BU CONFIDENTIAL



bution

PaaS Control Plane

etcd

API-Server

Scheduler

NCM

Infra

Kubernetes

Adapter

CloudFoundry

Adapter

Libnetwork

Adapter

NSX Container Plugin

Mesos

Adapter

NSX

Manager

API Client

Proj: foo Proj: bar

NSX topology for K8s / CF

• NSX Container Plugin (NCP) for integrating with Kubernetes

• NSX Features for K8s PODs

• IP address per container / POD

• Container Network – Routed (BGP) & NATed mode

• Microsegmentation – via K8s Network Policy or native NSX APIs (mapping

K8s labels to NSX tags)

• Network & Security automation – created as part of app deployment

• Multi-tenant network topologies

NSX-T Integration



bution

Structured Data

Metrics Alerts Events

VMware vRealize

Operations

Capacity, Performance and

Configuration Management Events

Launch in Context

Unstructured Data

Logs Messages

VMware vRealize

Log Insight

Log analytics, aggregation,

and search

Virtual Applications

vRealize Ops, vRealize Log Insight For Comprehensive Visibility



bution

46

K8S Summary –Nodes, Pods, etc.

K8S Topology -Health

K8S Pods - Health

vRealize Ops – Managing Kubernetes Clusters



bution

47

K8S Pod Relationship to Components

K8S Alerts

K8S Alerts

vRealize Ops – Kubernetes Integration Details



bution

4

8

UI and API Backend

Advanced Analytics Engine

Metrics Collection and Storage

Iterate & Troubleshoot Issues

Trend & Alert on Anomalies

Visualize Metrics at Scale

Self-Service Metrics Analytics for All

Engineering & Business

Introducing Wavefront By VMware SaaS-Based Metrics Monitoring and Analytics Platform



bution

App Containers

Docker Host

Docker Swarm

Container Metric Collector

Docker Host

Docker Host

Docker Cluster

AmazonECS

Real-time insight into Docker containers and orchestration

systems Kubernetes, Pivotal Cloud Foundry, Amazon ECS

Wavefront – Container Monitoring Suite



bution

50

Need Harbor screenshot

user management & access control

role-based access control

AD/LDAP integration

security

vulnerability scanning

content trust - image signing

policy based image replication

audit and logs

restful API

lightweight & easy deployment

open-source under Apache 2 license

Registry – Enterprise-grade Private Registry



bution

51

Registry – Content Trust, When Enabled Un-signed Images Can’t Be Pulled



bution

52

Registry – Image Vulnerability Scanning Details



bution

VMware PKS

Analytics Automation

SecurityOperations

MonitoringLogging

Physical Infrastructure

Container Registry

vSphere vSAN

Kubernetes on BOSH (Kubo)

NSX

BOSH

GCP Service Broker

masteretcd workermasteretcd worker



bution



bution



bution

Documents

CNA2080BU Deep Dive: How to Deploy and Operationalize or ...€¦ · Deep Dive: How to Deploy and Operationalize Kubernetes VMworld 2017 Content: Not ... –MongoDB, CouchDB, Couchbase,