Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
#vmworld
Architecting PKSfor Production:
Lessons Learnedfrom PKS Deployments
Romain Decker, VMware, Inc.Dominic Foley, VMware, Inc.
CNA2755BE
#CNA2755BE
VMworld 2018 Content: Not for publication or distribution
Disclaimer
2©2018 VMware, Inc.
This presentation may contain product features orfunctionality that are currently under development.
This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.
VMworld 2018 Content: Not for publication or distribution
Agenda
3©2018 VMware, Inc.
Containers & Kubernetes
PKS Architecture
PKS Deployment
VMworld 2018 Content: Not for publication or distribution
4©2018 VMware, Inc.
Containers/Kubernetes Levelset
VMworld 2018 Content: Not for publication or distribution
5©2018 VMware, Inc.
By 2020, 75% Applications Supporting Digital Business will be
“Built” not “Bought”
Modern Apps Digital TransformationApp Velocity & Customization Drive Competitive Advantage
Software Customization Innovation/Disruption
Source: Gartner
Speed
Boot EnvironmentsRapidly
Portability
Ability to MoveContainers Freely
Lightweight
Minimal Resources Needed
Containers are at the forefront of
what enables app velocity and
customizationVMworld 2018 Content: Not for publication or distribution
6©2018 VMware, Inc.
Cloud-Native application (CNA) - a methodology of building and running applications that fully exploits the power of the cloud computing, offers on-demand, limitless computing power, whether on public or private cloud.
Microservices architecture with small code-base packaged in containers is what enables CNA
Modern Applications are based on Distributed Microservices …that require containers to run
REST API
REST API
REST API
REST API
REST API
REST API
APIGateway
WEBUI
WEBUI
DeveloperDeveloper
Microservices• Small, modular code base • Easier to update, scale • Runs in containers
Monoliths • Single, large code base • Slower to update, scale • Runs in VMs or containers
VMworld 2018 Content: Not for publication or distribution
7©2018 VMware, Inc.
Application
Operating System & Dependencies
PhysicalInfrastructure
Application
Operating System & Dependencies
PhysicalInfrastructure
Containers and VMsA practical comparison
OS Dependencies
Compute | Net | Sec | Storage
OS Abstraction
Container Host OS
Container
Compute | Net | Sec | Storage
Hard Problem
Easier
Ubiquitous
Ubiquitous
Configuration Management
https://youtu.be/L1ie8negCjc
How does “someone” create this?
VMworld 2018 Content: Not for publication or distribution
8©2018 VMware, Inc.
Business App
Docker and Kubernetes
Core Docker functionality is ability run containers • Manual, no fault tolerance, coordinating
scale/upgrades, etc
Scheduling, provisioning, and resource management of multiple containers
• Major Container Solutions Kubernetes Support
• Public Clouds Kubernetes Container Service
$docker run container_web
$docker run container_web
$docker run container_LB
$docker run container_DB
$kubctl create –f App.yaml
The “App”
Wanted: Container Orchestrator!
Kubernetes Cluster
ContainersOne at a Time
Kubernetes (aka K8s) Orchestrating Multiple Containers
Kubernetes in 5 minhttps://youtu.be/PH-2FfFD2PU
VMworld 2018 Content: Not for publication or distribution
9©2018 VMware, Inc.
PlatformOps
Infrastructure
Compute Network Monitoring
Security Storage
Architecting w/Specific Application Requirements
Kubernetes Cluster
vSphere NSX Wavefront
NSX Datastores
Platform Operator
Focus on mappingKubernetes constructs to
a given infrastructure
Load Balancer
Persistent Volumes
Resources / Availability Zone
Security Policy
Application Metrics
ELK Spark Nth App
K8s API
Common
App Devarchitects with native
Kubernetes constructs
the SDDC & public clouds
AppDev
Mapping Kubernetes to SDDChttps://youtu.be/ex8jY7HNnUY
VMworld 2018 Content: Not for publication or distribution
10©2018 VMware, Inc.
Challenges of Running Kubernetes in Production
Source: Cloud native Computing Foundation User Survey 2017
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Difficultychoosing an
orchestrationsolution
Reliability ScalingDeployments
Logging Complexity Networking Monitoring Storage Security
Addressed by VMware PKS
VMworld 2018 Content: Not for publication or distribution
11©2018 VMware, Inc.
Desired state of Application
VMware PKS & Kubernetes
11
Container/Appscheduling, scale,
resiliency, and Day 2
Desired state of Kubernetes
Clusters
Kubernetes cluster creation, scale, resiliency, and Day 2
AppsApps
AppsApps
AppsApps
VMware PKS
VMworld 2018 Content: Not for publication or distribution
12©2018 VMware, Inc.
Kubernetes is Only One Layer of the Container Service Stack
Image Registry
Framework Lifecycle Management
Security and Networking
Persistence
Virtual Infrastructure
Physical Infrastructure
Mo
nito
ring
, Lo
gg
ing
, Ana
lyti
cs
Cluster Health Monitoring, Healing and Lifecycle Management
Scheduling, Orchestration, Service Creation
vSAN, Hatchway
vSphere
Physical Infrastructure
NSX-T
PKS Control Plane
BOSH (cluster LCM)
Kubernetes
Harbor
vRe
aliz
e S
uite
PKS
VMworld 2018 Content: Not for publication or distribution
13©2018 VMware, Inc.
PKS Architecture
VMworld 2018 Content: Not for publication or distribution
14©2018 VMware, Inc.
VMware PKS on SDDC Rapidly deliver and operationalize next-generation apps
Physical Infrastructure
BOSH
NSX-T
Service Broker
vSANvSphere
etcd worker
Container Registry
master etcd workermaster
PKS Control Plane
Kubernetes Cluster Kubernetes Cluster
vRealizeAutomation
vRealizeLog Insight
vRealizeOperations
vRealizeNetwork Insight
Wavefrontby VMware
VMworld 2018 Content: Not for publication or distribution
15©2018 VMware, Inc.
Identifying PKS Components
OPS MANAGER: provide UI to install Bosh Director and PKS Control Plane VM
BOSH: deploy and manage Kubernetes clusters
PKS: front end API for users to interact with PKS
HARBOR: private container registry
Management and data planes
vSphere
Physical Hardware
workermaster
Kubernetes Cluster
OPS MANAGER
BOSH
HARBOR
P
PKSworkermaster
Kubernetes Cluster
PKS DATA PLANE PKS MANAGEMENT PLANE
VMworld 2018 Content: Not for publication or distribution
16©2018 VMware, Inc.
Availability zones allow you to provide high-availability and load balancing to VMs.
Ops Manager will balance the instances across all of the configured availability zones (AZ’s).
Availability Zones (AZ)vSphere CPI Tile Configuration
Resource Pool could be left blank if using a vSphere cluster as the AZ, or could be used to limit resources consumption for example.
Add new Availability Zones• Minimum = 2 (Management x1, Kubernetes Node VMs x1)• Recommended = 4 to 7 (Management x1, multiple choices
for Kubernetes Masters and Workers)
VMworld 2018 Content: Not for publication or distribution
17©2018 VMware, Inc.
Availability Zones (AZ)PKS Tile Configuration (Assign AZ’s)
SINGLETON OPS MANAGER JOBS
• A service where only a single instance (VM) is deployed, e.g. Ops Manager, BOSH VM, Services Broker VM, Harbor.
• Singleton jobs are commonly the infrastructure/management VM’s and usually reside in the Management AZ.
• Otherwise singleton jobs can share the same AZ as the balanced jobs.
BALANCED OPS MANAGER JOBS
• A balanced job will have multiple instances deployed, e.g. a Kubernetes cluster with 3x Master nodes would be balanced across 3x AZ’s.
• This is why it is important to map your physical infrastructure to your Availability Zones!
VMworld 2018 Content: Not for publication or distribution
18©2018 VMware, Inc.
COMPUTE CLUSTER 2 COMPUTE CLUSTER 3COMPUTE CLUSTER 1MANAGEMENT CLUSTER
Topology Example – Multi Compute ClustersAZ are used to set locality of a VM against different locations
AZ-MGMT
P
AZ-COMP-01 AZ-COMP-02 AZ-COMP-03
workermaster
worker
workermaster
worker
workermaster
worker
workermaster
worker
worker
worker
worker
worker
worker
worker
worker
worker
PKS CLUSTER 1Medium plan, multi-master
PKS CLUSTER 2Small plan, single master
Singletons placed in AZ-MGMT
VDS
1:1 mapping between AZ and vSphere Clusters
Each compute cluster can resides in a dedicated rack or room
Storage must be accessible by all ESXi servers hosting Kubernetes Node VMs
Cluster doesn’t participate inNSX-T Fabric
NSX-T Transport Nodes (Geneve)
STORAGE STORAGEVMworld 2018 Content: Not for publication or distribution
19©2018 VMware, Inc.
COMPUTE CLUSTERMANAGEMENT CLUSTER
Topology Example – Single Compute ClusterResource pools used to segment availability zones
STORAGE
AZ-MGMT
AZ-COMP-01AZ-RES-01
AZ-COMP-02AZ-RES-02
AZ-COMP-03AZ-RES-03
workermaster
worker
workermaster
worker
workermaster
worker
workermaster
worker
worker
worker
worker
worker
worker
worker
worker
worker
PKS CLUSTER 1Medium plan, multi-master
PKS CLUSTER 2Small plan, single master
Singletons placed in AZ-MGMT
VDS
Mapping between AZ and Resource Pools
P
As Resource Pools are used to define AZ, there is no guarantee that Kubernetes Master Nodes will land on different ESXi hosts.
Storage must be accessible by all ESXi servers hosting Kubernetes Node VMs. vSAN can be used in this scenario.
Cluster doesn’t participate inNSX-T Fabric
STORAGE
NSX-T Transport Nodes (Geneve)
VMworld 2018 Content: Not for publication or distribution
20©2018 VMware, Inc.
Non-routable (internal to NSX-T) doesn’t imply non unique subnets
Requirements (scale, troubleshooting)
Dependencies
Deployment philosophies
CHOICES BASED ON
ADDITIONAL CONSIDERATIONS
Networking TopologyNO-NAT and NAT choices
EXTERNAL TO NSX-T(ROUTABLE / NO-NAT)
INTERNAL TO NSX-T (ROUTABLE / NO-NAT)
INTERNAL TO NSX-T (NON ROUTABLE / NAT)
INTERNAL TO NSX-T | (ROUTABLE / NO-NAT)
INTERNAL TO NSX-T(NON-ROUTABLE / NAT)
> PKS MANAGEMENT NETWORK <
> POD NETWORKS <
INTERNAL TO NSX-T | (NON-ROUTABLE / NAT)
> NODE NETWORKS <
VMworld 2018 Content: Not for publication or distribution
21©2018 VMware, Inc.
PKS & NSX-T Networking IntegrationDesign considerations
NAT MODE
Enable NAT mode for node network
POD IP BLOCK
Will be carved out to create networks to host Kubernetes pods belonging to the same namespace
Should be a multiple of /24
POOL ID
Used for: K8S Master VIP, SNAT from pods, Kubernetes Service kind (LoadBalancer L4),
Kubernetes Ingress kind (L7)
Cannot be on the same subnetas the uplink/transit network
T0 MAPPING
PKS supports only a single T0 currently
T0 must be configured in Active-Standbyregardless of networking topology
NODE IP BLOCK
Will be carved out to create networks to host Kubernetes cluster node VMs
Should be a multiple of /24
Scale is directly impacted by IP Blocks and Pool configurationVMworld 2018 Content: Not for publication or distribution
22©2018 VMware, Inc.
Networking Topology: Option #1
CONSIDERATIONS
• PKS Management external to NSX-T, deployed on a classic vSphere port group
• PKS Management and vSphere / NSX Management networks can be combined
PKS Management external to NSX-T + NO-NAT
POD NETWORK – ‚PKS-INFRASTRUCTURE‘
POD NETWORK – ‚KUBE-SYSTEM‘
KUBERNETES NODES
POD NETWORK – ‚DEFAULT‘
P
T0
T1
VIP
PHYSICAL NETWORK
T1
T1
T1
T1
MASTER W W W W
PKS MANAGEMENT
MANAGEMENT NA
T
NO NAT
NO
NA
T
ROUTABLE IP
VMworld 2018 Content: Not for publication or distribution
23©2018 VMware, Inc.
Networking Topology: Option #2
CONSIDERATIONS
• PKS Management internal to NSX-T, deployed on a logical switch
• The tier-1 logical router and logical switch required for the PKS Management network must be created upfront
PKS Management internal to NSX-T + NO-NAT
POD NETWORK – ‚PKS-INFRASTRUCTURE‘
POD NETWORK – ‚KUBE-SYSTEM‘
KUBERNETES NODES
POD NETWORK – ‚DEFAULT‘
P
T1
VIP
PHYSICAL NETWORK
T1
T1
T1
T1
MASTER W W W W
PKS MANAGEMENT
MANAGEMENT NA
TN
O N
AT
ROUTABLE IP
T0
NO NAT
T1VMworld 2018 Content: Not for publication or distribution
24©2018 VMware, Inc.
Networking Topology: Option #3
CONSIDERATIONS
• PKS Management internal to NSX-T, deployed on a logical switch
• The tier-1 logical router and logical switch required for the PKS Management network must be created upfront
• DNAT rules required for PKS Management
PKS Management internal to NSX-T + NAT
POD NETWORK – ‚PKS-INFRASTRUCTURE‘
POD NETWORK – ‚KUBE-SYSTEM‘
KUBERNETES NODES
POD NETWORK – ‚DEFAULT‘
P
T1
VIP
PHYSICAL NETWORK
T1
T1
T1
T1
MASTER W W W W
PKS MANAGEMENT
MANAGEMENT NA
T
ROUTABLE IP
T0
NAT
NA
T
T1VMworld 2018 Content: Not for publication or distribution
25©2018 VMware, Inc.
Kubernetes Cluster Nodes VM Storage & Persistent Volumes Hatchway solving persistent storage challenges in Kubernetes
worker workermaster
VMDK
vSAN Considerations• Availability zones do not map with vSAN Fault Domain• PKS with vSAN stretched cluster is not supported• vSAN is a vSphere cluster construct
worker
VMFSNFS
VSAN
Persistent Volumes Considerations• SDRS (Storage DRS) not supported on VMs hosting
Kubernetes Clusters• Datastore must be accessible by all ESXi servers
hosting Kubernetes VMs
PERSISTENT VOLUMEMapped to a VMDK usingProject Hatchwayhttps://vmware.github.io/hatchway/
worker
VMworld 2018 Content: Not for publication or distribution
26©2018 VMware, Inc.
PKS Design Highlights
InfrastructureUnderstand how
elements relate between themselves and build for
scale
NetworkingThe network topology depends on scale and
deployment philosophy
StorageHatchway is your friend, if the underlying storage
aligns to your Kubernetes clusters
VMworld 2018 Content: Not for publication or distribution
27©2018 VMware, Inc.
PKS Deployment
VMworld 2018 Content: Not for publication or distribution
‹#› 28©2018 VMware, Inc.
Get the infrastructure ready: 90% ready is not good enough.
VMworld 2018 Content: Not for publication or distribution
29©2018 VMware, Inc.
RequirementsPlanning is crucial for a successful deployment
SOFT HARD INFRA
INFRASTRUCTURE READINESS
Core infrastructure
vSphere (topology, permissions)
vSphere HA & DRS
NTP / DNS (forward and reverse)
Co-existence with NSX-V or PAS
SOFTWARE
vSphere 6.5 U1, 6.5 U2, 6.7
NSX-T 2.2, 2.3
HARDWARE
Hardware Compatibility List (HCL)
Resource Requirements
VMworld 2018 Content: Not for publication or distribution
30©2018 VMware, Inc.
Networking Requirements
NET
PKS
Network assignments
Reserved IP range for PKS (10.100.200.0/24)
Reserved IP ranges for Docker & Harbor (172.17.0.1/16, 172.18.0.1/16, 172.19.0.1/16, 172.20.0.1/16, 172.21.0.1/16, 172.22.0.1/16)
PHYSICAL CONNECTIVITY
Transport VLAN for Geneve, MTU (≥ 1700)
Transit VLAN(s)
Dynamic routing using eBGP (BFD recommended) or static routing (HA VIP recommended)
NSX-T Large Edge VM
• OpsManager communicates with vCenter and ESXi hosts
• Bosh communicates with vCenter, ESXi hosts, and Kubernetes master and worker nodes
• NCP should be able to reach NSX Manager
• Kube-DNS (on K8s worker node) should be able to reach K8s master node.
• K8s worker nodes should be able to reach vCenter (for persistent volumes – Hatchway project)
FIREWALL
VMworld 2018 Content: Not for publication or distribution
31©2018 VMware, Inc.
Deployment WorkflowNSX-T dial tone: infrastructure readiness for PKS
FOUNDATION
NSX-T Manager, Controller and Edge appliances OVA images deployment
Controller Cluster configuration and registration
OBJECTS AND PROFILES
Uplink profiles
IP Pools, IP Blocks
Transport Zones / N-VDS
Logical Switches (Overlay and VLAN for external connectivity)
EXTERNAL CONNECTIVITY
T0 router creation (Active-Standby), peering with physical router
T1 routers creation
** NAT rules creation
PREPARATION
Certificates (creation and registration against NSX-T Manager using FQDN): Super User Principal Identity Certificate and CA Certificate
Compute Managers
FABRIC
Edges and ESXi configured as transport nodes
Edge cluster creation
** If required, based on the network topology implemented
VMworld 2018 Content: Not for publication or distribution
32©2018 VMware, Inc.
Deployment WorkflowPivotal Container Service
FOUNDATION
Ops Manager OVA deployment
Configure Authentication System
PKS CONFIGURATION
Import PKS Tile
Assign AZ and Networks
Define PKS API FQDN
Configure Plans
Kubernetes Cloud Provider: vSphere IaaS
NSX-T integration: Container Networking Interface
OPS MANAGER FOR VSPHERE
vCenter config (DC, datastores, networking)
Availability Zones creation
Networks creation
PKS CONFIGURATION
Monitoring (Wavefront integration)
Errands: NSX-T Validation required
Upload Stemcell
FOUNDATION
Import Harbor Tile
Configure Harbor
Apply Changes
VMworld 2018 Content: Not for publication or distribution
33©2018 VMware, Inc.
Don’t Forget
BOSH DIRECTOR
Enable VM Resurrector Plugin: checked
Enable Post Deploy Scripts: checked
ERRANDS
NSX-T Validation errand: On
VMworld 2018 Content: Not for publication or distribution
34©2018 VMware, Inc.
TIP – Create a PKS Management VMCLI Tools
UUAC
PKS
KUBECTL
BOSH
OM
BOSH
BOSH CLI
Manage and troubleshoot PKS deployments (tasks, etc.)
Provide information of VMs that BOSH manages
USER AUTHENTICATION AND AUTHORIZATION CLI
Create and manage PKS users
Grant PKS cluster access to users
PKS CLI
Create, delete or scale-out PKS clusters
Get PKS credentials
OPS MANAGER CLI
Interact with Ops ManagerKUBECTL
Interact with Kubernetes by controlling the cluster manager
Deploy applications
Application dev
Application ops
Platform/Site ReliabilityEngineer or vSphere Admin
VMworld 2018 Content: Not for publication or distribution
‹#› 37©2018 VMware, Inc.
Demo
VMworld 2018 Content: Not for publication or distribution
38©2018 VMware, Inc.
Preparation• Understand the versions you want to run• Get infrastructure & development teams in a room together – understand the solution• Infrastructure readiness: requirements, NSX certificates replacement• Create a (dedicated) management box with CLI tools
Design• Availability Zones: Clusters vs Resource Pools• vSphere Topology – Shared or individual clusters (Management/Edge & Compute or
Management, Edge, Compute)• Networking: planning (reserved CIDR), understand how subnets (/24) will be used per
K8S cluster & namespace, Virtual Switch design (VDS & N-VDS)• Avoid cross data center Kubernetes clusters unless you REALLY know what you are
doing!
Takeaways
VMworld 2018 Content: Not for publication or distribution
39©2018 VMware, Inc.
Deployment• Use FQDN and not IP when configuring the connection to NSX from Ops Manager and
PKS tile• Ensure NTP is configured and time sync between vCenter, ESXi hosts, Ops Manager,
BOSH and PKS• Follow documentation
Operations• Before deploying a new PKS cluster, make sure that enough resources are still
available, otherwise the deployment will fail: Node and Pod IP blocks, IP Pool.• Basic troubleshooting – E.G. Failed cluster deployment
Takeaways
VMworld 2018 Content: Not for publication or distribution
40©2018 VMware, Inc.
ADDITIONNAL SESSIONS• NET1677BE – Kubernetes Container Networking with NSX-T Data Center Deep Dive
– Thursday, Nov 08, 3:00 p.m. – 4:00 p.m.
• CNA2009BE – Run Stateful Apps on Kubernetes with PKS: Highlight WebLogic Server– Thursday, Nov 08, 10:30 a.m. – 11:30 a.m.
• NET1561BE – Next-Generation Reference Design with NSX-T Data Center– Part 1: Thursday, Nov 08, 9:00 a.m. – 10:00 a.m.– Part 2: Thursday, Nov 08, 10:30 a.m. – 11:30 a.m.
HANDS-ON LABS• SPL-1931-01-CNA – VMware Pivotal Container Service and Kubernetes• SPL-1935-01-NET – VMware Pivotal Container Service on VMware NSX-T• SPL-1926-01-NET – VMware NSX-T Data Center – Getting Started
Find out more
VMworld 2018 Content: Not for publication or distribution
PLEASE FILL OUTYOUR SURVEY.Take a survey and enter a drawingfor a VMware company store gift card.
#vmworld #CNA2755BE
VMworld 2018 Content: Not for publication or distribution
THANK YOU!
#vmworld #CNA2755BE
VMworld 2018 Content: Not for publication or distribution