Deploying an OpenShift Cluster with “VMware Cloud Assembly”

#vmworld

CODE3455U

Deploying an OpenShift Cluster with “VMware Cloud Assembly”

Sajal Debnath, VMware, Inc. Rafael Brito, VMware, Inc.

#CODE3455U

VMworld 2019 Content: Not for publication or distribution

©2019 VMware, Inc.

Disclaimer

This presentation may contain product features or functionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.

2

The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no commitment or obligation to deliver any items presented herein. VMworld 2019 Content: Not for publication or distribution

©2019 VMware, Inc.

Agenda

3

Introduction

Overview of the overall Solution

Overview of Red Hat OpenShift (OCP)

Ansible Inventory File

“SSH Equiv” Helper for Ansible

OCP Install and Post-Install

Decisions and assumptions for Implementation

Blueprint & Other Configurations

Extensibility Workflows & Scripts

Yet to Do


©2019 VMware, Inc. 4

• Joined VMware (Office of CTO) in early 2019.

• Before at Citigroup (Containers Global Engineer Lead), NYSE Technologies and Architecture & Engineering.

• Background in OpenShift, GRID, HPC, High Frequency Trading, Linux and TCP/IP.

• Original from Rio de Janeiro, Brazil. Lived 12+ years in NY metro area. In Austin, TX since 2012.

• DadOps, Soccer Fan, Homebrewer and (slow) Runner.

• A professional with 16+ years of experience in Cloud and related technologies

• With VMware for 7+ years. Earlier worked with France Telecom and Hewlett Packard

• Author of the book “Mastering PowerCLI”

• Blog at https://sajaldebnath.com

• Pending patents on Hybrid Cloud Storage

• Reach me at @sajal_debnath

Rafael Brito Sajal Debnath

Introduction


https://sajaldebnath.com/

5©2019 VMware, Inc.

Overview of the Solution



The Problem – Complexity, Inflexibility

• Too many moving parts…

• Too rigid and fixed…

• How can I be Agile

providing Agility to the

deployment and time to

market?



One of the goals of OpenShift running on VMC is bridging engineering teams to a common ground: IaaS + PaaS

Bringing Down the Silos



OpenShift Deployment Layers

Infra Layer / VMware Automation – PART - 2

OpenShift Layer / App Layer – PART -1

CAS Blueprint Extensibility Workflows

Master Nodes Infra Nodes App Nodes HA-Proxy NodesAdmin Node



Deployment

Public Clouds

Deployment Flow

Users

VMware Cloud on AWS

Catalog Items

Private Cloud/Infra

Cloud Assembly

Policy

Tags

Compliance

Authenticate

VMware Cloud Services

OpenShift ClusterK8s Cluster managed by OpenShiftOpenShift UI

PART – 2 – CAS and Automation details

PART – 1 – OpenShift Details

Op

en

Sh

ift Insta

ller +

An

sible



Part - 1 OpenShift Layer



• Red Hat OpenShift Container Platform (“OCP”) is a Platform as a Service (PaaS) based on Kubernetes. Many corporations adopted OpenShift as their enterprise version of k8s.

• Key components of OCP:

• CI/CD (Jenkins, “Source to Image”), Service Catalog, Elastic Search, Prometheus, Docker Registry, Istio, etc.

• Installed and Managed via Ansible

• Most used Release: OCP 3.11 (based on Kubernetes 1.11). Newest Release: OCP 4.1 (based on Kubernetes 1.13). Released in June/2019.

• The scope and code of this session is for OCP 3.11 (applicable to OCP 4.1 only when noted).

• OCP 3.11 is *very* different from OCP 4.1 (based on RHCoreOS, Kubernetes Operator, depends on direct connectivity to Internet). There is no in-place upgrade from OCP 3.X.

Overview of Red Hat OpenShift (OCP)

What is OpenShift ?



OpenShift Components

Master Node Infra Node

• API

• Scheduler & Controller

• ETCD

• Router POD

App Node

• Your Actual App

Admin Node

• Ansible

Load Balancers

• HA-Proxy



Interaction Between the OCP Components

ROUTING LAYER

SERVICE LAYER

A

P

I

T

R

A

F

F

I

C

Secured External API Access (optional)

PassthroughLoad Balancer

Container ApplicationTrafficLoad Balancer

Load Balancerwith trusted CA signed cert and

XFF

Operations,App Developers,CI/CD

Internet

Master Nodes (3)

Infra Nodes (3)

App Nodes (N)

Operations,OCP

Cluster Admins

Other Application

Users

SSH / Ansible

Virtual Infrastructure Layer

Operations,ESXi Admins

external-api.example.com

internal-api.example.com

*.apps.example.com

Admin Node (1)



VM sizes to support up to 5,000-10,000 PODs

OpenShift: Recommended Node Sizes for Master/Infra/Admin/LB

MASTERS

3 VMs

8 vCPUs

32GB RAM

4 Disks

50 GB (root filesystem)

50 GB /var

40 GB /var/lib/etcd

50 GB (docker storage)

INFRA

3 VMs

4 vCPUs

16GB RAM

3 Disks


50 GB /var


ADMIN

2 vCPUs

8GB RAM

1 Disk


1 VM

HAPROXY

3 VMs

4 vCPUs

16GB RAM

3 Disks




App node size (horizontal and vertical) depends on multiple variables (application footprint, cluster size, etc.). For this session, each App node VM has 4 vCPUs and 64GB RAM.

OpenShift: App Node Sizes

APP Nodes

Minimum 3 VMs. Maximum 1,000 VMs.

Minimum 4 vCPUs. Maximum Recommended: Max vCPUs under NUMA Node minus ESXi overhead.

Minimum 64GB RAM. Maximum Recommended: Max Memory under NUMA Node minus ESXi overhead.

3 Disks


100 GB /var


HORIZONTAL SIZE:NUMBER OF VMs

VERTICAL SIZE:SPECS OF VMs



Ansible Inventory Host File is horsepower behind any OCP installation. We generate this file with three inputs:

• OCP parameters user input (generated from Cloud Assembly)

• Fixed VM name convention for master/infra/app/lb/admin nodes

• Jinja2 Template File

On the admin node, an Ansible playbook will generate the ansible host file. Before OCP installation, all OCP nodes (master/infra/app/load balancers) must have SSH root equivalency with the admin node.

Ansible Inventory File



“ocp-parameters.yaml” is a config file generated from user input.

The most critical parameters are “cluster_name” (it will generate DNS entries) “subdomain” and Red Hat subscription credentials (to pull containers and packages for the OCP installation) :

Ansible Inventory File: OCP Cluster Parameters



This OCP cluster example has 13 VMs. $cluster is the input of the name of the OCP cluster:

• 3 x Master Nodes named $cluster-master-0[1..3]

• 3 x Infra Nodes named $cluster-infra-0[1..3]

• 1 x Admin Node named $cluster-admin-0[1..3]

• 3 x App Nodes named $cluster-app-0[1..3]

• 3 x Load Balancer Nodes named $cluster-haproxy-0[1..3]

– $cluster-haproxy-01 is for passthrough API traffic between nodes

– $cluster-haproxy-02 is for External API traffic

– $cluster-haproxy-03 is for Application traffic (Wild Card DNS)

Ansible Inventory File: VM Name Convention



Ansible Inventory File: Jinja2 Template



At Admin Node Boot up:

• Generates root’s SSH key pair

• Runs a non-privileged python web-server serving the public key

• Start “Ansible pinging” all other nodes in a loop

At any other OCP Node Boot up:

• Wget the SSH public key and setup SSH root’s equivalency

“SSH Equiv” Helper



OCP Install and Post-Install

# =~ 10 minutesansible-playbook -i /root/CLUSTER/ansible-hosts /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml && \

# =~ 45 minutesansible-playbook -i /root/CLUSTER/ansible-hosts /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml && \

# post-install: setup load balancer, cluster-admin and other steps

ansible-playbook -i /root/CLUSTER/ansible-hosts /root/$cluster/config.yml




Part - 2 Overview of the Solution - Automation



Decisions and Assumptions



Programmable and Unified provisioning across clouds

VMware Automation: Why?

Catalog & Policy Definitions Continuous Delivery

Policy-Driven Provisioning

Cloud API

Cloud Admin,SRE

Blueprints: Multi-Cloud Templates

Business Users

On Premises: vRealize Automation SaaS: Cloud Automation Services

Developer, DevOps Admin

Application Service

Container Service

Function Service

Third-Party Tools

VCPP / VMC PUBLIC CLOUDSDDC (VCF) EDGE

Hybrid infraVMworld 2019 Content: Not for publication or distribution


AWS Global Infrastructure

VMware Cloud™ on AWSPowered by VMware Cloud Foundation

AWS Global InfrastructureCustomer Data Center

vSphere vSAN NSX

Operational Management

Native AWS Services

Amazon EC2

AmazonS3

AmazonRDS

AWS Direct

Connect

AWS IAMAWS IoT

…

…

…

…

vRealize Suite, vSphere Integrated Containers, ISV Ecosystem

vCentervCenter

Hassle free access to SDDC environment – Anytime Anywhere

VMware Cloud on AWS: Why?

• ESXi on Dedicated Hardware

• Support for VMs and Containers

• vSAN on Flash and EBS Storage

• Replication and DR Orchestration

• NSX Spanning on-premises and Cloud

• Advanced Networking & Security Services

AWS Global Infrastructure



DNS is very important for the deployment and overall working of the solution.

Two options:

• User has authoritative access to DNS server and hence can use our workflows. It does the following:

• At the deployment time, the VM’s gets IP from DHCP. Post IP assignment, VM’s creates the DNS entries for the respective nodes

• CNAME for the load balancer nodes are also created

• While destroying the VM’s, all the DNS entries are removed

• If the user does not have access to DNS, then they need to pre-create all the DNS entries. In this case current blueprint needs to be updated to get the static IP for all the nodes.

DNS Registration



Blueprint & Other Configurations



Blueprint with/without CloudInit



Subscriptions



Set VM Name

vRO Workflow – 1



OpenShift-on-VMware-CAS-Alpha-1-v1.0

vRO Workflow - 2



Un Register CAS VM from DNS-DHCP-RHN

vRO Workflow - 3



Register Un-register to DNS

Major Scripts

registerDns.ps1

registerCname.ps1

unregisterDns.ps1unregisterCname.ps1



Generic Host Preparation



Admin Node Setup Script



SSH Setup in Admin and Other Nodes



OpenShift Installation



Configure Docker Storage



Deployment View



OpenShift View



Grafana



Jenkins



https://labs.vmware.com/flings/enterprise-openshift-as-a-service-on-cloud-automation-services

Check out the Fling




• Support for OpenShift 4.1

• Do the installation through Cloudinit and ABX Actions (platform agnostics)

• Finetune and improve performance of the overall solution

• Put more failure checkpoints – increase stability

• Provide a workflow where end users dynamically can choose number of nodes in cluster

• Provide Day-2 action to add more nodes to the already deployed cluster

• Add an existing cluster as an endpoint in CAS and Code Stream (already in beta)

To-Do



Reach out to [email protected]

Read https://octo.vmware.com/vmware-octo-application-platforms-position-paper/

Let us know your use cases!!

Download the Package @ https://labs.vmware.com/flings/enterprise-openshift-as-a-service-on-cloud-automation-services

Twitter:

• @rafaelbrito

• @sajal_debnath

Next Steps (Call to Action)


mailto:[email protected]

https://octo.vmware.com/vmware-octo-application-platforms-position-paper/




Documents

Deploying an OpenShift Cluster with “VMware Cloud Assembly”