Table of Contents · The Kubernetes pod was deployed using upstream Kubernetes version 1.15.1. What is kubeadm? kubeadm is a tool included in every Kubernetes release that helps users

Table of ContentsLab Overview - HOL-2033-01-CNA - Managing and Extending Kubernetes - GettingStarted.............................................................................................................................. 2

Lab Guidance .......................................................................................................... 3Module 1 - Kubernetes cluster lifecycle management with Kubeadm (30 mins) ...............9

Introduction........................................................................................................... 10Use kubeadm to Expand your Cluster ................................................................... 13Safely Drain a Node for Maintenance.................................................................... 19Use kubeadm to Shrink Your Cluster ..................................................................... 21Conclusion............................................................................................................. 23

Module 2 - Contour - Control and manage ingress for Kubernetes (30 minutes) ............25Introduction........................................................................................................... 26Deploy Contour to Your Cluster ............................................................................. 28Deploy A Test Application...................................................................................... 32Remove Test Application and Contour................................................................... 37Conclusion............................................................................................................. 39

Module 3 - Velero - Back up and migrate Kubernetes resources and persistent volumes(30 minutes) ................................................................................................................... 41

Introduction........................................................................................................... 42Velero Overview .................................................................................................... 46Velero Install and Configure .................................................................................. 50Velero Backup and Restore ................................................................................... 59Velero Disaster Recovery ...................................................................................... 66Velero Cluster Migration........................................................................................ 72Conclusion............................................................................................................. 83

Module 4 - Sonobuoy: Validate Your Kubernetes Deployment - iSIM Module (15minutes).......................................................................................................................... 84

Introduction........................................................................................................... 85Sonobuoy Installation and Conformance Tests...................................................... 86ISim Notes - Do Not Publish................................................................................... 87Conclusion............................................................................................................. 94

HOL-2033-01-CNA

Page 1HOL-2033-01-CNA

Lab Overview -HOL-2033-01-CNA -

Managing and ExtendingKubernetes - Getting

Started

HOL-2033-01-CNA


Lab GuidanceNote: It will take more than 90 minutes to complete this lab. You shouldexpect to only finish 2-3 of the modules during your time. The modules areindependent of each other so you can start at the beginning of any moduleand proceed from there. You can use the Table of Contents to access anymodule of your choosing.

The Table of Contents can be accessed in the upper right-hand corner of theLab Manual.

Discover VMware's OSS Kubeadm solution and three other VMware open-sourceprojects that extend Kubernetes for use in production environments. Explore Kubernetescluster lifecycle management, Contour as an ingress controller for Kubernetes,Sonobuoy to understand the state of a Kubernetes cluster and Velero to backup andrestore your Kubernetes cluster resources and persistent volumes.

Lab Module List:

• Module 1 - Kubernetes Lifecycle Management- (30 minutes) (Basic) Discussthe lab environment setup, and how we can manage our PKS deploymentlifecycle using kubeadm.

• Module 2 - Contour: Configure Ingress Control - (30 minutes) (Advanced)Learn how Contour can help with Kubernetes ingress control management.

• Module 3 - Velero: Backup, Recover, and Migrate Resources - (30minutes)(Advanced) Learn how Velero can help you simplify the tasks of backingup, restoring, migrating, or replicating your Kubernetes deployment

• Module 4 - Sonobuoy: Test Your Kubernetes Deployment - (15 minutes)(iSIM) Test the health of your Kubernetes deployment by deploying Sonobuoy.

Lab Captains:

• Module 1 - Mark McGill, Sr Technical Account Manager, United States• Module 2 - Mark McGill, Sr Technical Account Manager, United States• Module 3 - Tiago Baeta, Staff Systems Engineer, Brazil• Module 4 - Tiago Baeta, Staff Systems Engineer, Brazil

This lab manual can be downloaded from the Hands-on Labs Document site found here:

http://docs.hol.vmware.com

This lab may be available in other languages. To set your language preference and havea localized manual deployed with your lab, you may utilize this document to help guideyou through the process:

http://docs.hol.vmware.com/announcements/nee-default-language.pdf

HOL-2033-01-CNA


http://docs.hol.vmware.com

http://docs.hol.vmware.com/announcements/nee-default-language.pdf

Location of the Main Console

1. The area in the RED box contains the Main Console. The Lab Manual is on the tabto the Right of the Main Console.

2. A particular lab may have additional consoles found on separate tabs in the upperleft. You will be directed to open another specific console if needed.

3. Your lab starts with 90 minutes on the timer. The lab can not be saved. All yourwork must be done during the lab session. But you can click the EXTEND toincrease your time. If you are at a VMware event, you can extend your lab timetwice, for up to 30 minutes. Each click gives you an additional 15 minutes.Outside of VMware events, you can extend your lab time up to 9 hours and 30

minutes. Each click gives you an additional hour.

Alternate Methods of Keyboard Data Entry

During this module, you will input text into the Main Console. Besides directly typing itin, there are two very helpful methods of entering data which make it easier to entercomplex data.

HOL-2033-01-CNA


Click and Drag Lab Manual Content Into Console ActiveWindow

You can also click and drag text and Command Line Interface (CLI) commands directlyfrom the Lab Manual into the active window in the Main Console.

Accessing the Online International Keyboard

You can also use the Online International Keyboard found in the Main Console.

1. Click on the Keyboard Icon found on the Windows Quick Launch Task Bar.

<div class="player-unavailable"><h1 class="message">An error occurred.</h1><div class="submessage"><ahref="http://www.youtube.com/watch?v=xS07n6GzGuo" target="_blank">Try watching this video on www.youtube.com</a>, or enableJavaScript if it is disabled in your browser.</div></div>

HOL-2033-01-CNA


Click once in active console window

In this example, you will use the Online Keyboard to enter the "@" sign used in emailaddresses. The "@" sign is Shift-2 on US keyboard layouts.

1. Click once in the active console window.2. Click on the Shift key.

Click on the @ key

1. Click on the "@ key".

Notice the @ sign entered in the active console window.

HOL-2033-01-CNA


Activation Prompt or Watermark

When you first start your lab, you may notice a watermark on the desktop indicatingthat Windows is not activated.

One of the major benefits of virtualization is that virtual machines can be moved andrun on any platform. The Hands-on Labs utilizes this benefit and we are able to run thelabs out of multiple datacenters. However, these datacenters may not have identicalprocessors, which triggers a Microsoft activation check through the Internet.

Rest assured, VMware and the Hands-on Labs are in full compliance with Microsoftlicensing requirements. The lab that you are using is a self-contained pod and does nothave full access to the Internet, which is required for Windows to verify the activation.Without full access to the Internet, this automated process fails and you see this

watermark.

This cosmetic issue has no effect on your lab.

Look at the lower right portion of the screen

HOL-2033-01-CNA


Please check to see that your lab is finished all the startup routines and is ready for youto start. If you see anything other than "Ready", please wait a few minutes. If after 5minutes your lab has not changed to "Ready", please ask for assistance.

HOL-2033-01-CNA


Module 1 - Kubernetescluster lifecycle

management withKubeadm (30 mins)

HOL-2033-01-CNA


Introduction• In Module 1, we will discuss the lab environment setup, and how we can manage

our PKS deployment lifecycle using kubeadm. You will use kubeadm to test thehealth of the environment, and deploy an additional worker node to the pod.

Lab Environment

HOL-2033-01-CNA


Our lab consists of 2 Kubernetes clusters, each with one master node and two workernodes, although you will join k8s-worker-02a to Cluster A in this module. All nodes arerunning CentOS 7.6. Calico is used in the Pod to provide networking. Harbor is used asa local image registry we will use to deploy applications.

The Kubernetes pod was deployed using upstream Kubernetes version 1.15.1.

What is kubeadm?

kubeadm is a tool included in every Kubernetes release that helps users set up a best-practice Kubernetes cluster. kubeadm provides support along the entire lifecycle of aKubernetes cluster including creation, upgrade, and teardown.

Due to the variety of ways in which machines can be provisioned, the scope of kubeadmis intentionally limited to bootstrapping rather than provisioning — it is intended to be abuilding block, and higher level tools take advantage of kubeadm. kubeadm allowsthese higher level tools to ensure clusters are conformant and look as much alike aspossible. Setting up add-ons such as CNI, the Kubernetes dashboard, and monitoringtools is outside the scope of kubeadm.

Why Does VMware Recommend kubeadm?

One of the reasons VMware recommends bootstrapping your Kubernetes clusterwith kubeadm is because it provides production-grade defaults for many flags of thecontrol plane components. In addition to these defaults, VMware recommends someminor changes to the kubeadm configuration to properly setup your cluster’s controlplane. As modifications are made, VMware recommends that you track any changes toKubernetes configuration in source control.

HOL-2033-01-CNA


kubeadm is the primary deployment tool VMware recommends for bootstrapping aKubernetes cluster, based on some key advantages:

• Ease of use. kubeadm supports creation, upgrade, and teardown, and is arelatively easy tool for new users to adopt.

• Use on any infrastructure. Organizations need to deploy Kubernetes across awide variety of infrastructure — including bare metal, vSphere, VMC on AWS,Native AWS, Azure, GCE, and more. It’s important that a cluster created on oneplatform looks like another created somewhere else. We even createdSonobuoy to test conformance so you can be sure that clusters were set upcorrectly and will behave as expected on any infrastructure.

• Extendable. Customers have unique enterprise requirements. kubeadmprovides a phases command that allows you to execute steps individually, so youcan customize actions as needed.

• Production ready. kubeadm rolls out secure Kubernetes clusters — adoptingbest practices such as enforcing RBAC, using secure communication between thecontrol plane components and between the API server and kubelets, locking downthe kubelet API, and more.

• Community contributions. kubeadm has become one of the most commonways to deploy Kubernetes, and as a result the community has rallied to hardenkubeadm and make inroads on every release.

So where does kubeadm fit into a complete deployment solution for Kubernetescustomers? kubeadm is not a one-click-install solution. As stated above, kubeadm isintended to be a building block and part of a larger solution. VMware is investingsignificantly in this area to bring a declarative, API-driven model to cluster creation andoperations, where clusters are treated as immutable (i.e. upgrades equate to a newdeployment versus an in-place upgrade). VMware plans to leverage and contribute tothe upstream work on the Cluster API to make this real. More will be shared in thefuture.

HOL-2033-01-CNA


https://blogs.vmware.com/cloudnative/2019/02/21/certifying-kubernetes-with-sonobuoy/

https://github.com/kubernetes-sigs/cluster-api/

Use kubeadm to Expand your ClusterYou will use kubeadm to add an additional worker node to your cluster

Accessing the VM-CLI machine

In our lab we have a Linux server configured with all the needed tools to get through thelab lessons. We access this box via PuTTY which already has a saved session for thismachine.

1. Click on the puTTY icon in the Windows toolbar;2. Scroll all the way down and select the "vm-cli.corp.local" machine;3. Click Open in order to open the VM-CLI console.

The user configured for auto-login in the puTTY VM-CLI session is holuser. We are goingto use this user for the whole module while connected to VM-CLI machine.

HOL-2033-01-CNA


Check the Status of the Cluster

Use kubectl to Check the Status of the Cluster

1. At the command line, type kubectl get nodes2. Note the 1 node with the role of master, and 1 node that has a role of <none>,

which is a worker node

HOL-2033-01-CNA


Join an Additional Worker Node to the Cluster

Open a New SSH Session to a Master Node

1. In the Putty window, click on the icon in the upper left hand corner2. Click on Saved Sessions3. Click on k8s-master-01a.corp.local

Get Join Token

HOL-2033-01-CNA


In this step, we will generate a bootstrap token that will be used to establish abidirectional trust between a node joining the cluster and our control-plane node.

1. At the command line for k8s-master-01a, type kubeadm token create--print-join-command and press <Enter>

2. Note the output. Highlight the output with the mouse to copy it to the clipboard.The output should be in this format: 'kubeadm join 192.168.120.11:6443 --token cil09b.qiu6je65v6e15orl --discovery-token-ca-cert-hashsha256:9c97f9dd6fad76265d82d049fabc5d56987596aff601bc7814bc97b1ab7838ba',but your token will differ.

We will paste that output into the command line of our new worker node to join it to thecluster.

HOL-2033-01-CNA


SSH to the New Worker Node

1. In the Putty window, click on the icon in the upper left hand corner2. Click on Saved Sessions3. Click on k8s-worker-02a.corp.local

HOL-2033-01-CNA


Join a New Worker Node to the Cluster

The VM "k8s-worker-02a" has already had Kubernetes installed and configured, so wewill use the command and token we generated in the previous step to join it to ourcluster.

1. At the command line, right-click to paste the join command and token, andpress <Enter>

2. Wait for the command to complete. You should see "This node has joined thecluster:" in the output

Let's return to the the vm-cli session to check the status of the cluster

Check the Status of the Cluster

1. On the taskbar, click on your vm-cli PuTTY session2. Type kubectl get nodes and press <Enter>3. Note that k8s-worker-02a is now a worker node (ROLES of <none>). It should

have a STATUS of Ready

Repeat Step 2 until k8s-worker-02a has a STATUS of Ready before proceeding.

HOL-2033-01-CNA


Safely Drain a Node for MaintenanceYou will use the "cordon" and "drain" commands to keep new pods from being scheduledon a node, then safely remove any running pods.

Cordon and Drain One of Your Worker Nodes

In the normal management of your Kubernetes Clusters, you would want to prepare anode for maintenance to ensure that any running pods are safely removed from yournode. Note that we don't currently have any pods running in our cluster, but thecommands will be the same.

Cordon Your Worker Node

Cordoning a node marks it as unschedulable and prevents new pods from beingscheduled to that node, but does not affect any existing pods on the node. This is usefulas a preparatory step before a node reboot, etc.

1. In the your VM-CLI PuTTY window, type kubectl get nodes and press <Enter>.Note the current status of your Master and Worker Nodes

2. Type kubectl cordon k8s-worker-02a.corp.local and press <Enter>. Note theresponse that the node has been cordoned

3. Type kubectl get nodes and press <Enter>. Note the current status of our workernode. Although it is still Ready, Scheduling any new Pods has been disabled.

HOL-2033-01-CNA


Drain Your Worker Node

You can use kubectl drain to safely evict all of your pods from a node before you performmaintenance on the node (e.g. kernel upgrade, hardware maintenance, etc.). Safeevictions allow the pod’s containers to gracefully terminate.

1. In the your VM-CLI PuTTY window, type kubectl drain k8s-worker-02a.corp.local--ignore-daemonsets and press <Enter>

2. Note that the output shows that the node is already cordoned, and the reports itas drained.

It is now safe to perform maintenance on the node. In the next steps, we will removethe node from the cluster.

HOL-2033-01-CNA


Use kubeadm to Shrink Your ClusterIn this step, you will remove the worker node you added to the cluster.

Remove New Worker Node from the Cluster

1. In your VM-CLI PuTTY session type kubectl delete node k8s-worker-02a.corp.local andpress <Enter>

2. You should confirmation that the node was deleted.

Use kubectl to Check the Status of the Cluster

1. At the command line, type kubectl get nodes and press <Enter>2. You should only see the original Master and Worker nodes

HOL-2033-01-CNA


Reset Changes on Worker Node

We will use kubeadm on the worker node to reset any changes to k8s-worker-02a madeduring the join command

1. Click on the session for root@k8s-worker-02... on the taskbar.2. Type kubeadm reset -f and press <Enter>3. Note the output while the worker node is reset.

You have successfully removed the worker node k8s-worker-02a from Cluster A.

HOL-2033-01-CNA


ConclusionThis concludes "Module 1 - Kubernetes Lifecycle Management with Kubeadm"

You should now have an understanding of why and how to use kubeadm to help manageyour Kubernetes deployment.

You've finished Module 1

Congratulations on completing Module 1!

If you are looking for additional information on kubeadm, try one of these:

• Overview of kubeadm• Moving the needle on kubeadm• Or use your smart device to scan the QRC Code above.

Proceed to any module below which interests you most regarding Managing andExtending Kubernetes.



• Module 4 - Sonobuoy: Validate Your Kubernetes Deployment - (15minutes) (iSIM) Test the health of your Kubernetes deployment by deployingSonobuoy.

HOL-2033-01-CNA


https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

https://blog.heptio.com/moving-the-needle-on-kubeadm-in-kubernetes-1-11-90a87ddb9d0f

How to End Lab

To end your lab click on the END button.

HOL-2033-01-CNA


Module 2 - Contour -Control and manage

ingress for Kubernetes(30 minutes)

HOL-2033-01-CNA


IntroductionThis Module contains the following lessons:

• Introduction to Contour• Deploy Contour in your lab• Deploy and test a sample application

Introduction to Contour

One of the most critical needs in running workloads at scale with Kubernetes is efficientand smooth traffic ingress management at the Layer 7 level. Getting an application upand running is not always the entire story; it may still need a way for users to access it.Filling that operational gap is what Contour was designed to do by providing a way toallow users to access applications within a Kubernetes cluster. Contour is an Ingresscontroller for Kubernetes that works by deploying the Envoy proxy as a reverse proxyand load balancer. Contour supports dynamic configuration updates out of the box whilemaintaining a lightweight profile.

Contour offers the following benefits for users: * A simple installation mechanism toquickly deploy and integrate Envoy * Safely support ingress in multi-team Kubernetesclusters * Clean integration with the Kubernetes object model * Dynamic updates toingress configuration without dropped connections

What is Ingress?

Kubernetes Ingress is a set of configurations that define how external traffic can berouted to an application inside a Kubernetes cluster. A controller (Contour) watches forchanges to objects in the cluster, then wires together the configurations to create a datapath for the request to be resolved, implementing the configurations defined. It makesdecisions based on the request received (e.g., example.com/blog), provides TLStermination, and performs other functions.

Ingress is an important component of a cloud native system because it allows for aclean separation between the application and how it’s accessed. A cluster administratordeals with providing access to the controller, and the application engineer just dealswith deploying the application. Ingress is the glue that ties the two together.

HOL-2033-01-CNA


https://en.wikipedia.org/wiki/OSI_model#Layer_7:_Application_Layer

https://kubernetes.io/docs/concepts/services-networking/ingress/

Contour in Detail

Since it was added in Kubernetes 1.1, Ingress hasn't gotten much attention but is stillvery popular in the community. Many controllers rely on annotations on the Ingressobject to clarify, restrict, or augment the structure imposed by the Ingress object, whichis no different from how Contour supports Ingress.

At the same time a number of web application deployment patterns, such as blue/greendeployments, explicit load balancing strategies, and presenting more than oneKubernetes Service behind a single route, are difficult to achieve with Ingress as itstands today. Contour has introduced a new Custom Resource Definition (CRD) thatallows for a new data model called IngressRoute and enhances what Ingress can dotoday by enabling new features not previously possible.

IngressRoute is designed to provide a sensible home for configuration parameters aswell as to share an ingress controller across multiple namespaces and teams in thesame Kubernetes cluster. We do this by using a process we call delegation. Thisdelegation concept patterns off of the way a subdomain is delegated from one domainname server to another, and allows for teams to define and self-manage IngressRouteresources safely.

HOL-2033-01-CNA


Deploy Contour to Your ClusterIn this step, you will deploy Contour from a local registry.

Connecting to the VM-CLI machine

In our lab we have a Linux server configured with all the needed tools to get through thelab lessons. We access this box via PuTTY which already has a saved session for thismachine.

1. Click on the "PuTTY" icon in the Windows toolbar;2. Scroll all the way down and select the "vm-cli.corp.local" machine;3. Click Open in order to open the VM-CLI console.

The user configured for auto-login in the PuTTY VM-CLI session is holuser. We are goingto use this user for the whole module while connected to VM-CLI machine.

HOL-2033-01-CNA


Deploy Contour From a Local Registry

Review Current Deployments and ApplicationConfiguration File

Since our lab does not have internet access, we are using a local registry residing inHarbor on harbor-01a. This is where we will retrieve the images needed for Contour.

1. At the command line, type kubectl get deployments and press <Enter>2. Type kubectl get pods and press <Enter>3. Note that there are currently no deployments or pods in the cluster4. Type cat contour-files/contour-deployment-rbac.yaml press <Enter>5. Review the configuration. Note the new Namespace (heptio-contour) as we will

use this in our commands to view the deployment

HOL-2033-01-CNA


Deploy Contour Using yaml File

1. Type kubectl apply -f contour-files/contour-deployment-rbac.yaml and press <Enter>

HOL-2033-01-CNA


Note: If you receive an error "Unexpected error when reading response body: ..." afterstep 1, press <Ctrl> + <C> and re-run the command.

2. Once the deployment completes, type kubectl get deployments -n heptio-contour andpress <Enter>

3. Type kubectl get pods -n heptio-contour and click <Enter>4. Note the status of the replicas5. Type kubectl get events -n heptio-contour --sort-by='{.lastTimestamp}' and press

<Enter>6. Review the events

Wait until both replicas are showing READY of 2/2 and STATUS Running (step 4) beforemoving on to the next step. Re-run the command from step 3 to refresh the status ifnecessary

HOL-2033-01-CNA


Deploy A Test ApplicationIn this step, you will deploy a test application to show Contour working for ingresscontrol

HOL-2033-01-CNA


Deploy kuard Test Application

HOL-2033-01-CNA


HOL-2033-01-CNA


kuard is a demo application for Kubernetes, and stands for "Kubernetes up and Running"

1. At the command line for VM-CLI, type cat contour-files/contour-kuard-example.yamland press <Enter>

2. Review the yaml configuration file for the deployment3. Type kubectl apply -f contour-files/contour-kuard-example.yaml and press <Enter>

Note: If you receive an error "Unexpected error when reading response body: ..." afterstep 1, press <Ctrl> + <C> and re-run the command.

Check Status of kuard

1. Type kubectl get pods and press <Enter>2. Wait for all replicas to have the STATUS of "Running" before proceeding3. Type kubectl get -n heptio-contour service contour -o wide and press <Enter>4. Note the ingress ports (number after the ":" after 80 and 443). We will use the

port number that translates to port 80 to connect to the kuard application. ie, inthis example it is 31810. Your number will be different.

5. Note the TYPE is NodePort, and is not load balanced, so we will connect directly toone of the worker nodes where the application is running

HOL-2033-01-CNA


Test kuard

1. Click on the Chrome icon to open a browser2. In the address bar, type http://k8s-worker-01a.corp.local:<your_port_number>,

where <your_port_number> will be from step 4 previously, and press <Enter>3. Feel free to explore the "Kubernetes Up and Running" application by clicking

on the tabs on the left hand side

HOL-2033-01-CNA


Remove Test Application and ContourIn this step, you will remove Kuard and Contour from the cluster

Remove kuard

1. Return to the vm-cli PuTTY session by clicking on the "holuser@vm-cli:" icon onthe taskbar.

2. At the command line, type kubectl delete -f contour-files/contour-kuard-example.yamland press <Enter>

3. Type kubectl get pods and press <Enter>4. Note that you may see a pod terminating. Re-run the command from step 3 until

you see "No resources found."

Remove Contour

1. Type kubectl delete -f contour-files/contour-deployment-rbac.yaml and press <Enter>2. Type kubectl get pods -n contour-heptio and press <Enter>

HOL-2033-01-CNA


3. You should see "No resources found."

HOL-2033-01-CNA


ConclusionThis concludes "Module 2 - Contour - Control and manage ingress for Kubernetes"

You should now have an understanding how to deploy Contour and use it in anapplication you deploy.


Congratulations on completing Module 2.

If you are looking for additional information on Contour and ingress control, try one ofthese:

• Contour GitHub Page• Routing Traffic to Applications in Kubernetes with Contour• Or use your smart device to scan the QRC Code.

Proceed to any module below which interests you most regarding Managing andExtending Kubernetes.

• Module 1 - Kubernetes Lifecycle Management - (30 minutes) (Basic) Discussthe lab environment setup, and how we can manage our Kubernetes deploymentlifecycle using kubeadm.



HOL-2033-01-CNA


https://github.com/heptio/contour

https://blogs.vmware.com/cloudnative/2019/03/08/routing-traffic-kubernetes-contour-0-10/

How to End Lab


HOL-2033-01-CNA


Module 3 - Velero - Backup and migrate

Kubernetes resources andpersistent volumes (30

minutes)

HOL-2033-01-CNA


Introduction

This Module contains the following lessons:

• Velero Overview• Velero Install and Configure• Velero Backup and Restore• Velero Disaster Recovery• Velero Cluster Migration

Why Velero?

Velero was designed specifically to back up and restore Kubernetes cluster resources.This gives Velero some advantages over etcd backups.

Velero uses the Kubernetes API discovery capabilities to collect backup data. This meansthat Velero can back up new APIs without updating Velero itself. Velero does not need totake backups of etcd.

A discovery approach allows Velero to back up clusters that include aggregated APIservers, which otherwise requires creating an etcd backup of each server. Velero canalso perform backups in scenarios where there is no direct access to etcd, such as acluster running on GKE.

Velero lets you select specific resources to back up, because it does not create anatomic snapshot by backing up etcd. This approach also means that Velero lets yourestore a subset of a backup.

Velero also associates snapshots of persistent volumes with each backup. Thesesnapshots allow Velero to restore both what was running in the cluster and the dataassociated with the cluster.

Backup

Velero supports on-demand backups as well as scheduled periodic backups. You canconfigure Velero to backup your entire cluster or only specified resources based onnamespaces, resources, and labels you include or exclude.

HOL-2033-01-CNA


Every cluster has different backup needs. You should carefully consider your own needswhen creating a backup plan. Consider the following scenarios as you plan for yourenvironment.

Full Cluster

A full cluster backup is the default behavior if you run velero create backup without anyadditional flags. This approach is the simplest way to get started with Velero, but mightnot be adequate for all purposes.

Per-Namespace

A per-namespace backup lets you restore a namespace with just the base velero createrestore command.

This approach is useful if you have multi-tenant clusters where each tenant has its ownnamespace in the cluster. You can restore a single namespace without disrupting theother tenants.

Depending on your needs, you might also want full cluster backups together with yournamespace backups.

Strategy

We recommend that you identify critical functions in your system and create backupsthat include only the resources needed to restore these critical functions. This makesrestoring simpler because you restore the entire backup instead of having to filter it.This is especially important to help reduce errors in case of disaster recovery.

Scheduling Backups

Velero supports scheduled backups with a cron syntax. You can also set a time to live(TTL). These two features allow you to create backup configurations for differentrecovery point objectives (RPO).

A common configuration is the hourly backup. Here is an example of an hourly backupconfiguration that includes hourly, daily, weekly, and monthly backup schedules all withTTLs that remove backups once they are part of the larger chronological archive.

Backup Hooks

Hooks in Velero let you run a command inside a container before and after a backup.Velero provides both pre-backup and post-backup hooks. Hooks are configured usingpod annotations. The Velero documentation provides an example of hooks to call

HOL-2033-01-CNA


https://github.com/heptio/velero/blob/master/examples/nginx-app/with-pv.yaml

fsfreeze on a file system before and after Velero performs a backup, plus the full detailsof hook annotations.

Restore

Velero supports restoring backups that are created manually or according to a schedule.It lets you perform full or partial restores of backups. Velero performs a full restore of abackup by default.

Lifecycle of Velero objects

Understanding the lifecycle of an Velero object can help you understand the details ofyour Velero jobs.

1. The New phase shows that the requested backup, restore, or schedule object iscreated by the API, but the object has not been processed by its respectivecontroller.

2. The next phase is validation. The Newobject is validated by the processingcontroller. If the controller cannot validate the object, it moves to theFailedValidation phase and no further processing is attempted.

3. After successful validation, if you are creating a schedule, the object moves to theEnabled phase. When a schedule is Enabled it triggers backups in accordancewith its schedule spec.

4. If you are creating a backup or restore, the object moves to the InProgressphase. During the InProgress phase, Velero attempts to perform all theoperations codified in the backup or restore object. Relevant errors and warningsare counted during this process and captured in the status. The type of objectbeing created determines the next phase.

5. Both backup and restore objects have a Completed phase. This phase showsthat the requested backup or restore object and all its operations have beenperformed. The Completed phase does not automatically mean that there wereno errors or warnings.

Two other phases are possible for backups:

• If you delete a backup, it has a Deleting phase.• A Failed phase is also possible for backups. The Failed phase shows that there

was a critical error that prevented the backup from completing successfully.

Monitoring

Velero provides metrics for backups and restores. You can use these metrics to addrecording and alerting rules to your Prometheus configuration.

HOL-2033-01-CNA


https://github.com/heptio/velero/blob/master/examples/nginx-app/with-pv.yaml

https://heptio.github.io/velero/v0.11.0/hooks

https://heptio.github.io/velero/v0.11.0/hooks

Migrations

Moving resources from one cluster to another is a common scenario. Velero can helpwith migration. The Velero documentation provides further details on how to use Veleroto perform a cluster migration.

Disaster Recovery

VMware recommends that you create a periodic backup schedule as part of yourdisaster recovery plan. Velero can then help with the recovery of your cluster and itsresources in case of a disaster.

During a recovery, Velero can be configured to restoreOnly mode. restoreOnly modeensures that Velero does not take any backups during the restore process so that you donot have worry about cleaning up backups that might contain only a partially restoredcluster.

The Velero documentation provides more details on how to use Velero to recover clusterresources.

Testing Your Backups

Testing your backups is just as important as creating a periodic backup schedule. Testinggives you the opportunity to verify that the backups restore all of the expected state.Testing and verifying your backups is a critical part of any disaster recovery plan.

Security

By default, Velero runs using a service account with cluster-admin permissions, and isnot scoped by any role-based access controls (RBAC). These permissions allow Velero toback up and restore all Kubernetes resources. However, this means users who havepermissions to back up and restore with Velero effectively have cluster-adminpermissions.

This means that only trusted administrators should have access to create backups andrestores.

You can configure Velero to run with reduced permissions, but this means that only theresources the related service account can access can be backed up and restored.

Velero does not currently support multi-tenancy in a single instance. A scenario withmultiple, tightly scoped instances is untested.

HOL-2033-01-CNA


https://heptio.github.io/velero/v0.11.0/migration-case

https://heptio.github.io/velero/v0.11.0/migration-case

https://heptio.github.io/velero/v0.11.0/disaster-case

https://heptio.github.io/velero/v0.11.0/disaster-case

Velero OverviewVelero gives you tools to back up and restore your Kubernetes cluster resources andpersistent volumes. Velero lets you:

• Take backups of your cluster and restore in case of loss.• Migrate cluster resources to other clusters.• Replicate your production cluster to development and testing clusters.

Velero consists of:

• A server that runs on your cluster• A command-line client that runs locally

You can run Velero in clusters on a cloud provider or on-premise. For detailedinformation, see Compatible Storage Providers.

Because our lab environment has no Internet access, we are going to install and useVelero in an on-premise fashion.

How Velero Works

Each Velero operation - on-demand backup, scheduled backup, restore - is acustom resource, defined with a Kubernetes Custom Resource Definition (CRD) andstored in etcd. Velero also includes controllers that process the custom resources toperform backups, restores, and all related operations.

You can back up or restore all objects in your cluster, or you can filter objects by type,namespace and/or label.

Velero is ideal for the disaster recovery use case, as well as for snapshotting yourapplication state, prior to performing system operations on your cluster (e.g. upgrades).

On-demand Backups

The backup operation:

1. Uploads a tarball of copied Kubernetes objects into cloud object storage.2. Calls the cloud provider API to make disk snapshots of persistent volumes, if

specified.

You can optionally specify hooks to be executed during the backup. For example, youmight need to tell a database to flush its in-memory buffers to disk before taking asnapshot. More about hooks.

HOL-2033-01-CNA


https://velero.io/docs/v1.0.0/support-matrix/

https://velero.io/docs/v1.0.0/hooks/

Note that cluster backups are not strictly atomic. If Kubernetes objects are being createdor edited at the time of backup, they might not be included in the backup. The odds ofcapturing inconsistent information are low, but it is possible.

Scheduled backups

The schedule operation allows you to back up your data at recurring intervals. The firstbackup is performed when the schedule is first created, and subsequent backupshappen at the schedule's specified interval. These intervals are specified by a Cronexpression.

Scheduled backups are saved with the name <SCHEDULE NAME>-<TIMESTAMP>,where <TIMESTAMP> is formatted as YYYYMMDDhhmmss.

Restores

The restore operation allows you to restore all of the objects and persistent volumesfrom a previously created backup. You can also restore only a filtered subset of objectsand persistent volumes. Velero supports multiple namespace remapping--for example,in a single restore, objects in namespace "abc" can be recreated under namespace"def", and the objects in namespace "123" under "456".

The default name of a restore is <BACKUP NAME>-<TIMESTAMP>, where<TIMESTAMP> is formatted as YYYYMMDDhhmmss. You can also specify a customname. A restored object also includes a label with key velero.io/restore-name andvalue <RESTORE NAME>.

You can also run the Velero server in restore-only mode, which disables backup,schedule, and garbage collection functionality during disaster recovery.

Backup workflow

When you run velero backup create test-backup :

HOL-2033-01-CNA


1. The Velero client makes a call to the Kubernetes API server to create a Backupobject.

2. The BackupController notices the new Backup object and performs validation.3. The BackupController begins the backup process. It collects the data to back up

by querying the API server for resources.4. The BackupController makes a call to the object storage service - for example,

AWS S3 - to upload the backup file.

By default, velero backup create makes disk snapshots of any persistent volumes. You canadjust the snapshots by specifying additional flags. Run velero backup create --help to seeavailable flags. Snapshots can be disabled with the option --snapshot-volumes=false .

Backed-up API versions

Velero backs up resources using the Kubernetes API server's preferred version for eachgroup/resource. When restoring a resource, this same API group/version must exist inthe target cluster in order for the restore to be successful.

For example, if the cluster being backed up has a gizmos resource in the things APIgroup, with group/versions things/v1alpha1, things/v1beta1, and things/v1, and theserver's preferred group/version is things/v1, then all gizmos will be backed up fromthe things/v1 API endpoint. When backups from this cluster are restored, the targetcluster must have the things/v1 endpoint in order for gizmos to be restored. Note thatthings/v1 does not need to be the preferred version in the target cluster; it just needsto exist.

Set a backup to expire

When you create a backup, you can specify a TTL by adding the flag --ttl <DURATION> . IfVelero sees that an existing backup resource is expired, it removes:

• The backup resource• The backup file from cloud object storage• All Persistent Volume snapshots• All associated Restores

Object storage sync

Velero treats object storage as the source of truth. It continuously checks to see that thecorrect backup resources are always present. If there is a properly formatted backup filein the storage bucket, but no corresponding backup resource in the Kubernetes API,Velero synchronizes the information from object storage to Kubernetes.

This allows restore functionality to work in a cluster migration scenario, where theoriginal backup objects do not exist in the new cluster.

HOL-2033-01-CNA


Likewise, if a backup object exists in Kubernetes but not in object storage, it will bedeleted from Kubernetes since the backup tarball no longer exists.

HOL-2033-01-CNA


Velero Install and ConfigurePrerequisites

• Access to a Kubernetes cluster, version 1.7 or later. We already have aKubernetes cluster on version 1.15.0.

• A DNS server on the cluster. Already configured with Kuberentes.• kubectl installed. It is already installed on the VM-CLI virtual machine.• A Compatible Storage Provider configured. In our environment we are going

to use MinIO, an object storage server compatible with Amazon S3 cloud storageservice.

About MinIO

MinIO is a cloud-native application designed to scale in a sustainable manner in multi-tenant environments. It is an object storage server released under Apache License v2.0.Orchestration platforms provide perfect launchpad for MinIO to scale. MinIO is supportedon multiple orchestration platforms:

• Docker Swarm• Docker Compose• Kubernetes• DC/OS

Why is MinIO cloud-native?

The term cloud-native revolves around the idea of applications deployed as microservices, that scale well. It is not about just retrofitting monolithic applications ontomodern container based compute environment. A cloud-native application is portableand resilient by design, and can scale horizontally by simply replicating. Modernorchestration platforms like Swarm, Kubernetes and DC/OS make replicating andmanaging containers in huge clusters easier than ever.

While containers provide an isolated application execution environment, orchestrationplatforms allow seamless scaling by helping replicate and manage containers. MinIOextends this by adding an isolated storage environment for each tenant.

MinIO is built ground up on the cloud-native premise. With features like erasure-coding,distributed and shared setup, it focuses only on storage and does it very well. While, itcan be scaled by just replicating MinIO instances per tenant via an orchestrationplatform.

If you wish to know more about MinIO, take a look at the documentation.

We are going to use MinIO in our lab. Since we do not have access to the Internet, wecan't access a Kubernetes cluster running on Amazon AWS. We are going to use MinIO

HOL-2033-01-CNA


https://docs.min.io/

in an on-premises fashion, which means we are going to use MinIO with our localKubernetes cluster leveraging its features to provide our cluster with S3 compatiblestorage for our Velero operations.

We are going to run MinIO in a docker container using vm-cli as our docker host

In our lab we are going to explore basic Velero functionality. Configuring MinIO forproduction is out of scope.


In our lab we have a Linux server configured with all the needed tools to get through thelab lessons. We access this box via PuTTY which already has a saved session for thismachine

HOL-2033-01-CNA


1. Click on the "puTTY" icon in the Windows toolbar;2. Scroll all the way down and select the vm-cli.corp.local machine;3. Click <Open> in order to open the vm-cli console.

The user configured for auto-login in the puTTY VM-CLI session is holuser. We are goingto use this user for the whole module while connected to VM-CLI machine.

Installing Velero

In our lab we are going to set up the Velero server and client, then back up and restorea sample application.

For simplicity, we are going to use MinIO, an S3-compatible storage service that runslocally on your cluster. For additional functionality with this setup, see the docs on howto expose Minio outside your cluster.

Velero is an OpenSource project backed by VMware that is publicly available from ourgit repository at https://github.com/heptio/velero/releases.

Since it is just a binary file, the velero command line client, the only thing we need todo is download the archive for the right platform and extract the binary to a place thatmake sense. In our lab, the archive was already downloaded so we only need to extractit to a reasonable place like the local binary directory.

HOL-2033-01-CNA


https://velero.io/docs/v1.0.0/get-started/#expose-minio-outside-your-cluster

https://github.com/heptio/velero/releases

Extracting Velero binary

1. Type pwd and hit <ENTER> to make sure you are in the right directory. It shoulddisplay "/home/holuser";

2. Type ls -l and hit <ENTER>. You should see a directory called "/velero-files";3. Type cd velero-files and hit <ENTER>;4. Type ls -l and hit <ENTER>. Find the Velero archive and its version - "velero-

v1.0.0-linux-amd64.tar.gz";5. Type sudo tar -zxvf velero-v1.0.0-linux-amd64.tar.gz -C /usr/local/bin/

--strip-components=1 velero-v1.0.0-linux-amd64/velero and hit <ENTER> to extract theVelero binary to the local binary directory (the command must be typed in asingle line);

6. Type which velero and hit <ENTER> to check if the binary was extract to theright location. It should display "/usr/local/bin/velero".

HOL-2033-01-CNA


Checking our Kubernetes Cluster

Before we start running Velero, let's take a look at our Kubernetes cluster, its nodes andconfiguration.

1. Type kubectl get nodes and hit <ENTER> to check the nodes and their roles inthe kubernetes cluster. You should see 2 nodes, 1 master/etcd and 1 worker;

2. Type kubectl get namespaces and hit <ENTER> to check the existing namespacesin our cluster. Notice that we have only the default ones: "default", "kube-node-lease", "kube-public", "kube-system".

Setting up Velero Server

Now that we have the Velero command line client configured let's set up the Veleroserver.

HOL-2033-01-CNA


Running the MinIO Storage Service container

In order to have a S3-like storage service we are going to run MinIO in a container usingour VM-CLI machine as a docker host. We are going to expose the MinIO service on port9000 and use a local directory to map to the MinIO data directory. We are also going tocreate the MinIO server credentials in order to pass to Velero server later on. Theimage for the MinIO container is stored in our local Harbor registry.

1. Type sudo mkdir /mnt/minio-data en press <ENTER> in order to create the localdirectory to be mapped to the MinIO data directory;

2. Type sudo docker run --name minio -d -p 9000:9000 -e "MINIO_ACCESS_KEY=minio" -e"MINIO_SECRET_KEY=minio123" -v /mnt/minio-data/:/data harbor-01a.corp.local:80/minio/minioserver /data and hit <ENTER> to run the MinIO server in a container;

3. Type sudo docker ps to make sure the MinIO container is running.

HOL-2033-01-CNA


Accessing the MinIO UI and Creating the S3 bucket

1. Click on the <Chrome> icon in the Windows toolbar to open the Chromebrowser;

2. Type vm-cli.corp.local:9000 in the Chrome address bar and hit <ENTER>. Youshould get the MinIO UI login page;

3. Type the MinIO credentials (defined in the arguments passed to the container)where the Access Key is "minio" and the Secret Key is "minio123" and hit<ENTER>;

4. Click on the PLUS sign in the bottom left and then in the yellow icon to create abucket (not shown). Type velero as the name of the bucket and hit <ENTER>;

5. Now you can see we have one S3 compatible bucket called "velero". If you lookat the right side you will see that this bucket is empty. When we create ourbackups they will be stored on this bucket.

HOL-2033-01-CNA


Starting Velero Server

Since we don't have Internet access we need to specify the container images needed byVelero in the velero install command. Our images are stored on a local Harbor registryrunning on harbor-01a.corp.local server.

1. Type pwd and hit <ENTER> to make sure we are in the "/velero-files" directory;2. Type cat credentials-velero to check the MinIO credentials to be passed to Velero

install;3. Type velero install --provider aws --bucket velero --secret-file ./credentials-velero

--use-volume-snapshots=false --backup-location-configregion=minio,s3ForcePathStyle="true",s3Url=http://vm-cli.corp.local:9000 --imageharbor-01a.corp.local:80/heptio-images/velero:latest in a single line and hit<ENTER> to start the Velero server. Notice the message "Velero isInstalled!";

4. Type kubectl get ns to check the new namespace created called velero;5. Type kubectl get deployments -n velero and hit <ENTER>. Notice that we also have

a deployment called "velero";

HOL-2033-01-CNA


6. Type kubectl get pod -n velero and hit <ENTER>. Check the new Velero podcalled "velero-<id>".

HOL-2033-01-CNA


Velero Backup and RestoreNow that we have our Velero server up and running with our MinIO storage service weare ready to backup some Kubernetes resources.

In our lab we are going to deploy a sample NGINX application test the backup andrestore features.

If you are not logged in the VM-CLI console please do it now following the "Accessing theVM-CLI console" topic.

Deploying a Sample Application

1. Type pwd and hit <ENTER> to make sure you are on the "/velero-files"directory;

2. Type ls and hit <ENTER>. Look for a directory called "/nginx-app";3. Type cd nginx-app and hit <ENTER> to change to this directory;4. Type ls and hit <ENTER>. Look for a file called "base.yaml". This file contains

all the specifications to our nginx deployment;5. Type vi base.yaml and hit <ENTER> to edit the deployment;

HOL-2033-01-CNA


6. Change the deployment image. Look for the line "- image:harbor-01a.corp.local:80/library/nginx:1.7.9" and change the tag from 1.7.9to 1.15-alpine.

7. Type <ESC> and hit <ENTER> . Type :wq and hit <ENTER> to save and quitvi editor;

8. Type kubectl apply -f base.yaml and hit <ENTER> to create the nginx sampleapplication deployment;

9. Type kubectl get ns and hit <ENTER> and notice that we now have a namespacecalled "nginx-example";

10. Type kubectl get deployment -n nginx-example and hit <ENTER> to list the nginxdeployment in the "nginx-example" namespace. You should see a deploymentcalled "nginx-deployment" with 2/2 annotation in the "READY" column. Thatmeans we have two nginx pod in this deployment.

11. Type kubectl get pod -n nginx-example and hit <ENTER> to list the pods in thenginx deployment;

12. Type kubectl get svc -n nginx-example and hit <ENTER> to find out what type ofservice is configured for our nginx deployment and also on which port it isrunning. You should see a service called "my-nginx" of type "NodePort" withthe translation from the default web port 80 to port 31355 (this port may bedifferent for your deployment);

13. Since we only have one worker node those nginx pods are certainly running on it.Type curl http://k8s-worker-01a.corp.local:31355 and hit <ENTER>. You should seea nginx welcome page in HTML code of course! You can test it in a browser to ifdesired. The sample application is working. Remember that the port may bedifferent for your environment.

HOL-2033-01-CNA


Backing Up the Sample Application

1. Type cat base.yaml | grep -A 1 selector and hit <ENTER> to list the "selectors"our sample app has. Notice that it has at least a selector called "app:nginx". Weare going to use this selector to create Velero backups.

2. Type velero backup create nginx-backup --selector app=nginx and hit <ENTER> tocreate a backup from our sample app called "nginx-backup";

3. Type velero backup get and hit <ENTER> to list all the backups we have. Noticethat right now we only have the backup we've just created, "nginx-backup" withthe status "Completed";

4. Type velero schedule create nginx-daily --schedule="0 1 * * *" --selector app=nginx andhit <ENTER> to create a daily schedule backup for our sample app called"nginx-daily". Notice that the flag --schedule uses a pattern based in the Cronutility;

5. Type velero schedule get and hit <ENTER> to list all the existant schedule. Rightnow we only have the schedule we've just created, "nginx-daily" with the status"Enabled";

6. Type velero backup get and hit <ENTER> again and notice that now we haveanother backup called "nginx-daily-<TIMESTAMP>" with the status

HOL-2033-01-CNA


"Completed". This backup was created by the backup schedule created on theprevious steps;

7. Now let's delete our sample app to simulate a disaster. Type kubectl deletenamespace nginx-example and hit <ENTER>;

8. Type kubectl get ns and hit <ENTER> and notice that we no longer have thenamespace "nginx-example";

9. Type kubectl get svc -n nginx-example and hit <ENTER> to check if we still havethe service and you should see "No resources found.". Thats because everytime we delete a Kubernetes namespace all the resources associated with it arealso deleted;

10. Type curl http://k8s-worker-01a.corp.local:31355 and hit <ENTER>. You should geta "Failed to connect..." message because our sample app was completelyremoved. Remember that the port in your deployment may be different.

HOL-2033-01-CNA


Checking the Backup from MinIO UI

1. Click on the <Chrome> icon in the Windows toolbar to open the ChromeBrowser;

2. Type vm-cli.corp.local:9000 and hit <ENTER> to open the MinIO UI. You should bealready logged in, if not, get back to the step where you did login and repeat it.Remember that the port used may be different from this one;

3. Click on <velero> and notice that now we have a folder called "/backups". Feelfree to explore it.

HOL-2033-01-CNA


Restoring the Sample Application

1. Type velero backup get and hit <ENTER>. Confirm we have a backup of oursample app called "nginx-backup";

2. Type velero restore create --from-backup nginx-backup and hit <ENTER> to restoreour sample app from the "nginx-backup";

3. Type velero restore get and hit <ENTER>. Confirm that we have a restore jobcalled "nginx-backup-<TIMESTAMP>" with the status "Completed" andwithout any warning or errors;

4. Type kubectl get ns and hit <ENTER>. Confirm that the namespace "nginx-example" is listed again;

5. Type kubectl get svc -n nginx-example and hit <ENTER>. Notice that the service"my-nginx" is back in place but this time in a different port: 32300. Rememberthat the port may be different for your deployment;

6. Type curl http://k8s-worker-01a.corp.local:32300 and hit <ENTER> and you shouldget the welcome page for our sample app again! Our restore worked like acharm!

HOL-2033-01-CNA


Checking the Restore from MinIO UI

1. Click on the <Chrome> icon in the Windows toolbar to open the ChromeBrowser;

2. Type vm-cli.corp.local:9000 and hit <ENTER> to open the MinIO UI. You should bealready logged in, if not, get back to the step where you did login and repeat it.Remember that the port used may be different from this one. Refresh the browserif the session was already opened;

3. Click on <velero> and notice that besides the "/backups" folder we now alsohave a "/restores" folder. Feel free to explore it.

HOL-2033-01-CNA


Velero Disaster RecoveryUsing Schedules and Restore-Only Mode

If you periodically back up your cluster's resources, you are able to return to a previousstate in case of some unexpected mishap, such as a service outage.

The process is very similar to a regular backup with the difference that we take it for allthe resources in the cluster which gives us a backup for the entire cluster.

In this lesson we are going to create a scheduled backup, simulate a disaster and then,restore the cluster resources.

If you are not connected to VM-CLI Linux box, follow the instructions from the "VeleroInstall and Configure" lesson to do it.

Creating a Kubernetes Cluster Scheduled Backup

HOL-2033-01-CNA


1. Type pwd and hit <ENTER> to make sure you are in the "velero-files"directory, if not change to it before proceeding;

2. Type kubectl get ns and hit <ENTER>. Notice that we still have the namespaces"velero" and "nginx-example";

3. Type velero create schedule k8s-cluster-a --schedule "0 7 * * *" and hit <ENTER> tocreate a weekly scheduled backup for the entire cluster. Notice that we did notuse the component selector as we did before and the schedule name is in a"domain form". That means we are creating a scheduled backup for the entirecluster and not only for specific resources.

4. Type velero get schedule and hit <ENTER>. We now also have a scheduledbackup for the entire cluster with the name "k8s-cluster-a";

5. Type velero get backup and hit <ENTER> to see our first backup from the createdschedule routine was already taken under the name "k8s-cluster-a-<TIMESTAMP>" and has the status "Completed".

HOL-2033-01-CNA


Simulating a Disaster

To simulate a disaster we are going to delete Velero and nginx-example deployment.In fact, we are going to delete the entire namespace for both of them. Since we haveour MinIO S3 Bucket with all the needed backups running as a container, we will onlyneed to connect the new Velero install to it.

1. Type kubectl get ns and notice we have both namespaces nginx-example andvelero;

2. Type kubectl delete ns nginx-example and hit <ENTER> to delete the namespace"nginx-example";

3. Type kubectl get deployments -n velero and hit <ENTER> to see the the existingdeployments in the "velero" namespace;

4. Type kubectl delete namespace/velero clusterrolebinding/velero and hit <ENTER>.This will delete the entire velero namespace as well as the Cluster Role Bindingscreated during Velero install;

5. Type kubectl delete crds -l component=velero and hit <ENTER>. This will delete allthe Custom Resources created during Velero install;

6. Type kubectl get ns and hit <ENTER>. Notice that we no longer have the veleroand nginx-example namespaces.

HOL-2033-01-CNA


The disaster was simulated. Now, let's recover from it.

Running Velero in Restore-Only Mode

In order to recover our kubernetes cluster from a disaster we need to re-install theVelero server pointing to the same storage server where our backups were stored. Thedifference now is that we are going to use the flag --restore-only true . This is because itwill prevent Backup objects from being created or deleted during your Restore process.

1. Type pwd and hit <ENTER> to make sure you are in the "velero-files"directory, if not change to it before proceeding;

2. Type velero install --provider aws --bucket velero --secret-file ./credentials-velerorestore-only --use-volume-snapshots=false --backup-location-configregion=minio,s3ForcePathStyle="true",s3Url=http://vm-cli.corp.local:9000 --imageharbor-01a.corp.local:80/heptio-images/velero:latest in a single line and hit <ENTER>to start the Velero server in "Restore-Only" mode;

3. Type kubectl get deployment -n velero and hit <ENTER>. You should now see thevelero server up and running again.

HOL-2033-01-CNA


Recovering the Kubernetes Cluster

1. Type velero get backup and hit <ENTER>. You should see the last cluster backupwe've taken in the previous lesson with the name "k8s.corp.local-<TIMESTAMP>";

2. Type velero restore create --from-backup k8s-cluster-a-<TIMESTAMP> and hit <ENTER>.The restore process begins;

3. Type velero get restore and hit <ENTER>. You'll see our restore job with a status"InProgress". Wait a couple of minutes and try the command again. You shouldnow see that the restore is with a "Completed" status;

4. Type kubectl get ns and hit <ENTER>. You should see the namespace "nginx-example" again and our cluster fully restored;

5. Type kubectl get pod -n nginx-example and hit <ENTER> to see the "nginx" podsback to life;

6. Type kubectl get svc -n nginx-example and hit <ENTER> to see that the "my-nginx" service is also back.

HOL-2033-01-CNA


Our lab environment is fairly simple and we just saw on resource to be recoveredbecause it all we had at the time of backup. Keep in mind that this recovery processcould work for a infinite number of resources in a Kubernetes cluster.

HOL-2033-01-CNA


Velero Cluster MigrationUsing Backups and Restores

Velero can help you port your resources from one cluster to another, as long as youpoint each Velero instance to the same cloud object storage location. In this scenario,we are also assuming that your clusters are hosted by the same cloud provider.

The migration process is very similar to the restore process we just saw in the lastchapter, the only difference is that the restore takes place in another Kubernetes cluster.

Note: Velero does not support the migration of persistent volumes acrosscloud providers.

In our lab we have two kubernetes clusters so we can test the Cluster Migrationfunctionality. The process is very similar to the Disaster Recovery, but the difference isthat the recover process occurs in another kubernetes cluster and not on the same one.

Migration Overall Process

Follow the migration overall process:

1. (Cluster A) Assuming you haven't already been checkpointing your data with theVelero schedule operation, you need to first back up your entire cluster(replacing <BACKUP-NAME> as desired) with the command velero backup create<BACKUP-NAME> . The default TTL is 30 days (720 hours); you can use the --ttl flagto change this as necessary;

2. (Cluster B) Add the --restore-only flag to the server spec in the Velerodeployment YAML.

3. (Cluster B) Make sure that the BackupStorageLocation and VolumeSnapshotLocationCRDs match the ones from Cluster 1, so that your new Velero server instancepoints to the same bucket.

4. (Cluster B) Make sure that the Velero Backup object is created with thecommand velero backup describe <BACKUP-NAME> . Velero resources are synchronizedwith the backup files in cloud storage. Note: The default sync interval is 1minute, so make sure to wait before checking. You can configure this interval withthe --backup-sync-period flag to the Velero server.

5. (Cluster B) Once you have confirmed that the correct Backup (<BACKUP-NAME>) is now present, you can restore everything with velero restore create--from-backup <BACKUP-NAME>

6. (Cluster B) Run the command velero restore get and velero restore describe<RESTORE-NAME-FROM-GET-COMMAND> to check that the second cluster is behaving asexpected.

7. If you encounter any issues make sure that Velero is running in the samenamespace in both clusters.

HOL-2033-01-CNA


Checking Kubernetes Clusters

In our lab environment we have to Kubernetes Clusters, Cluster A and Cluster B. Theway we access those different clusters within the same kubectl environment is makinguse of the kubeconfig files. We already have both kubeconfig files saved in our VM-CLImachine and also two variables configured for each one of them.

Notice that there are other way to access multiple Kubernetes clusters. To know moreabout it take a look at the article "Configure Access to Multiple Clusters".

1. Type kubectl get nodes --kubeconfig=$KUBECONFIG_A and press <ENTER>. This is thecluster, that we are now calling Cluster A, we've been working with during thelab. As already know, this cluster have 2 nodes, 1 master/etcd and 1 worker;

2. Type kubectl get nodes --kubeconfig=$KUBECONFIG_B and press <ENTER>. This is thenew cluster, Cluster B, that we are going to use to migrate the resources to inthis chapter. As you can see, this cluster has 3 nodes, 1 master/etcd and 2workers;

HOL-2033-01-CNA


https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/

3. Type kubectl get ns --kubeconfig=$KUBECONFIG_A and press <ENTER>. As expected,we have both namespaces we created from previous modules, nginx-exampleand velero;

4. Type kubectl get ns --kubeconfig=$KUBECONFIG_B and press <ENTER>. Notice that weonly have the default kubernetes namespaces since we did not create any kind ofresource in this cluster yet.

HOL-2033-01-CNA


Migrating Resources to Cluster B - Simulating a Disaster

In order to simulate a disaster we are going to completely shutdown the Cluster A. Sinceour backup files are stored in our MinIO server running in a docker container in the VM-CLI machine, we can safely shutdown the Cluster A without loosing access to the backupstore.

1. Click on the PuTTY icon on the upper left corner on the current windows session;

HOL-2033-01-CNA


2. Go to Saved Sessions and click on k8s-worker-01a.corp.local ;3. Once the session to the worker node of the Cluster A is opened type shutdown -h

now and press <ENTER>. !!!MAKE SURE YOU ARE SHUTTING DOWN THERIGHT HOST BY CHECKING THE NAME IN THE COMMAND PROMPT!!!

4. Click OK and close the inactive window session;5. (NOT SHOWN) Repeat the same above steps to shutdown the

k8s-master-01a.corp.local . !!!MAKE SURE YOU ARE SHUTTING DOWN THERIGHT HOST BY CHECKING THE NAME IN THE COMMAND PROMPT!!!

6. Back to the VM-CLI machine console make sure the Cluster A is out of service.Type kubectl get nodes --kubeconfig=$KUBECONFIG_A . You should receive an error"Unable to connect to the server...", if not, wait a couple more minutes untilthe Cluster A nodes are powered off.

HOL-2033-01-CNA


Installing Velero on Cluster B

Before we install Velero on Cluster B, let's move the Cluster B kubeconfig file to thedefault location so we don't have to pass it as a parameter every time we run acommand.

HOL-2033-01-CNA


1. Type cp $KUBECONFIG_B /home/holuser/.kube/config and press <ENTER> to copy theCluster B config file to the default config file location;

2. Type kubectl get nodes and press <ENTER> to confirm that we are now accessingthe Kubernetes Cluster B;

3. Type velero install --provider aws --bucket velero --secret-file ./credentials-velerorestore-only --use-volume-snapshots=false --backup-location-configregion=minio,s3ForcePathStyle="true",s3Url=http://vm-cli.corp.local:9000 --imageharbor-01a.corp.local:80/heptio-images/velero:latest in a single line and press<ENTER>. Notice that this command is the exact same command we ran in theprevious lesson to install Velero after the simulated disaster. The only different isthat we are now installing Velero in a different cluster.

4. Type kubectl get ns and press <ENTER>. Notice that we now have the Veleronamespace.

5. Type kubectl get deployment -n velero and press <ENTER> to check the Velerodeployment;

6. Type kubectl get pods -n velero and press <ENTER> to check that the Velero podis running.

HOL-2033-01-CNA


Checking Existing Backups

In the previous chapter, we created a backup and a backup schedule for the entirecluster configuration. It is a good practice to keep a backup schedule for yourKubernetes clusters in order to recover in case of a failure or migrate to another clusterdepending on the use case. We are going to use that previously created backup tomigrate resources from Cluster A to Cluster B.

1. Type velero backup get and press <ENTER> to get the backup name. The nameshould be k8s-cluster-a-<TIMESTAMP>;

2. Type velero backup describe k8s-cluster-a-<TIMESTAMP> and press <ENTER> to get adetailed description of the backup. Notice that every namespace and everyresource in the cluster was backed up. We ensured that by not specifying aLabel selector in the backup command.

HOL-2033-01-CNA


Migrating (Restoring) Resources

1. Type velero restore create --from-backup k8s-cluster-a-<TIMESTAMP> and press<ENTER> to restore the backup we just got from previous steps;

2. Type velero restore get and press <ENTER>. You'll see our restore job with astatus "InProgress". Wait a couple of minutes and try the command again. Youshould now see that the restore is with a "Completed" status and about 10warnings;

3. Type velero restore describe k8s-cluster-a-<TIMESTAMP>-<TIMESTAMP> and press<ENTER>. Notice that we now have two TIMESTAMPS in the restore job'sname, the first is related to the time the backup was taken and the second isrelated to the time the restore took place.

4. Here you can see what the 10 warnings were about. Since we backed upeverything in the Cluster A, as expected, some of the default resources/namespaces were not restored because they were already there. Those are someof the default resources common to all Kubernetes deployments. In the next stepswe are going to see what was in deed restored.

HOL-2033-01-CNA


Checking the New Cluster B Resources

1. Type kubectl get ns and press <ENTER> and notice that we now have anamespace called "nginx-example". This is the namespace created with thenginx appliction from ou Cluster A. This application was still there before thecluster shutdown.

2. Type kubectl get deployment -n nginx-example and press <ENTER> to list the nginxdeployment in the "nginx-example" namespace. You should see a deploymentcalled "nginx-deployment" with 2/2 annotation in the "READY" column. Thatmeans we have two nginx pod in this deployment as expected from the formercluster;

3. Type kubectl get pod -n nginx-example -o wide and press <ENTER> to list the podsin the nginx deployment and also on which worker they are running. Since wehave two worker nodes, Kubernetes intelligently run each pod on each host forthe sake of load balancing;

4. Type kubectl get svc -n nginx-example and press <ENTER> to find out what type ofservice is configured for our nginx deployment and also on which port it isrunning. You should see a service called "my-nginx" of type "NodePort" with

HOL-2033-01-CNA


the translation from the default web port 80 to port 32415 (this port may bedifferent for your deployment);

5. You could use either worker 1 or 2 since we have a pod running on each. Theresults should be the same. Type curl http://k8s-worker-01b.corp.local:32415 andpress <ENTER>. You should see a nginx welcome page in HTML code of course!You can test it in a browser too if desired. Remember that the port may bedifferent for your environment.

HOL-2033-01-CNA


ConclusionThis concludes "Module 3 - Velero - Back up and migrate Kubernetes resourcesand persistent volumes".

In this module we saw how to use Velero to backup, restore, recover from adisaster and also how to migrate resources in a Kubernetes deployment.



If you are looking for additional information on Velero, try one of these:

• Velero Home Page - https://velero.io• Velero Documentation - https://velero.io/docs• Velero GitHub Page - https://github.com/heptio/velero

Proceed to any module below which interests you most:

• Module 1 - Kubernetes Lifecycle Management with Kubeadm - (30minutes) (Basic) Discuss the lab environment setup, and how we can manage ourKubernetes deployment lifecycle using kubeadm.



How to End Lab


HOL-2033-01-CNA


https://velero.io

https://velero.io/docs/

https://github.com/heptio/velero

Module 4 - Sonobuoy:Validate Your KubernetesDeployment - iSIM Module

(15 minutes)

HOL-2033-01-CNA


Introduction

Welcome to Module 4 - Sonobuoy: Test your Kubernetes Deployment

This Module contains the following lessons:

• Sonobuoy Installation• Sonobuoy Conformance Tests

Sonobuoy Overview

Sonobuoy is a diagnostic tool that makes it easier to understand the state of aKubernetes cluster by running a set of plugins (including Kubernetes conformance tests)in an accessible and non-destructive manner. It is a customizable, extendable, andcluster-agnostic way to generate clear, informative reports about your cluster.

Its selective data dumps of Kubernetes resource objects and cluster nodes allow for thefollowing use cases:

• Integrated end-to-end (e2e) conformance-testing• Workload debugging• Custom data collection via extensible plugins

Sonobuoy supports 3 Kubernetes minor versions: the current release and 2 minorversions before. Sonobuoy is currently versioned to track the Kubernetes minor versionto clarify the support matrix. For example, Sonobuoy v0.14.x would support Kubernetes1.14.x, 1.13.x, and 1.12.x.

You can skip this version enforcement by running Sonobuoy with the --skip-preflightflag.

HOL-2033-01-CNA


Sonobuoy Installation andConformance TestsThis part of the lab is presented as a Hands-on Labs Interactive Simulation. This willallow you to experience steps which are too time-consuming or resource intensive to dolive in the lab environment. In this simulation, you can use the software interface as ifyou are interacting with a live environment.

1. Click here to open the interactive simulation. It will open in a new browserwindow or tab.

2. When finished, click the “Return to the lab” link to continue with this lab.

The lab continues to run in the background. If the lab goes into standby mode, you canresume it after completing the module.

HOL-2033-01-CNA


http://docs.hol.vmware.com/hol-isim/hol-2020/hol-isim-player.htm?isim=HOL-2033-CNA-Module3.htm

ISim Notes - Do Not PublishThis part of the lab is presented as a Hands-on Labs Interactive Simulation. In thissimulation, you can use the software interface as if you are interacting with a liveenvironment.

The orange boxes show where to click, and the left and right arrow keys can also beused to move through the simulation in either direction.

Note: This simulation includes long command line entries which fill bypressing any key then <enter> or just using the right arrow key.


In our lab we have a Linux server configured with all the needed tools to get through thelab lessons. We access this box via PuTTY which already has a saved session for thismachine

1. Click on the "puTTY" icon in the Windows toolbar;2. Click at the bottom arrow of the scroll bar to scroll to the end;3. Click the vm-cli.corp.local machine;4. Click <Open> in order to open the vm-cli console.

Checking the Prerequisites

1. Type echo $KUBECONFIG to make sure the variable is properly set and hit<ENTER>. Notice that it points to a config file inside "/home/holuser/.kube/"hidden directory. This file contains all the information necessary to get access toour Kubernetes cluster. The utility "kubectl" looks for this file every time it runs;

2. Type kubectl cluster-info and hit <ENTER>. Check if the Kubernetes cluster isrunning. We can see that the master is running and it is ready to accept APIrequests;

3. Type kubectl get nodes and hit <ENTER>. Notice that we have a Kubernetescluster composed by 1 master/etcd and 1 worker node. Also notice that theKubernetes cluster API version points to the version 1.15.1.

Installing Sonobuoy

Sonobuoy is an OpenSource project backed by VMware that is publicly available fromour git repository at https://github.com/heptio/sonobuoy/releases.

Since it is just a binary file the only thing we need to do is download the archive for theright platform and extract the binary to a place that make sense. In our lab, the archive

HOL-2033-01-CNA


https://github.com/heptio/sonobuoy/releases

was already downloaded so we only need to extract it to a reasonable place like thelocal binary directory.

Extracting the Sonobuoy Binary

1. Type cd sonobuoy-files and hit <ENTER> to change to this directory;2. Type ls -l and hit <ENTER> to list the Sonobuoy archive and its version. It should

read "sonobuoy_0.15.0_linux_amd64.tar.gz";3. Type sudo tar -zxvf sonobuoy_0.15.0_linux_amd64.tar.gz -C /usr/local/bin

sonobuoy and hit <ENTER> to extract Sonobuoy binary file to the local binarydirectory;

4. Type which sonobuoy and hit <ENTER> to check if the binary was copied tothe right directory;

5. Type sonobuoy version --kubeconfig $KUBECONFIG and hit <ENTER>;6. We should see some important information about Sonobuoy version including the

minimum and maximum Kubernetes version that it supports;7. We should also see what is the current Kubernetes API version of the cluster

described in the kubeconfig file. Notice that our Kubernetes cluster version is theversion 1.15.1.

Checking our Kubernetes Namespaces

Before we start running Sonobuoy Conformance Tests let's take a look at our kubernetescluster, its nodes and configuration.

1. Type kubectl get ns and hit <ENTER> to check the existing namespaces in ourcluster. Notice that we have only the default ones: "default", "kube-node-lease", "kube-public", "kube-system".

Running a Quick Conformance Test

Do not run Sonobuoy conformance tests without the --mode quick flag or itcould take up to 3 hours for the test to finish! For demonstration purpose aSonobuoy quick test should be enough.

The general architecture of Sonobuoy comprises a lot of applications that run oncontainers. First it needs to run two containers - one for the Sonobuoy application andanother one for the Conformance application. Once those applications are available weare ready to run end-to-end conformance tests (e2e). In order to run those testsSonobuoy needs to pull a lot of container images from public repositories. Thoserepositories are listed in a YAML file. Since our lab does not have Internet access, wehave to point not only the Sonobuoy and Conformance application container imagesto a local repository but we also have to change the public repositories URLs in our YAMLfile. To accomplish that we use 3 sonobuoy command flags and a custom YAML file. Thecustom YAML file in our lab is located at "/sonobuoy-files/sonobuoy-repos.yaml"and the flags are --e2e-repo-config, --sonobuoy-image, --kube-conformance-

HOL-2033-01-CNA


image. Remember that if we are in a environment with Internet access those flagswouldn't be necessary.

1. Type sonobuoy run --mode quick --e2e-repo-config sonobuoy-repo.yaml --sonobuoy-image harbor-01a.corp.local:80/heptio-images/sonobuoy:v0.15.0 --kube-conformance-image harbor-01a.corp.loca:80/google-containers/conformance:v1.14.0 and hit <ENTER>. Wait for thecommand to finish giving you back the command prompt;

2. Type sonobuoy status and hit <ENTER> to check the status of the test. Thestatus is still "running". Repeat the command again and see the statuschanges to "complete" for the default e2e (end-to-end) conformance testplugin;

3. Type sonobuoy logs and hit <ENTER> to display the logs for the test we've justran;

Checking Resources Created

Sonobuoy creates a few resources in order to run and expects to run within its ownnamespace. Let's check some of them after we've ran the conformance test.

1. Type kubectl get ns and hit <ENTER>. Notice that we now have a namespacecalled "heptio-sonobuoy";

2. Type kubectl get pod -n heptio-sonobuoy and hit <ENTER> to get all thepods running in the namespace "heptio-sonobuoy". You should see a pod called"sonobuoy" with the status "Running";

3. Type kubectl get svc -n heptio-sonobuoy and hit <ENTER> to get all theservices created in the namespace "heptio-sonobuoy". You should see aservice called "sonobuoy-master" of type "ClusterIP";

4. Type kubectl get sa -n heptio-sonobuoy and hit <ENTER> to get all theservice accounts created in the namespace "heptio-sonobuoy". You shouldsee a service account called "sonobuoy-serviceaccount".

Retrieving the Test Results

1. Make sure you are in the "sonobuoy-files" directory, if not change to it beforeproceeding. To view the output, copy the output directory from the aggregatorSonobuoy pod to your local machine (and save the name of the file to a variablefor reference): Type results=$(sonobuoy retrieve) and hit <ENTER>;

2. Type echo $results and hit <ENTER> to check if the variable have the propervalue set. It should display a gzipped tarball that we call a Sonobuoy "snapshot"which will be explained later (the name of the tarball is probably different fromthe one you get since its name is based in date/time variables);

3. The general results can be inspected without being extracted. To list the numberof tests failed and their names Type sonobuoy e2e $results and hit <ENTER>.You should have "0" as the number of failed tests.

HOL-2033-01-CNA


Extracting the Test Results

1. Type ls and hit <ENTER> to see the name of the file that we saved in thevariable $results. A Sonobuoy snapshot is a gzipped tarball, namedYYYYmmDDHHMM_sonobuoy_<uuid>.tar.gz where YYYYmmDDHHMM is atimestamp containing the year, month, day, hour, and minute of the run. The<uuid> string is an RFC4122 UUID, consisting of lowercase hexadecimalcharacters and dashes. This UUID should match the UUID from the snapshot's/meta/config.json, stored at the root of the tarball.

2. Type mkdir ./results; tar xzf $results -C ./results and hit <ENTER> in orderto extract the snapshot to a "results" directory;

3. Type ls and hit <ENTER> to confirm that a "results" directory was created;4. Type cd results and hit <ENTER> to change to the "results" directory;5. Type ls -l and hit <ENTER> to check the "results" directory's structure.

Analyzing the Test Results

The top-level directories in the results tarball has a specific structure which will bediscussed next.

The hosts Directory

The "hosts" directory contains the information gathered about each host in the systemby directly querying their HTTP endpoints. This is different from what you find in"resources/cluster/Nodes.json" - it contains items that aren't part of the KubernetesAPI objects:

• "hosts/<hostname>/configz.json" - Contains the output of querying the/configz endpoint for this host - that is, the component configuration for the host;

• "hosts/<hostname>/healthz.json" - Contains a json-formatted representationof the result of querying /healthz for this host, for example {"status":200}.

1. Type cd $HOME/sonobuoy-files/results/hosts and hit <ENTER> to change tothe "hosts" directory;

2. Type ls -l and hit <ENTER>. You should see one directory for each node of ourkubernetes cluster. In our case there are 2 nodes, 1 master/etcd and 1 worker;

3. Type cd k8s-worker-01a.corp.local and hit <ENTER> to change to this nodedirectory;

4. Type ls -l and hit <ENTER>. You will see the two files discussed above.

The meta Directory

The "meta" directory contains metadata about this Sonobuoy run, includingconfiguration and query runtime:

HOL-2033-01-CNA


• "meta/query-time.json" - Contains metadata about how long each query took,example: {"queryobj":"Pods","time":12.345ms"};

• "meta/config.json" - A copy of the Sonobuoy configuration that was set upwhen this run was created, but with unspecified values filled in with explicitdefaults, and with a UUID field in the root JSON, set to a randomly generatedUUID created for that Sonobuoy run;

• "meta/run.log" - Contains the log of this Sonobuoy test run (It is the results ofthe sonobuoy logs command we executed earlier on this module).

1. Type cd $HOME/sonobuoy-files/results/meta and hit <ENTER> to get back tothe "/results" directory;

2. Type ls -l and hit <ENTER>. You will see the 3 files discussed above.

The plugins Directory

The "plugins" directory contains output for each plugin selected for this Sonobuoy run:

• "plugins/<plugin>/results.<format>" - For plugins that run on an arbitrarynode to collect cluster-wide data, for example using the Job driver. Contains theresults for the plugin, using the format that the plugin expects;

• "plugins/<plugin>/results/<hostname>.<format>" - For plugins that runonce on every node to collect node-specific data, for example using theDaemonSet driver. Contains the results for the plugin, for each node, using theformat that the plugin expects.

Some plugins can include several files as part of their results. The extracted files forthese plugins comprise:

• "plugins/<plugin>/results/<extracted files>" - For plugins that collectcluster-wide data into a .tar.gz file

• "plugins/<plugin>/<node>/<extracted files>" - For plugins that collect per-node data into a .tar.gz file

1. Type cd $HOME/sonobuoy-files/results/plugins and hit <ENTER> to get backto the "/results" directory;

2. Type ls -l and hit <ENTER>. Depending on the plugins configured to run with thetest, this content may vary according to the above explanation. In our case wejust ran a quick test using the e2e plugin;

3. Type cd e2e and hit <ENTER> to change to the "e2e" plugin directory;4. Type ls -l and hit <ENTER>. You will see a "results" directory;5. Type cd results and hit <ENTER> to change to the plugin "results" directory;6. Type ls -l and hit <ENTER>. In our case you will see 2 files, e2e.log and

junit_01.xml.

HOL-2033-01-CNA


The podlogs Directory

The "podlogs" directory contains logs for each pod found during the Sonobuoy run,similar to what you get with kubectl logs -n <namespace> <pod> <container>.

• "podlogs/<namespace>/<podname>/<containername>.log" - Contains thelogs for each container, for each pod in each namespace.

1. Type cd $HOME/sonobuoy-files/results/podlogs and hit <ENTER> to getback to the "results" directory;

2. Type ls -l and hit <ENTER>. You will see a directory called "heptio-sonobuoy"that represents the namespace created by the Sonobuoy test;

3. Type cd heptio-sonobuoy and hit <ENTER> to change to the "heptio-sonobuoy" directory;

4. Type ls -l and hit <ENTER>. You will see two directories representing the podscreated by the Sonobuoy test;

5. Type cd sonobuoy and hit <ENTER> to change to the "sonobuoy" poddirectory;

6. Type ls -l and hit <ENTER>. You will see the logs directory;7. Type cd logs and hit <ENTER> to change to the "logs" directory for the pod;8. Type ls -l and hit <ENTER>. You will see a log file called "kube-sonobuoy.txt".

This is the log file for the sonobuoy pod.

The resources Directory

The "resources" directory lists JSON-serialized Kubernetes objects, taken from queryingthe Kubernetes REST API. The directory has the following structure:

• "resources/ns/<namespace>/<type>.json" - For all resources that belong toa namespace, where <namespace> is the namespace of that resource (eg.kube-system), and <type> is the type of resource, pluralized (eg. Pods).

• "resources/cluster/<type>.json" - For all resources that don't belong to anamespace, where <type> is the type of resource, pluralized (eg. Nodes).

1. Type cd $HOME/sonobuoy-files/results/resources and hit <ENTER> to getback to the "results" directory;

2. Type ls -l and hit <ENTER>. You will see the two directories mentioned above,"ns" and "cluster";

3. Type cd cluster and hit <ENTER> to change to the "cluster" directory;4. Type ls -l and hit <ENTER>. Here you can see all resources that don't belong to

a namespace as explained above;5. Type cd ../ns and hit <ENTER> to change to the sonobuoy "ns" directory;6. Type ls -l and hit <ENTER>. You will see all the existing namespaces in the

Kubernetes cluster. You should also see all the resources created for eachnamespace inside those directories.

HOL-2033-01-CNA


The servergroups.json and serverversion.json files

• "servergroups.json" lists the Kubernetes APIs that the cluster supports;• "serverversion.json" contains the output from querying the server's version,

including the major and minor version, git commit, etc.

1. Type cd $HOME/sonobuoy-files/results/ and hit <ENTER> to get back to the"results" directory;

2. Type ls -l and hit <ENTER>. You will see the two files "servergroups.json" and"serverversion.json" as explained above. Feel free to check the file's content.

Cleaning Up Sonobuoy

Sonobuoy creates a few resources in order to run and expects to run within its ownnamespace. Deleting Sonobuoy entails removing it's namespace as well as a few clusterscoped resources.

1. Type sonobuoy delete --wait and hit <ENTER>. Watch the created resourcesbeing deleted. The --wait option ensures the Kubernetes namespace is deleted,avoiding conflicts if another Sonobuoy run is started quickly;

2. Type kubectl get pods -n heptio-sonobuoy and hit <ENTER>. Notice thatthere were "No resources found.". That's because the namespace "heptio-sonobuoy" was deleted and therefore there will be no pod as well. Rememberthat when a Kubernetes namespace is deleted all the resources belonging to thatnamespace are also deleted;

3. Type sonobuoy status and hit <ENTER>. You'll get an error telling that thenamespace "heptio-sonobuoy" was not found.

All the resources created by the Sonobuoy conformance test were now successfullyremoved.

To return to the lab, click the link in the top right corner or close this browser tab.

HOL-2033-01-CNA


ConclusionThis concludes "Module 4 - Test your Kubernetes Deployment".

In this module we saw how to use Sonobuoy to run conformance test inaccordance with "CNCF - Cloud Native Computing Foundation" and how toanalyze the test results.



If you are looking for additional information on Sonobuoy, try one of these:

• Sonobuoy Home Page - https://sonobuoy.io• Sonobuoy Documentation - https://sonobuoy.io/docs• Sonobuoy GitHub Page - https://github.com/heptio/sonobuoy

Proceed to any module below which interests you most:

• Module 1 - Kubernetes Lifecycle Management with Kubeadm - (30minutes) (Basic) Discuss the lab environment setup, and how we can manage ourKubernetes deployment lifecycle using kubeadm.



How to End Lab


HOL-2033-01-CNA


http://sonobuoy.io

https://sonobuoy.io/docs/v0.15.0/

https://github.com/heptio/sonobuoy

ConclusionThank you for participating in the VMware Hands-on Labs. Be sure to visithttp://hol.vmware.com/ to continue your lab experience online.

Lab SKU: HOL-2033-01-CNA

Version: 20200310-124121

HOL-2033-01-CNA


http://hol.vmware.com/

Documents

Table of Contents · The Kubernetes pod was deployed using upstream Kubernetes version 1.15.1. What is kubeadm? kubeadm is a tool included in every Kubernetes release that helps users