Dell EMC XtremIO X2 CSI Plugin Integration with Pivotal PKS · 2019-08-06 · Pivotal Container Service (PKS) is a purpose-built product that enables enterprises to deploy and consume

Dell EMC XtremIO X2 CSI Plugin Integration with Pivotal PKS © 2019 Dell Inc. or its subsidiaries.

Dell Customer Communication - Confidential Dell Customer Communication - Confidential

DELL EMC XTREMIO X2 CSI PLUGIN INTEGRATION WITH PIVOTAL PKS A Detailed Review

Abstract

This white paper evaluates the best-in-class performance and scalability delivered by Dell EMC XtremIO X2, using the Container Storage Interface (CSI) plugin to deliver persistent storage for Cloud-Native Applications and DB workloads managed by Pivotal PKS platform.

July 2019

WHITE PAPER

2 | Dell EMC XtremIO X2 CSI Plugin Integration with Pivotal PKS © 2019 Dell Inc. or its subsidiaries.


Contents Abstract ................................................................................................................................................................. 1

Intended Audience ................................................................................................................................................ 3

Introduction ........................................................................................................................................................... 3

Business Challenge .............................................................................................................................................. 4

Key Benefits .......................................................................................................................................................... 5

Pivotal Container Service (PKS) Overview ........................................................................................................... 7

VMware NSX-T ..................................................................................................................................................... 7

Dell EMC XtremIO X2: The Next Generation All-Flash Array ............................................................................... 7

What is Kubernetes? ............................................................................................................................................ 8

What is Container Storage Interface? ................................................................................................................... 8

Dell EMC XtremIO and Pivotal PKS – Better Together ........................................................................................ 9

Integrating Dell EMC XtremIO CSI Plugin with Pivotal PKS ............................................................................... 11

Platform and Software Dependencies .............................................................................................................................. 11

Supported XtremIO Versions: ....................................................................................................................................... 11

Installing PKS .................................................................................................................................................................... 11

Downloading and Extracting the Installer ...................................................................................................................... 12

Configuring the Installer ................................................................................................................................................ 12

Generating Dell EMC XtremIO CSI Plugin YAML ......................................................................................................... 14

Deploying Mandatory BOSH Add-Ons ............................................................................................................... 16

Managing PKS with the PKS CLI ........................................................................................................................ 16

Connecting to the Kubernetes Dashboard .......................................................................................................... 20

Health Monitoring and Troubleshooting .............................................................................................................. 21

Deploying Persistent Volumes and Applications ................................................................................................ 23

Example Use Case: PostgreSQL ...................................................................................................................................... 24

Conclusion .......................................................................................................................................................... 28

References ......................................................................................................................................................... 29

How to Learn More ............................................................................................................................................. 30



Intended Audience

This white paper is designed for developers, system architects and storage administrators tasked with evaluating or deploying Pivotal PKS, and interested in persistent storage within their deployment. While this document focuses on Dell EMC XtremIO X2 products, the concepts discussed and implemented can be applied to other storage vendor products.

Understanding the material in this document requires prior knowledge of containers, Kubernetes, PKS and of the Dell EMC XtremIO X2 cluster type. These concepts are not covered in detail in this document. However, links to helpful resources on these topics are provided throughout the document and in the References section. In-depth knowledge of Kubernetes is not required.

Introduction

To improve competitiveness in a fast-paced world, organizations are embracing software as a prime differentiator. A software-driven business is more agile and efficient, innovates faster and responds dynamically to changing market and customer demands. Increasingly, gaining these compelling advantages requires adopting cloud-native applications.

Kubernetes is the de facto standard for container orchestration for microservices and applications. However, enterprise adoption of big data and databases using containers and Kubernetes is hindered by multiple challenges, such as complexity of persistent storage, availability, and application lifecycle management. Kubernetes provides the agility and scale that modern enterprises require. However, Kubernetes provides the building blocks for infrastructure, but not a turnkey solution.

Originally containers and Kubernetes were optimized for stateless applications, restricting enterprises to use container technology for workloads, such as websites and user interfaces. More recently, stateful workloads have been increasingly run on Kubernetes. Whereas it is relatively easy to run stateless microservices using container technology, stateful applications require slightly different treatment.

There are multiple factors which need to be taken into account when considering handling persistent data using containers, including:

• Containers are ephemeral by nature, so the data that needs to be persistent must survive through the restart/re-scheduling of a container.

• When containers are rescheduled, they can die on one host, and might get scheduled on a different host. In such cases, the storage should also be shifted and made available on a new host for the container to start gracefully.

• The application should not have to deal with the volume/data and underlying infrastructure, and should handle the complexity of unmounting and mounting, etc.

• Certain applications have a strong sense of identity (such as Kafka, Elastic, etc.), and the disk used by a container with a specific identity is tied to it. If, for any reason, a container with a specific identity becomes rescheduled, it is important that the disk specifically associated with that identity, gets reattached on a new host.

In this white paper, we present a solution that provides the robust capabilities and built-in high availability and scalability of XtremIO and PKS, bringing simplicity to Kubernetes, and creating a turnkey solution for data-heavy workloads. XtremIO CSI Plugin provides Pivotal PKS technology with built-in enterprise-grade container storage and uncompromising performance that extends Kubernetes’ multi-cloud portability to the private cloud.

With this proven integration, organizations can accelerate their digital business objectives via cloud-native applications, from development through testing and into production, on a single platform with ease and agility. Most importantly, eliminating the risk of application downtime preserves critical revenue streams, helps ensure customer satisfaction and strengthens your competitive advantage.



Business Challenge

Planning, designing and building a private or hybrid cloud to support cloud-native applications can be a complex and lengthy project that does not address immediate business needs. IT also must maintain and manage the infrastructure that supports these new applications, to ensure that the environment is reliable, secure and upgradeable.

Kubernetes offers great capabilities. However operationalization of these capabilities is not a trivial task. There are many layers involved, such as hardware, operating system, container runtime, software-defined networking, and Kubernetes itself. And all the layers require Day 0, Day 1 and Day 2 activities. Most of these layers also require constant patching (for example, Kubernetes ships a new version every three months), so the effort to maintain this infrastructure grows exponentially.

Kubernetes is the de facto standard for container orchestration platforms. However, Kubernetes can be difficult to operationalize and maintain for the following reasons:

• High availability — There is no out-of-the-box fault-tolerance for the cluster components themselves (masters and etcd nodes).

• Scaling — Kubernetes clusters handle scaling the pod/service within the nodes, but do not provide a mechanism to scale masters and etcd Virtual Machines (VMs).

• Health checks and healing — The Kubernetes cluster performs routine health checks for node health only.

• Upgrades — Rolling upgrades on a large fleet of clusters is difficult. Who manages the system it runs on?

• Operating systems — A Kubernetes installation requires multiple machines where an OS is installed. Installing and patching these systems is an overhead.

Pivotal Container Service (PKS) is a purpose-built product that enables enterprises to deploy and consume container services with production-grade Kubernetes, built with high availability, security, multi-tenancy and operational efficiency across private and public clouds.

It effectively solves the complexities around Kubernetes infrastructure and enables you to focus on your business applications. Furthermore, it benefits your developers, providing them with the latest version of Kubernetes and simple integrations with monitoring and logging tools.



Key Benefits

With Pivotal Container Service (PKS), not only do you receive a vanilla Kubernetes distribution supported by Pivotal and VMware, but you also receive the automation to maintain the Kubernetes stack with reduced effort, including the operating system, container runtime, software-defined networking and Kubernetes itself. The automation provided by BOSH enables you to install (Day 0), configure (Day 1), patch and maintain your Kubernetes infrastructure in a healthy state over time (Day 2).

This automation also extends your capabilities to have an effective Kubernetes as a service offering, and lets you create Kubernetes clusters with a single command, to either expand or shrink the clusters, or to repair them as needed. This is a unique capability that enables you to provide your Kubernetes users with as many clusters as they need, and to resize them, leveraging automation as your workload requirements change.

PKS also simplifies the networking configuration, by providing you with VMware NSX-T, which is a software-defined network infrastructure to build cloud-native application environments.

PKS also provides you with a container registry to host your containers, namely, Harbor, a graduated CNCF project.

Pivotal PKS Components



For the above reasons, Pivotal and VMware, in partnership with Google, launched PKS with the vision of offering a turnkey solution to provision, operate and manage enterprise-grade Kubernetes clusters:

• Built with open-source Kubernetes — Constant compatibility with the current stable release of Kubernetes, operated by BOSH. No proprietary extensions.

• Production-ready — Highly available from apps to infrastructure with no single points of failure. Built-in health checks, scaling, auto-healing and rolling upgrades.

• Multicloud — BOSH provides a reliable and consistent operational experience for any cloud.

• Network management and security — Out-of-the-box with VMware NSX-T. Multi-cloud, multi-hypervisor.

• Fully automated Ops — Fully automated deploy, scale, patch and upgrade. No downtime. Can also use CD pipelines to deploy your platform.

• Harbor — Enterprise-class container registry server. Harbor provides vulnerability scanning and identity management for Docker images, and supports multiple registries.

• Storage persistence for stateful applications — PKS leverages open source projects Hatchway to implement persistent storage for K8s. No additional manual installation is required. The plugin supports static and dynamic provisioning of volumes for Kubernetes.

• Monitoring integration — Monitoring integrations with vRLI and vROPs, and eventually Wavefront. Also, as PKS delivers OSS Kubernetes, any monitoring tool that works with Kubernetes can be integrated into clusters provisioned by PKS.

• Load Balancing — Load balancing functionalities for Kubernetes are provided via NSX-T, and HA for the load balancing solution is achieved thanks to the vSphere infrastructure powering PKS and NSX-T.

By running Kubernetes on top of BOSH, the following benefits are achieved:

DAY 1 – BUILD

• Deploy Kubernetes cluster via BOSH.

• Deploy Kubernetes clusters on demand.

DAY 2 – OPERATE

• Self-healing VMs and monitoring.

• Elastic scaling for clusters.

• Rolling upgrades to latest Kubernetes release.

• High-availability and multi-availability Zones support.



Pivotal Container Service (PKS) Overview

PKS runs enterprise-grade Kubernetes containers and uses BOSH to build and maintain highly-available Kubernetes clusters. PKS uses software-defined networking that is incorporated with VMware NSX and uses a Harbor-based container registry. Developers receive on-demand managed vanilla Kubernetes clusters, and operators can easily run secure Kubernetes clusters.

PKS is production-ready and highly available, and includes dynamic load balancing, zero downtime updates, automated health checks and more. PKS enables IT teams to flexibly deploy and consume Kubernetes on-premises with vSphere, or in a public cloud. Due to PKS being powered by BOSH, you get a reliable and consistent operating experience on any cloud.

PKS uses NSX-T to programmatically manage software-defined virtual networks, and to keep your Kubernetes network secure. IT operators can use BOSH to deploy on-demand Kubernetes clusters through the command line and the API. BOSH constantly monitors the health of Kubernetes clusters deployed by PKS.

PKS integrates easily with VMware software like vRealize Operations Manager, VMware Wavefront, vRealize Automation and vRealize Log Insight. PKS comes with Harbor; an enterprise-class container registry server. Harbor provides vulnerability scanning and identity management for Docker images and supports multiple registries.

VMware NSX-T

VMware NSX-T provides an agile, software-defined network infrastructure to build cloud-native application environments.

NSX-T network virtualization programmatically creates, snapshots, deletes, and restores software-based virtual networks. With network virtualization, the functional equivalent of a network hypervisor reproduces the complete set of Layer 2 through Layer 7 networking services, such as switching, routing, access control, firewalling, and QoS in software. These services can be programmatically assembled in any combination to produce unique, isolated virtual networks in a matter of seconds.

NSX-T works by implementing three separate but integrated planes: management, control, and data. The three planes are implemented as a set of processes, modules, and agents residing on three types of nodes: manager, controller, and transport nodes.

NSX-T is designed for management, operation and consumption by development organizations. NSX-T enables IT and development teams to choose the technologies best-suited for their applications. IaaS providers can use NSX-T to provide self-service functionality to their users, by deploying it on-premises. NSX-T is built for developer-ready infrastructure and it can automatically integrate with Kubernetes.

Dell EMC XtremIO X2: The Next Generation All-Flash Array

Dell EMC XtremIO is a purpose-built all-flash array that provides consistent high performance with low latency; unmatched storage efficiency with inline, all-the-time data services; rich application integrated copy services; and unprecedented management simplicity.

The next-generation platform, XtremIO X2, builds upon unique features of XtremIO to provide even more agility and simplicity for your data center and business. Content-aware in-memory metadata, and inline all-the time data services have made XtremIO the ultimate platform for virtual server and desktop environments and workloads that benefit from efficient copy data management. The architecture offers VMware XCOPY operations that are twice as fast, which can reach as high as 40 GB/s bandwidth, due to them being in-memory operations.

X2 includes inline, all-the-time data services, thin provisioning, deduplication, compression, replication, D@RE, XtremIO Virtual Copies (XVC), and double SSD failure protection with zero performance impact. Only XtremIO’s unique in-memory metadata architecture can accomplish this.



What is Kubernetes?

Kubernetes is a container orchestration system that automates lifecycle operations such as deployment, scaling and management for containerized applications. Initially developed by Google and later open-sourced, Kubernetes is now managed by Cloud Native Computing Foundation (CNCF).

Kubernetes is built using master-slave architecture common to most distributed computing systems. A Kubernetes cluster consists of at least one master node and multiple worker nodes. Master nodes are responsible for running the master components that provide the cluster’s control plane. Master components can run on multiple master nodes, but, for simplicity, they are started on the same node using set up scripts. The designated master node does not run any user containers. On the slave side, node components run on each worker node, and are responsible for maintaining running pods and providing the Kubernetes runtime environment.

What is Container Storage Interface?

The CSI is a community-led effort to develop a standardized API for container orchestration platforms (COs) to talk to storage plugins. Theoretically, with a standardized communication protocol, storage vendors only need to write a plugin to a single specification. CSI sets out the definition of how to talk to the plugin. Exactly how the plugin is managed or operates is up to the storage provider.

CSI provides the following capabilities:

• Dynamic provisioning and decommissioning of volumes

• Attachment and detachment of volumes from a host node

• Mounting and unmounting of a volume from a host node

CSI makes installing new volume plugins as easy as deploying a pod and enables third-party storage providers to develop their plugins without needing to add code to the core Kubernetes code base.

Various community members from Kubernetes, Mesosphere and Docker worked on this specification together. This new interface is a major benefit to the container ecosystem, because it standardizes the model for integrating storage systems across container orchestrators. Specifically, it frees the storage system driver from the Kubernetes release schedule, as the driver is no longer incorporated in the same code base (in-tree). With CSI, you can install storage system drivers independently of the container orchestration releases, providing faster bug fixes and features, while reducing code sprawl.

With all major orchestration engines working toward supporting CSI, DevOps teams have only to worry about one configuration for their environments, which helps in keeping everything portable regardless of where you are running your computer. This new plugin model further eases development and provides peace of mind for users running persistent workloads on Dell EMC XtremIO, both in terms of performance and availability.

Before beginning with provisioning storage, it is important to understand the Kubernetes concepts of a persistent volume and a persistent volume claim, and how they work together in a cluster. You can find detailed information in the Dell EMC XtremIO CSI White Paper.



Dell EMC XtremIO and Pivotal PKS – Better Together

Over the last few years, software containers have become a big deal. According to Gartner, by the end of 2016, 25% of workloads were deployed using containers. The wide adoption of containers has been primarily driven by developers and yet challenges remain, such as how to operate and manage an environment in which these containers run securely, at scale.

PKS is exactly that environment; a platform that allows application teams to self-serve provision Kubernetes clusters that are fully managed on their behalf, enabling them to focus on the workloads that serve their consumers rather than the substrate needed to run those workloads.

The platform is enterprise grade, providing the following capabilities:

• Production-ready — Highly available from apps to infrastructure, with no single points of failure. Built-in health checks, scaling, auto-healing and rolling upgrades.

• Fully automated Ops — Fully automated deploy, scale, patch, upgrade. No downtime. Can also use CD pipelines to deploy your platform.

• Multicloud infrastructure automation — BOSH for automated infrastructure provisioning, configuration, operation, and remediation on any cloud or IaaS.

• Commercial support — Enterprise grade assistance from Pivotal’s support network.

There are several factors to consider when running stateful apps in Kubernetes:

• Pods are ephemeral by nature. Data that needs to be persisted must be accessible upon restart and rescheduling of a pod.

• When a pod is rescheduled, it may be on a different host. Storage must be available on the new host for the pod to start gracefully.

• The app should not manage the volume and data. The underlying infrastructure should handle the complexity of unmounting and mounting.

• Certain apps have a strong sense of identity. When a container with a certain ID uses a disk, the disk becomes tied to that container. If a pod with a certain ID gets rescheduled, the disk associated with that ID must be reattached to the new pod instance.

The integration between Pivotal PKS and Dell EMC XtremIO merges the benefits of the PKS and the unmatched performance and data services of XtremIO to the world of containers. PKS is highly available up and down the stack, from apps to infrastructure. The service also includes dynamic load balancing, zero downtime updates, automated health checks and much more. With PKS, you can deploy, scale, patch and upgrade all the Kubernetes clusters in your system without downtime, and rapidly apply security fixes when new vulnerabilities are detected. Combining Dell EMC XtremIO CSI Plugin with PKS enables customers to deliver container-based applications on-premises for both development and production scale. XtremIO CSI Plugin support for Kubernetes enables container-based applications to be deployed, managed and scaled. With Dell EMC XtremIO X2’s unmatched levels of performance, availability and TCO, organizations can now achieve the same levels of business agility offered by the public cloud, with the benefits of running them on premises.



The solution presented in this white paper is fully engineered and pre-tested to ensure consistency across multiple platform deployments and reliable operation from day one, through its entire life cycle of upgrades and enhancements. Container’s high-availability feature keeps your most critical cloud-native applications running twenty-four hours a day, seven days a week. Using built-in health monitoring and orchestrated self-healing, this solution eliminates the risk of application downtime and the costly damage it could inflict on revenue streams and customer confidence.

Pivotal PKS and Dell EMC XtremIO X2 Architecture



Integrating Dell EMC XtremIO CSI Plugin with Pivotal PKS

Platform and Software Dependencies

• Pivotal PKS 1.4 and above

• iSCSI Portal configured at the XtremIO level

• iSCSI vSphere network for PKS virtual machines

Note: Pivotal PKS with NSX-T has not tested.

Supported XtremIO Versions:

• XtremIO X2 XIOS 6.2.0

• XtremIO X2 XIOS 6.2.1

• XtremIO X1 XIOS 6.2.0 (limited functionally, please contact Dell EMC support for specific details)

• XtremIO X1 XIOS 6.2.1 (limited functionally, please contact Dell EMC support for specific details)

Note: Multi-cluster functionally is supported. Note that all Dell EMC XtremIO X-Bricks must be of the same generation.

Installing PKS

To install PKS: 1. Download the product file from Pivotal Network.

2. Open a browser and navigate to https://YOUR-OPS-MANAGER-FQDN/ to log in to the Ops Manager Installation Dashboard.

3. Click Import a Product to upload the product file.

4. In the left column, under Pivotal Container Service, click the plus sign to add this product to your staging area.

5. Click the orange Pivotal Container Service icon, as shown in Figure 3, to start the configuration process.

Pivotal Container Service Icon

The complete installation instructions are in the VMware Enterprise PKS Documentation.

https://your-ops-manager-fqdn/



Downloading and Extracting the Installer

Clone the latest version of the XtremIO CSI installer bundle from the CSI XtremIO Deploy GitHub Repository to the Linux machine with network connectivity to the XMS, and extract it by running the following commands:

Configuring the Installer

Edit the csi.ini configuration file.

Note: The installation script is configured to download the required Docker images from the Internet by default. The installation script gets the current XtremIO CSI Docker image from the Dell EMC Storage Docker Hub Repository. Alternatively, if your environment has no Internet access directly or via proxy, you can use 33 (or any other local registry) as an internal Docker repository, and then upload the images to the registry and set plugin_path and csi_.._path variables, accordingly. See Figure 4 on page 13.

Mandatory Fields:

• management_ip — The management IP address or FQDN of the XtremIO Management Server.

• csi_user and csi_password — The credentials used by the plugin to connect to the XtremIO Management Server. csi_user and csi_password can be created in advance (as suggested above) or they can be created by an installation script. If user creation is left to the script, the user is required to provide initial_user and initial_password in the csi.ini. For example; the credentials of the Storage Administrator.

• kubelet_root=/var/vcap/data/kubelet — The Kubelet path for PKS deployments.

• ephemeral_node=True — This parameter is set to True when the Kubernetes nodes are ephemeral.

$ git clone https://github.com/dell/csi-xtremio-deploy.git

$ cd csi-xtremio-deploy && git checkout 1.2.0-1

https://hub.docker.com/r/dellemcstorage/csi-xtremio

https://goharbor.io/



Figure 4 presents the case when the Storage Administrator has the credentials admin/password123 and the plugin uses it to create the CSI user during the installation - csiuser/password456.

The example in Figure 4 also shows that the PKS nodes do not have direct access to the Internet. The required Docker images have been downloaded, and uploaded to VMware Harbor, so the plugin is set to get the necessary Docker images from the local Harbor registry.

csi.ini Example File points to VMware Harbor Registry

Note: The initial user is no longer in use by the plugin following the installation.

management_ip=vxms-xbrick1100.pks.xtremio

initial_username=admin

initial_password=password123

csi_username=csiuser

csi_password=password456

force=No

verify=Yes

kubelet_root=/var/vcap/data/kubelet

plugin_path=harbor.pks.xtremio/csi/csi-xtremio:pks

csi_attacher_path=harbor.pks.xtremio/csi/csi-attacher:v1.0.1

csi_cluster_driver_registrar_path=harbor.pks.xtremio/csi/csi-cluster-driver-registrar:v1.0.1

csi_node_driver_registrar_path=harbor.pks.xtremio/csi/csi-node-driver-registrar:v1.0

csi_provisioner_path=harbor.pks.xtremio/csi/csi-provisioner:v1.0.1

csi_snapshotter_path=harbor.pks.xtremio/csi/csi-snapshotter:V1

storage_class_name=csi-xtremio-sc

list_of_clusters=xbrick1100

list_of_initiators=

csi_qos_policy_id=

csi_high_qos_policy=15m

csi_medium_qos_policy=5m

i l li 1




Generating Dell EMC XtremIO CSI Plugin YAML

Unlike native Kubernetes deployment, in PKS the CSI plugin is provided as Custom Workloads. Custom Workloads define what is included in a cluster out-of-the-box, and can be attached to a PKS plan that defines a set of resource types that are used for deploying clusters.

PKS uses BOSH to install the XtremIO CSI plugin for each new Kubernetes environment deployed by PKS from that specific plan, which includes the CSI Plugin YAML file.

To generate Dell EMC XtremIO CSI Plugin YAML: 1. Navigate to the template directory, and update the secret-template.yaml with the XMS CSI user’s password.

CSI Secret YAML File

2. After specifying the environment parameters, run the install-csi-plugin.sh script with -g parameter to generate the plugin YAML file. This script also configures the CSI user and the CSI QoS policies on the XMS.

Upon completion, the YAML file is in the current location, under the csi-tmp directory.

3. Using a text editor, open the plugin.yaml file and, at the beginning of the file, add the secret content that you edited in the previous step.

Make sure to add the ‘---’ to separate the sections.

4. Navigate to the Ops manager website and click Enterprise PKS, as shown in Figure 6.

Ops Manager Web Interface

apiVersion: v1

kind: Secret

metadata:

name: csixtremio

namespace: kube-system

stringData:

password: "#XMS PASSWORD#"

$ ./install-csi-plugin.sh -g



5. Select Active to activate the plan and to make it available to developers deploying clusters.

6. Enter a unique name and description in the respective fields, as shown in Figure 7.

PKS Configuration Plan

7. At the bottom of the page, under (Optional) Add-ons - Use with caution, enter the additional YAML configuration file that you generated earlier, including the Kubernetes secret, as shown in Figure 8.

This enables PKS to add custom workloads to each cluster in this plan.

8. Select Allow Privileged to enable the CSI plugin containers to access physical devices, as shown in Figure 8.

9. Click Save.

PKS Configuration Plan – Add-ons Sections



Deploying Mandatory BOSH Add-Ons

This section explains how to deploy BOSH Add-ons as a part of installing the Pivotal Cloud Foundry (PCF) platform. BOSH Add-ons are optional platform extensions that you deploy with BOSH, rather than installing them as Ops Manager tiles. Pivotal recommends installing BOSH Add-ons immediately after deploying the BOSH and Ops Manager, prior to using Ops Manager to add tiles to the platform.

To integrate PKS with the XtremIO CSI Plugin, it is necessary to install the iSCSI add-on, located at: https://github.com/svrc-pivotal/iscsi-release

Clone the latest version of the iSCSI-release bundle from the svrc-pivotal GitHub Repository to BOSH.

Edit the applying.sh script according to the instructions in the README file, and then run it on your BOSH machine.

This script installs the necessary packages to support the CSI plugin on every Kubernetes cluster deployed by PKS.

Managing PKS with the PKS CLI

To manage PKS-deployed clusters, use the PKS Command Line Interface (CLI). When you log into PKS successfully for the first time, the PKS CLI generates a local creds.yml file containing the API endpoint, refresh token, access token and CA certificate (if applicable). By default, creds.yml is saved to the ~/.pks directory on your local system. You can use the PKS_HOME environment variable to override this location, and store creds.yml in any directory on your system.

You will use PKS to get credential information from the deployed Kubernetes cluster, and set the context for kubectl. The context associates cluster and authentication information to a context name and makes that name the current context.

To Manage PKS with PKS CLI: 1. Install PKS CLI on your local system. See Installing the PKS CLI.

2. See Configuring PKS API Access for a username and password with access to the PKS API.

3. Run the following command in the CLI to authenticate to the PKS API:

4. Run the following command to create a new Kubernetes cluster. Make sure you specify the correct plan name (‘XtremIO’, as shown in the example):

pks login -a #PKS UAA ip/fqdn# -u #username# -p #password#

pks create-cluster #cluster name# --external-hostname #cluster ip/fqdn# --plan XtremIO

https://github.com/svrc-pivotal/iscsi-release


https://docs.pivotal.io/runtimes/pks/1-3/installing-pks-cli.html

https://docs.pivotal.io/runtimes/pks/1-3/configure-api.html



5. Run the following command to monitor the cluster status:

‘pks cluster’ Command Output

Once completed, the Last Action Description changes to Instance provisioning completed.

6. Create a DNS record with the Kubernetes Master Host and its IP address and make sure you can ping it.

‘pks cluster’ Command Output

7. Run the pks get-credentials command to perform the following actions:

o Fetch the cluster’s kubeconfig.

o Add the cluster’s kubeconfig to the existing kubeconfig.

o Create a new kubeconfig (if non-existent).

o Switch the context to the CLUSTER-NAME provided.

‘pks get-credentials’ Command Output

pks cluster #cluster name#

pks get-credentials CLUSTER-NAME



8. Run the following command and verify that all nodes are up and running:

'kubectl get nodes’ Showing All Kubernetes Nodes

During the PKS initial configuration, PKS uses BOSH to deploy the Kubernetes master and node virtual machines on the resources specified in advance for each new Kubernetes cluster.

PKS Virtual Machines

The XtremIO CSI plugin is installed on each Kubernetes virtual machine, as well as the necessary iSCSI and multipath drivers to support this configuration during the deployment.

An iSCSI initiator group is created by the XMS for each PKS Kubernetes node. This initiator group is used by the CSI plugin for mapping volumes to the PKS Kubernetes nodes during static and dynamic provisioning operations, on Kubernetes volumes.

The XMS Showing Automatically Created PKS Kubernetes Node Initiator Groups

kubectl get nodes



9. Run the following command and verify that the CSI plugin deployment is completed successfully:

‘kubectl get pods –all-namespaces’ Showing All Kubernetes Pods



Connecting to the Kubernetes Dashboard

The easiest way to access the Kubernetes dashboard is via kube proxy. We will run the proxy command on RDSH in PowerShell. This redirects all traffic destined for 127.0.0.1:{port_number} on this server, to the Kubernetes Master Node.

To connect to the Kubernetes dashboard: 1. Run the following command in order to run kube proxy:

Running the above command binds the Kubernetes dashboard to an available port on localhost. In our case, this turned out to be 8001.

‘kubectl proxy’ Command Output

2. In your web browser, navigate to: 127.0.0.1:{port_number}/ui

The screen shown in Figure 17 opens.

Kubernetes enables authentication to the UI by either using the Kubeconfig file or Token.

For authentication in this lab, we select Kubeconfig, as shown in Figure 17.

The Kubernetes Dashboard Login Page

3. Navigate to C:\Users\<username>\.kube and select the config file.

Note: You must use your username to locate your Kubeconfig file.

kubectl proxy



Health Monitoring and Troubleshooting

PKS leverages BOSH to provision infrastructure. BOSH receives periodic heartbeats from the nodes or VMs it has provisioned and ensures that all the required processes on these nodes are running. If BOSH detects a failure, it repairs the nodes in the background, using various methods. For example, if a node or VM is not responding, BOSH detaches its persistent disk, spins up a new VM, attaches the persistent disk to this new VM, and then starts the required processes on this node. All this is performed in the background and the issue is resolved without user interference.

Figure 18 shows all current BOSH tasks.

‘bosh tasks’ Command Output

Figure 19 shows all BOSH managed virtual machines.

‘bosh vms’ Command Output

https://bosh.io/docs/



Figure 20 shows all BOSH deployments.

‘bosh deployments’ Command Output

PKS Administrators can collect logs for each Kubernetes cluster or for the individual nodes, for advanced debugging.

The command shown in Figure 21 collects logs from a Kubernetes cluster into a single tarball file.

Generating BOSH Log Bundle



Deploying Persistent Volumes and Applications

Although it is relatively easy to run stateless microservices using container technology, stateful applications require slightly different treatment. There are multiple factors which need to be considered when handling persistent data using containers. Such considerations include:

• Kubernetes pods are ephemeral by nature, so the data that needs to remain persistent must survive through the restart and/or rescheduling of a pod.

• When pods are rescheduled, they can die on one host, and might get scheduled on a different host. In such cases, the storage should also be shifted and made available on the new host, for the pod to start gracef`ully.

• The application should not have to be concerned about the volume and data. The underlying infrastructure should handle the complexity of unmounting and mounting.

• Certain applications have a strong sense of identity (such as Kafka and Elastic), and the disk used by a container with a certain identity is tied to it. It is important that if for any reason a pod with a certain ID gets rescheduled, the disk associated with that ID is reattached to the new pod instance.

The following section demonstrates the successful deployment of PostgreSQL, using the XtremIO CSI driver with PKS.

In the example we will look at, PKS leverages Dell EMC XtremIO Storage for Kubernetes to enable pods to use enterprise-grade persistent storage. Before beginning with provisioning storage, it is important to understand the Kubernetes concept of a persistent volume and a persistent volume claim, and how they work together in a cluster. You can find detailed information about Kubernetes storage building blocks in the Dell EMC XtremIO CSI White Paper.

To demonstrate the capabilities of managing Kubernetes persistent storage applications using the XtremIO CSI plugin with Pivotal PKS, we deployed a Postgres multi-node cluster on top of a Pivotal PKS Kubernetes cluster, that is connected to a single XtremIO X2 cluster type X-Brick.

In the XtremIO plan we created in PKS, we chose to deploy a three-node cluster with the ‘2xlarge.cpu’ type in order to eliminate any bottleneck at the virtual machine level, and to highlight the great performance of the XtremIO X2 cluster type storage array.

Worker Node Configuration in Pivotal PKS XtremIO Plan



Example Use Case: PostgreSQL

PostgreSQL is a general purpose and object-relational database management system. It is one of the widely-used open-source database systems.

PostgreSQL was designed to run on UNIX-like platforms. However, PostgreSQL was also designed to be portable, so that it could run on various platforms such as Mac OS X, Solaris and Windows. PostgreSQL is designed to be extensible. In PostgreSQL, a user can define their own data types, index types, functional languages, etc.

Before creating the application, we create the Kubernetes ConfigMap and Service, by running the following kubectl commands.

Postgres Config Map YAML File

kubectl create -f config.yaml

Postgres Config Map

apiVersion: v1

kind: ConfigMap

metadata:

name: postgres-config-demo

labels:

app: postgres

data:

POSTGRES_DB:demodb

POSTGRES_USER: admin

POSTGRES_PASSWORD: Password123!



Postgres Service YAML File

The following example shows that we set the Storage Class Name to ‘csi-xtremio-high’, which is generated automatically during the CSI Plugin installation. This feature enables the user to automatically assign QoS policies at the XMS level to Kubernetes volumes, to guarantee service for critical applications and prevent a noisy neighbor. You can find detailed information about Kubernetes QoS in the Dell EMC XtremIO CSI White Paper.

For each new pod, we request a 30 GB volume with ReadWriteOnce accessModes.

When creating our first volume, the XtremIO CSI plugin creates the iSCSI Initiator Groups at the XMS level, and runs the iSCSI login, discovery, and multipath configurations for each node.

kubectl create -f service.yaml

Postgres Service

apiVersion: v1

kind: Service

metadata:

name: postgres

labels:

app: postgres

spec:

ports:

- port: 5432

name: postgres

clusterIP: None

selector:

app: postgres



To create the Postgres DB, we run the following command, which deploys the Postgres stateful set below:

Postgres Stateful Set YAML File

kubectl create -f postgres.yaml

Postgres Stateful Set

apiVersion: apps/v1

kind: StatefulSet

metadata:

name: postgres-demo

spec:

serviceName: "postgres"

replicas: 3

selector:

matchLabels:

app: postgres

template:

metadata:

labels:

app: postgres

spec:

containers:

- name: postgres

image: postgres:latest

envFrom:

- configMapRef:

name: postgres-config-demo

ports:

- containerPort: 5432

name: postgredb

volumeMounts:

- name: postgredb

mountPath: /var/lib/postgresql/data

subPath: postgres

volumeClaimTemplates:

- metadata:

name: postgredb

spec:

accessModes: [ "ReadWriteOnce" ]

storageClassName: csi-xtremio-high

resources:

requests:

storage: 30Gi



The following steps verify that all Postgres pods are up, by running the following kubectl command:

‘kubectl get pods’ Command Showing the Postgres Pods

Verify that the PersistentVolumeClaims were created for all Postgres pods, by running the following kubectl command:

‘kubectl get pods’ Command Showing the Postgres Persistent Volume Claims

Verify that the PersistentVolumes were created, formatted and mounted on all Postgres pods, by running the following kubectl command:

‘kubectl get pods’ Command Showing the Postgres Persistent Volumes

At this point, all volumes were automatically created at the XtremIO X2 cluster level, mapped to the relevant Kubernetes worker nodes, formatted with a filesystem, and mounted onto the relevant Postgres application pod.

By navigating to the XMS, we can see the dynamically created volumes by the CSI plugin and their characteristics.

The XtremIO CSI plugin automatically assigned the desired QoS Policy (QoS_High) to the Postgres pod’s Persistent Volumes:

The XMS Showing the Dynamically Created Volumes via a Kubernetes PersistentVolume Claim

In order to demonstrate XtremIO’s unique performance capabilities, we run a simple benchmark tool called pgbench. Running pgbench runs the same sequence of SQL commands over and over, in multiple concurrent database sessions against our running PostgreSQL cluster, which resides on multiple XtremIO X2 LUNs. Each command represents a Persistent Volume, which is bound to a single container.

More information about pgbench can be found here.

https://www.postgresql.org/docs/10/pgbench.html



Figure 30 shows the XMS Dashboard view of a Postgres DB workload. The Postgres DB workload drives about 770 MB/s throughput in sub-millisecond latency, while the workload is distributed across the Postgres pods. Each Kubernetes pod has its dedicated persistent volume.

The XMS – Postgres Workload Performance

Conclusion

Containers present significant benefits for supporting cloud-native apps and the transformation to micro-services, as they continue to grow in popularity and adoption. Several important challenges associated with managing containers have been addressed by popular orchestration solutions, such as Kubernetes. However, deploying production-ready Kubernetes clusters faces additional issues. The most noteworthy challenge is determining the right persistent storage solution for containers.

The integration between Pivotal PKS and Dell EMC XtremIO brings together the benefits of PKS and the unmatched performance and data services of XtremIO to the containers world. PKS is highly available up and down the stack, from apps to infrastructure. The service also includes dynamic load balancing, zero downtime updates, automated health checks, and much more. With PKS, you can deploy, scale, patch and upgrade all the Kubernetes clusters in your system—without downtime— and rapidly apply security fixes whenever new vulnerabilities are detected. Combining Dell EMC XtremIO CSI Plugin with PKS enables customers to deliver container-based applications on-premises, for both development and production scale. XtremIO CSI Plugin support for Kubernetes enables container-based applications to be deployed, managed and scaled. With Dell EMC XtremIO X2’s unmatched levels of performance, availability and TCO, organizations can now achieve the same levels of business agility offered by the public cloud, with the benefits of running them on-premises. The solution presented in this white paper is fully engineered and pre-tested to ensure consistency across multiple platform deployments, and reliable operations from day one, through its entire life cycle of upgrades and enhancements. Container’s high-availability feature keeps your most critical cloud-native applications running twenty-four hours a day, seven days a week. Through built-in health monitoring and orchestrated self-healing, this solution eliminates the risk of application downtime and the costly damage it could inflict on revenue streams and customer confidence.



References • Dell EMC XtremIO CSI White Paper:

– https://www.dellemc.com/resources/en-us/asset/white-papers/solutions/white-paper_xtremio-x2_interface_plugin_for_kubernetes_h17583.pdf

• VMware Enterprise PKS Documentation:

– https://docs.vmware.com/en/VMware-Enterprise-PKS/index.html

• Dell EMC XtremIO CSI Docker Repository:

– https://hub.docker.com/r/dellemcstorage/csi-xtremio

• Dell EMC XtremIO CSI Deployment Script:

– https://github.com/dell/csi-xtremio-deploy

• Harbor Cloud Native Registry:

– https://goharbor.io/

• Cloud Foundry BOSH:

– https://bosh.io/docs/

• ISCSI BOSH Add-On:

– https://github.com/svrc-pivotal/iscsi-release

• Dell EMC XtremIO Main Page:

– https://github.com/svrc-pivotal/iscsi-release

• Dell EMC XtremIO X2 Datasheet:

– http://www.emc.com/collateral/data-sheet/h16095-xtremio-x2-next-generation-all-flash-array-ds.pdf

• Introduction to Dell EMC XtremIO X2 Storage Array: – https://www.emc.com/collateral/white-papers/h16444-introduction-xtremio-x2-storage-array-wp.pdf

• CSI Spec:

– https://github.com/container-storage-interface/spec/blob/master/spec.md

• CSI GitHub Repository:

– https://kubernetes-csi.github.io/docs/Home.html

• Kubernetes Documentation site:

– https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

• CSI Explanation: – https://blog.architecting.it/container-storage-interface/

https://www.dellemc.com/resources/en-us/asset/white-papers/solutions/white-paper_xtremio-x2_interface_plugin_for_kubernetes_h17583.pdf

https://www.dellemc.com/resources/en-us/asset/white-papers/solutions/white-paper_xtremio-x2_interface_plugin_for_kubernetes_h17583.pdf

https://docs.vmware.com/en/VMware-Enterprise-PKS/index.html

https://hub.docker.com/r/dellemcstorage/csi-xtremio

https://github.com/dell/csi-xtremio-deploy


https://bosh.io/docs/



http://www.emc.com/collateral/data-sheet/h16095-xtremio-x2-next-generation-all-flash-array-ds.pdf

https://www.emc.com/collateral/white-papers/h16444-introduction-xtremio-x2-storage-array-wp.pdf

https://github.com/container-storage-interface/spec/blob/master/spec.md

https://kubernetes-csi.github.io/docs/Home.html

https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

https://blog.architecting.it/container-storage-interface/



How to Learn More For a detailed presentation explaining XtremIO X2 Storage Array's capabilities and how XtremIO substantially improves performance, operational efficiency, ease-of-use and total cost of ownership, please contact XtremIO, at [email protected].

We will schedule a private briefing, in person or via a web meeting. XtremIO X2 provides benefits in many environments and mixed workload consolidations, including virtual server, cloud, virtual desktop, database, analytics and business applications.

Learn more about Dell EMC XtremIO

Contact a Dell EMC Expert

View more resources

Join the conversation @DellEMCStorage and

#XtremIO

© 2019 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners. Reference Number H17839

mailto:[email protected]

https://store.emc.com/en-us/Product-Family/EMC-XtremIO-Products/Dell-EMC-XtremIO-All-Flash-Array/p/EMC-XtremIO-Flash-Scale-Out

https://store.emc.com/en-us/Product-Family/EMC-XtremIO-Products/Dell-EMC-XtremIO-All-Flash-Array/p/EMC-XtremIO-Flash-Scale-Out

mailto:[email protected]?subject=XtremIO%20inquiry



https://www.dellemc.com/en-us/storage/xtremio-all-flash.htm



Documents

Dell EMC XtremIO X2 CSI Plugin Integration with Pivotal PKS · 2019-08-06 · Pivotal Container Service (PKS) is a purpose-built product that enables enterprises to deploy and consume