Executive SummaryOver the last few years, containerization technology has matured greatly, helping enterprises to increase business agility. Four out of five enterprises are now running container technologies. In 2017, only 67 percent of enterprises were running containers in a production environment—that number jumped to 83 percent in 2018.1

In this paper, we explore the solution architecture of the Red Hat OpenShift* Container Platform. This end-to-end container platform has been optimized to take advantage of the latest Intel® technology, including 2nd Generation Intel® Xeon® Scalable processors, Intel® Solid State Drives, and Intel® Optane™ DC persistent memory. Red Hat and Intel collaborated to create this reference architecture, which can help enterprises to take the next step toward unleashing the power of containers in a hybrid cloud environment.

RegistryRed HatEnterprise Linux*

Node

Pod

Pod Pod

Pod

Red HatEnterprise Linux

Node

Pod

Pod Pod

Pod

Red HatEnterprise Linux

Node

Pod

Pod Pod

Pod

2nd Gen Intel® Xeon®Scalable Processor

Intel® Optane™ DCPersistent Memory

Intel® Optane™ DCSolid State Drives

Figure 1. Red Hat OpenShift* Container Platform solution architecture.

Use this reference architecture to accelerate your path to a hybrid cloud, enterprise-grade container platform optimized for Intel® technology

Deploying Red Hat OpenShift* Container Platform v3.11

RefeRence ARchitectuReData CenterContainers

Reference Architecture | Deploying Red Hat OpenShift* Container Platform v3.11 2

IntroductionToday’s hybrid cloud environments demand a high-performance infrastructure that is flexible and reliable. Intel’s latest family of Intel® Xeon® processors represents a significant step toward data center modernization. Multiple CPU and chipset options with increased core counts, integrated accelerators, and network interfaces make important component choices more critical than ever. Combining the new Intel Xeon Scalable processor-based technologies with advances in data storage technologies provides significant leaps in the solution price-performance. This remarkable generation of storage technology helps remove storage bottlenecks, achieving higher CPU utilization. Countering this performance trend, many ISV’s scale-out architectures can force the performance bottleneck to the network. Several Intel® Ethernet solutions are available: 10, 25, 40, and 100 GbE options can be deployed where necessary to provide balanced system performance that scales well.

Container technology, such as Docker* and Kubernetes* is growing ever more popular, because containers solve a host of data center problems such as portability, repeatability, and scalability. But building a container platform from the ground up can be daunting. Red Hat OpenShift* Container Platform helps simplify this process by providing an end-to-end container solution that is optimized for Intel® technologies. The reference architecture detailed in this document can guide you through the often-complex hardware and software

selection process when setting up a container platform. We provide details for three configurations:

• Small Cluster. With three nodes, this configuration is appropriate for proofs of concept (PoC).

• Medium Cluster. With six nodes, this configuration is suitable for small production environments.

• Large Cluster. With 15 nodes, this configuration is designed for high-availability production environments.

Hardware ComponentsThe reference architecture for Red Hat OpenShift Container Platform consists of various Intel Xeon processor-based technologies and optimized software and firmware configurations. The following sections provide details about each hardware component.

2nd Gen Intel® Xeon® Scalable Processor Family

With more cores and more memory channels than the previous generation, 2nd generation Intel Xeon Scalable processors are architected and optimized for high-performance computing (HPC) simulation and modeling applications, artificial intelligence (AI) usages, and demanding infrastructure-as-a-service (IaaS) workloads. This family of processors is characterized by pervasive performance, hardware-enhanced security features, and agility and efficiency for improved cost efficiency.

Table of Contents

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2

Hardware Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22nd Gen Intel® Xeon® Scalable Processor Family . . . . . . . 2Intel® Server Board S2600WF Family . . . . . . . . . . . . . . . . . . 3Intel® Solid State Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Intel® Ethernet Network Adapters . . . . . . . . . . . . . . . . . . . . . 3Intel® Optane™ DC Persistent Memory . . . . . . . . . . . . . . . . . 3

Software Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4Red Hat Enterprise Linux* (RHEL*) . . . . . . . . . . . . . . . . . . . . . 4Red Hat Hyperconverged Infrastructure for Virtualization* (RHHI-V*) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4Red Hat OpenShift Container Platform . . . . . . . . . . . . . . . . 4Red Hat OpenShift Container Storage . . . . . . . . . . . . . . . . . 4

Intel® Reference Architecture for Red Hat OpenShift* Container Platform . . . . . . . . . . . . . . .5

Small Cluster Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Medium Cluster Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 6Large Cluster Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Hardware Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . .8Optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Network Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Firmware Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8BIOS Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Operating System Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Software Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9RHHI-V Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9VM Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

OpenShift Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9Inventory File for Medium Cluster Configuration . . . . . . . 9

Intel® Optane™ DC Persistent Memory Module Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Topology Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . 10Memory Mode Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 11App Direct Mode Configuration . . . . . . . . . . . . . . . . . . . . . . 11Using Apache Spark* with Intel® Optane™ DC Persistent Memory in Memory Mode . . . . . . . . . . . . . . . . . 12

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Reference Architecture | Deploying Red Hat OpenShift* Container Platform v3.11 3

This processor family includes technologies for accelerating and securing specific workloads:

• Intel® Advanced Vector Extensions 512 (Intel® AVX-512) expands the number of registers and adds instructions to streamline processing. The result is higher performance of mixed workloads. This technology has been enhanced with the addition of Intel® Deep Learning Boost (Intel® DL Boost), which is an embedded AI acceleration technology. This technology uses new Vector Neural Network instructions (VNNI) that can accelerate deep-learning inference.

• Intel® QuickAssist Technology (Intel® QAT) can accelerate compression and cryptographic workloads using a chipset-based hardware acceleration technology. This results in high levels of efficiency while delivering enhanced data transport and protection across server, storage, and network infrastructure.

• Intel® Trusted Execution Technology (Intel® TXT) provides the necessary underpinnings to evaluate the computing platform and its security.

• Distributed Hardware Security Module (DHSM) with Intel® Key Protection Technology (Intel® KPT). Working with Intel QAT, the DHSM helps secure keys in hardware.

• Intel® Platform Trust Technology (Intel® PTT) provides secure encryption keys in a Trusted Platform Module (TPM) integrated directly into the chipset.

This reference design for small, medium, and large configurations are based on the Intel® Xeon® Gold processors, which provide workload-optimized performance, power efficiency, and advanced reliability.

Intel® Server Board S2600WF Family

The Intel® Server Board S2600WF family delivers power and performance at peak efficiency in 1U and 2U rack mount server form factors that feature energy-efficient dual Intel Xeon Scalable processors. High memory capacity, networking, storage, and I/O flexibility combine with innovative design to provide an exceptional and reliable server for business IT, appliance, data center, cloud, and HPC applications.

Intel® Server System R1208WFTYS is a 1U rack server based on Intel Server Board S2600WFT, which is a dual-processor board optimized for a cloud/data center market. It is compatible with Intel Xeon Scalable processors with support of up to 24 DIMMs, eight 2.5-inch hot-swap front drives, two M2.SSD internal drives, and two 10 GbE ports. It includes the Intel® OCP Module for additional features without losing a PCIe* add-in slot and may be configured with a redundant power supply.

Intel® Solid State Drives

The Intel® SSD D3-S4510 Series is a SATA-based solid state drive optimized for read-intensive workloads. Based on tri-level cell (TLC) Intel® 3D NAND technology, these larger-capacity SSDs enable data centers to increase data stored per rack unit. The Intel SSD D3-S4510 Series is built for compatibility with legacy infrastructures, enabling easy storage upgrades that minimize the costs associated with modernizing data centers. This 2.5-inch 7 mm form factor offers a wide range of capacities from 240 GB up to 3.8 TB.

The Intel SSD DC P4510 is packed with a deep feature set. This Intel 3D NAND SSD for data centers is optimized for the data caching needs of cloud storage and software-defined infrastructures. It modernizes the data center with a combination of performance, capacity, manageability, and reliability. The Intel SSD DC P4510 can significantly increase server agility and utilization—while also accelerating applications—across a wide range of cloud workloads. This PCI Express* (PCIe*) Non-volatile Memory Express* (NVMe*) SSD series comes in two configuration options with a 4 TB capacity.

Intel® Ethernet Network Adapters

Intel® Ethernet Network Adapter XXV710 can deliver excellent performance with a theoretical throughput of 50 Gb/s (25 Gb/s single-port bi-directional), in a PCIe v3.0 x8 slot. The Intel Ethernet Network Adapter XXV710 is based on an innovative new architecture, with the ability to auto negotiate for 1/10/25 GbE connections, and is designed to meet the needs of customers who have multiple speeds deployed in their environment.

Intel® Optane™ DC Persistent Memory

Intel® Optane™ DC persistent memory is an additional level of memory to bridge the gap between DRAM and NAND SSD in DIMM form factor. Intel Optane DC persistent memory offers large, affordable memory capacity and native persistence that can maintain a working dataset through power cycles. This unique technology creates opportunity for amazing operational efficiencies. Intel Optane DC persistent memory is comparable in speed to DRAM, and is less expensive per GB. Beyond its persistence, performance, capacity, and affordability, some of the other benefits of Intel Optane DC persistent memory include:• DDR4 pin compatible• Hardware encryption• High reliability

Reference Architecture | Deploying Red Hat OpenShift* Container Platform v3.11 4

This reference architecture introduces Intel Optane DC persistent memory in Red Hat OpenShift Container Platform in Memory Mode. For additional details for App Direct Mode, see “App Direct Mode Configuration” and “Using Apache Spark* with Intel® Optane™ DC Persistent Memory in Memory Mode” for more details.

Software Components

Red Hat Enterprise Linux* (RHEL*)This solution uses RHEL* v7.6 as the OS. The RHEL OS helps ensure high quality and a reliable base for the whole system. It provides strong security features and supports business-critical workloads, interoperability between a variety of operating systems, and much more.

Red Hat Hyperconverged Infrastructure for Virtualization* (RHHI-V*)RHHI-V* is a solution that includes the Red Hat Virtualization (RHV) platform and integrates Red Hat Gluster Storage* (RHGS*). This open source solution provides a compact combination of compute, storage, networking, and management facilities in a single deployment. The Red Hat Virtualization Host (RHVH) is used as a central administration host for further configuration.

Red Hat OpenShift Container PlatformThe OpenShift Container Platform is a complete container application platform that provides all aspects of the application development and deployment process in one consistent solution across multiple infrastructure footprints. OpenShift integrates the architecture, processes, platforms, and services needed to help development and operations teams traverse traditional siloed structures and produce applications that help businesses succeed.

The Red Hat OpenShift Container Platform is managed by the Kubernetes* container orchestrator, which manages containerized applications across a cluster of systems running the Docker* container runtime environment. OpenShift Container Platform contains the Kubernetes container orchestration and scheduler software.

Red Hat OpenShift Container StorageOpenShift Container Storage can make the OpenShift Container Platform a fully hyperconverged infrastructure (HCI) where storage containers co-reside with the compute containers. The storage plane is based on containerized RHGS services, which control storage devices on every server. Heketi* is a part of the OpenShift Container Storage architecture and it controls all the nodes that are members of the storage cluster.

RHGS services use Heketi to manage GlusterFS-based volumes. Heketi exposes its features via the RESTful API.

Understanding Memory Operating ModesIntel® Optane™ DC persistent memory has three different operating modes that determine which capabilities of the Intel Optane memory are active and available to software.

Memory Mode

Applications and the OS perceive a pool of volatile memory the same as on traditional DRAM-only systems. In this mode, no specific persistent memory programming is required in the applications, and the data will not be saved in the event of a power loss. In Memory Mode, the DRAM acts as a cache for the most frequently-accessed data, while the Intel Optane DC persistent memory provides large memory capacity. Memory Mode seamlessly brings large memory capacity at an affordable cost to legacy applications. Virtualized database deployments and big-data analytics applications are great candidates for Memory Mode.

App Direct Mode

Applications and the OS are explicitly aware that there are two types of direct load/store memory in the platform, and can direct which type of data read or write is suitable for either DRAM or Intel Optane DC persistent memory. Operations that require the lowest latency and don’t need permanent data storage can be executed on DRAM, such as database “scratch pads.” Data that needs to be made persistent or structures that are very large can be routed to the Intel Optane DC persistent memory. In-memory databases, in-memory analytics frameworks, and ultrafast storage applications are good examples of workloads that greatly benefit from using App Direct Mode.

Storage Over App Direct Mode

Intel Optane DC persistent memory is configured as accessible block storage devices residing on the memory bus. In this configuration, the technology operates as conventional block storage. As with Memory Mode, no modification to the OS or applications is required.

Reference Architecture | Deploying Red Hat OpenShift* Container Platform v3.11 5

Table 1. Small Cluster Configuration. This cluster configuration is appropriate for PoC. It consists of three hyperconverged nodes.

INGREDIENT (Per Node) SMALL CLUSTER CONFIGURATION

3x RHHI-V* NodesProcessor 2x Intel® Xeon® Gold 6230 Processor at 2.1 GHzMemory 384 GB (12x32 GB)Intel® Optane™ DC Persistent Memory 512 GB (4x128 GB)Boot Drive 2x Intel® SSD D3-S4510 Series 480 GB 2.5-inch RAID1RHHI-V Storage 1x Intel® SSD DC P4510 Series 2 TBOpenShift* Container Storage 1x Intel® SSD D3-S4510 Series 2 TB 2.5-inchStorage HBA Controllera Intel® RAID Module RMSP3HD080ERemote Management Modulea Intel® Remote Management Module 4 Lite 2 AXXRMM4LITE2Network 1x Intel® Ethernet Network Adapter XXV710-DA2, Dual-port 25 Gbps SFP28

a Recommended but not required.

Small Cluster Architecture

This architecture is designed for PoC and development environments. Three hyperconverged nodes comprise the cluster (see Figure 3). OpenShift instances for Master, Infrastructure, and Compute are Red Hat Hyperconverged Infrastructure for Virtualization (RHHI-V) VMs.

Intel® SSD DC P4510 2 TB drives are used for RHHI-V storage (engine, VMs, and data). To provide a large data pool for container-native storage (images and container persistent data), Intel® SSD D3-S4510 2 TB drives are used. Table 1 details the reference architecture for this three-node design.

Bastion VM Master VM Infrastructure VM Application VM

Master VM Infrastructure VM Application VM

3x RHHI-V* NodesWeb Traffic Load Balancer VM Master VM Infrastructure VM Application VM

Small Cluster Architecture

Figure 3. Designed for PoC and development environments. The small cluster architecture consists of three hyperconverged nodes.

Figure 2. This reference architecture is flexible, allowing you to build small, medium, or large clusters, depending on your needs.

Bastion VM Master VM Infrastructure VM Application VM

Master VM Infrastructure VM Application VM

3x RHHI-V* ClusterWeb Traffic Load Balancer VM Master VM Infrastructure VM Application VM

SMALL CLUSTER ARCHITECTURE

+

+

3x Bare Metal OpenShift* Application Nodes

12x Bare Metal OpenShift Application Nodes

MEDIUM CLUSTER ARCHITECTURE

LARGE CLUSTER ARCHITECTURE

Application VMs only used in small configuration

OR

Heketi also provides an API through which storage space for containers can be easily requested. While Heketi provides an endpoint for a storage cluster, the object that makes calls to its API from OpenShift clients is called a Storage Class. Storage Classes implement VolumePlugin functionalities by defining and managing the type of storage available for the cluster and can dynamically send storage requests when a persistent volume claim is generated.

Intel® Reference Architecture for Red Hat OpenShift* Container PlatformThe following section describes the required and recommended components needed to build small, medium, and large cluster architectures (see Figure 2).

Reference Architecture | Deploying Red Hat OpenShift* Container Platform v3.11 6

Medium Cluster Architecture

This architecture is designed for small production environments. It consists of six nodes—three hyperconverged infrastructure nodes and three application nodes (see Figure 4). Three OpenShift Application (compute) nodes are physical machines without a VM-hypervisor. Intel Optane DC persistent memory modules extend physical memory.

An additional Intel SSD DC P4510 2 TB drive provides fast storage for container persistent data, while an Intel SSD D3-S4510 4 TB drive provides storage backend for application images. The large cluster architecture provides 2.4x the capacity of the small cluster architecture. Table 2 details the reference architecture for this six-node design.

Table 2. Medium Cluster Configuration. This configuration is suitable for small production environments. It consists of six nodes (three hyperconverged infrastructure nodes and three bare-metal application nodes that are dedicated for running containerized applications).

INGREDIENT (Per Node) MEDIUM CLUSTER CONFIGURATION

3x RHHI-V* Nodes

Processor 2x Intel® Xeon® Gold 5218 Processor at 2.3 GHz

Memory 384 GB (12x32 GB)

Boot Drive 2x Intel® SSD D3-S4510 Series 480 GB 2.5-inch RAID1

RHHI-V Storage 1x Intel® SSD DC P4510 Series 2 TB

OpenShift* Container Storage (Registry) 1x Intel® SSD D3-S4510 Series 4 TB

Storage HBA Controllera Intel® RAID Module RMSP3HD080E

Remote Management Modulea Intel® Remote Management Module 4 Lite 2 AXXRMM4LITE2

Network 1x Intel® Ethernet Network Adapter XXV710-DA2, Dual-port 25 Gbps SFP28

3x OpenShift* Application Nodes

Processor 2x Intel® Xeon® Gold 6230 processor (2.1 GHz, 22 cores, 44 threads)

Memory 384 GB or higher (12x32 GB)

Intel® Optane™ DC Persistent Memory 1024 GB (4x256 GB)

Boot Drive 2x Intel® SSD D3-S4510 Series 480 GB 2.5-inch RAID1

OpenShift* Container Storage (Application) 1x Intel® SSD DC P4510 Series 4 TB (NVMe*)

Storage HBA Controllera Intel® RAID Module RMSP3HD080E

Remote Management Modulea Intel® Remote Management Module 4 Lite 2 AXXRMM4LITE2

Network 1x Intel® Ethernet Network Adapter XXV710-DA2, Dual-port 25 Gbps SFP28 a Recommended but not required.

Bastion VM Master VM Infrastructure VM

Master VM Infrastructure VM

3x RHHI-V* NodesWeb Traffic Load Balancer VM Master VM Infrastructure VM

+3x Bare Metal OpenShift* Application Node

Medium Cluster Architecture

Figure 4. Designed for small production environments. The medium cluster architecture consists of three hyperconverged infrastructure nodes and three application nodes.

Reference Architecture | Deploying Red Hat OpenShift* Container Platform v3.11 7

Large Cluster Architecture

This architecture is suitable for high-availability production environments. It consists of 15 nodes—three hyperconverged infrastructure nodes and 12 application nodes (see Figure 5).

The large configuration extends the medium configuration by increasing the number of physical application nodes from three to 12. The large cluster architecture provides 4x the capacity of the medium cluster architecture. Table 3 details the reference architecture for this 15-node design.

Bastion VM Master VM Infrastructure VM

Master VM Infrastructure VM

3x RHHI-V* NodesWeb Traffic Load Balancer VM Master VM Infrastructure VM

+12x

12x Bare Metal OpenShift* Application Node

Large Cluster Architecture

Figure 5. Designed for large production environments. Suitable for high-availability production environments, the large configuration consists three hyperconverged infrastructure nodes and 12 application nodes.

Table 3. Large Cluster Configuration. This configuration is suitable for high-availability production environments. It consists of 15 nodes (three hyperconverged infrastructure nodes and 12 application nodes).

INGREDIENT (Per Node) LARGE CLUSTER CONFIGURATION

3x RHHI-V* Nodes

Processor 2x Intel® Xeon® Gold 5218 Processor at 2.3 GHz

Memory 384 GB (12x32 GB)

Boot Drive 2x Intel® SSD D3-S4510 Series 480 GB 2.5-inch RAID1

RHHI-V Storage 1x Intel® SSD DC P4510 Series 4 TB

OpenShift* Container Storage (Registry) 1x Intel® SSD DC P4510 Series 4 TB

Storage HBA Controllera Intel® RAID Module RMSP3HD080E

Remote Management Modulea Intel® Remote Management Module 4 Lite 2 AXXRMM4LITE2

Network 1x Intel® Ethernet Network Adapter XXV710-DA2, Dual-port 25 Gbps SFP28

12x OpenShift* Application Nodes

Processor 2x Intel® Xeon® Gold 6230 processor (2.1 GHz, 22 cores, 44 threads)

Memory 384 GB or higher (12x32 GB)

Intel® Optane™ DC Persistent Memory 1024 GB (4x256 GB)

Boot Drive 2x Intel® SSD D3-S4510 Series 480 GB 2.5-inch RAID1

OpenShift* Container Storage (Application) 1x Intel® SSD DC P4510 Series 4 TB (NVMe*)

Storage HBA Controllera Intel® RAID Module RMSP3HD080E

Remote Management Modulea Intel® Remote Management Module 4 Lite 2 AXXRMM4LITE2

Network 1x Intel® Ethernet Network Adapter XXV710-DA2, Dual-port 25 Gbps SFP28 a Recommended but not required.

Reference Architecture | Deploying Red Hat OpenShift* Container Platform v3.11 8

Hardware ConfigurationsThe following are settings that apply to all nodes.

Optimizations

Table 4 provides the optimizations in the reference hardware used for all reference designs.

Table 4. Optimizations that Apply to All Nodes

OPTIMIZATION CONFIGURATION

Trusted Platform Module (TPM)

TPM 2.0 discrete or Firmware TPM—Intel® PTT

Intel® Technologies

Intel® HT Technology Enabled

Intel® Turbo Boost Technology Native

Intel® Speed Shift Technology HWP Native

Intel Turbo Boost Technology/HWP

EPP/ EPB settings balanced

Three-way Mirroring Least overhead on processing power (recommended but not required)

Patches Updated with all available patches (not required, but recommend applying pertinent Intel® technology errata frequently)

Network Configuration

Table 5 provides the network configurations in the reference hardware used for all reference designs.

Table 5. Network Configuration

HARDWARE + DESCRIPTION QUANTITYREQUIRED ORRECOMMENDED

Top of the Rack: 100 GbE SFP+ Switch

2 Recommended

Management (Out of Band): Integrated 1 GbE

1 Recommended

Software Requirements

Table 6 provides details about the software requirements for all reference designs.

Table 6. Software Requirements

SOFTWARE VERSION

Red Hat Enterprise Linux* 7.6

Red Hat OpenShift* Container Platform 3.11

Red Hat OpenShift Container Storage 3.11

RHHI-V* 1.6

Docker* 1.13.1

Etcd* 3.2.22

Open vSwitch* 2.9.0

HAProxy* 1.5.18

Firmware Settings

Table 7 provides details about minimal firmware requirements for hardware components.

Table 7. Firmware Versions

INGREDIENT VERSION

BIOS SE5C620.86B.0D.01.0338.011120192100

BMC 1.89.5697d1e5

ME 04.01.03.239

SDR 1.90

Intel® Optane™ DC Persistent Memory

01.00.00.5127

Intel® SSD D3-S4510 Series XCV10110

Intel® DC P4510 Series VDV10152

BIOS Settings

Table 8 provides information about system BIOS settings for all reference designs.

Table 8. BIOS Configurations

INGREDIENT SETTINGREQUIRED ORRECOMMENDED

Intel® Hyper Threading Technology

Enabled Required

C-States No Limit Recommended

PU Power and Performance Policy

Performance Required

CPWorkload Configuration Balanced Recommended

Operating System Settings

This reference design uses default settings defined by RHHI for Virtualization and Red Hat OpenShift installation.

Red Hat Subscription is mandatory for the deployment to succeed.

Table 9. Operating System Settings

SOLUTION SETTING

RHHI-V* Installation Default

Red Hat OpenShift* Installation Default

Red Hat Subscription Mandatory

Reference Architecture | Deploying Red Hat OpenShift* Container Platform v3.11 9

Software Configuration

RHHI-V Configuration

RHHI-V is deployed on three physical nodes. Storage for RHHI-V is configured on Intel SSD DC P4510 drives. The deployment process was executed according to the official documentation.

VM Configuration

To integrate the Red Hat OpenShift Container Platform with RHHI-V, the cluster must meet minimum resource requirements (see Table 10). Please follow the recommendations in the official documentation. In this reference architecture, the application nodes in the small configuration are set to the maximum of cluster capacity. In the medium and large configurations, master nodes have more memory than is recommended in the official recommendation to guarantee increased API performance. All virtual nodes have more CPU resources.

An Intel SSD D3-S4510 Series drive serves as the OS (boot) drive. RHHI-V Storage uses Intel SSD DC P4510 drives, which are Non-Volatile Memory Express* (NVMe*)-based to configure persistent volumes for containerized applications that require data persistence. NVMe-based drives accelerate computing operations because of high sequential as well as high random read and write capabilities. Equally low latency of these operations on these drives increases performance levels.

Infrastructure VMs use the SCSI to access Intel SSD D3-S4510 Series drives for direct access without the participation of a hypervisor component.

Table 10. VM Configuration Details

SMALL CLUSTERNODE ROLE COUNT CPU MEMORYMaster 3 4 cores 32 GBInfrastructure 3 8 cores 16 GBLoad Balancer 1 2 cores 16 GBBastion 1 2 cores 16 GBApplication 3 32 cores 330 GB

MEDIUM CLUSTERNODE ROLE COUNT CPU MEMORYMaster 3 4 cores 32 GBInfrastructure 3 8 cores 16 GBLoad Balancer 1 2 cores 16 GBBastion 1 2 cores 16 GB

LARGE CLUSTERNODE ROLE COUNT CPU MEMORYMaster 3 4 cores 32 GBInfrastructure 3 8 cores 16 GBLoad Balancer 1 2 cores 16 GBBastion 1 2 cores 16 GB

OpenShift Configuration

Inventory File for Medium Cluster Configuration

To configure nodes and install Red Hat OpenShift Container Platform for Red Hat Virtualization, follow the official instructions. The inventory file that is located in /etc/ansible/hosts is the environment’s brief description. The main sections of the inventory file are detailed below.

The OSEv3:children section is used to specify the OpenShift role for each node. Required groups of nodes are: nodes, masters, and etcd. Two optional groups are available:

• A load balancer (lb) group is for load balancing the master API on all master hosts. If the lb variable isn’t set, the user must configure external load balancing separately.

• The glusterfs _ registry is used for persistent data storage for containers.

[OSEv3:children]nodesmastersetcdlbglusterfs_registryglusterfs

The OSEv3:vars section includes all of the general cluster variables. Click here for a more detailed description for each variable.

[OSEv3:vars]ansible_ssh_user=rootopenshift_master_cluster_method=nativeopenshift_master_cluster_hostname=openshift.medium.localopenshift_master_cluster_public_hostname=openshift.example.comopenshift_master_default_subdomain=apps.openshift.example.comopenshift_deployment_type=openshift-enterpriseopenshift_release=’v3.11’openshift_master_identity_providers=[{‘name’: ‘htpasswd_auth’, ‘login’: ‘true’, ‘challenge’: ‘True’, ‘kind’:‘HTPasswdPasswordIdentityProvider’}]os_sdn_network_plugin_name=redhat/openshift-ovs-multitenantoreg_auth_user=’<serviceaccount_username>’oreg_auth_password=’<serviceaccount_password>’openshift_hosted_registry_storage_kind=glusterfsopenshift_storage_glusterfs_image= registry.redhat.io/rhgs3/rhgs-server-rhel7:v3.11openshift_storage_glusterfs_block_image= registry.redhat.io/rhgs3/rhgs-gluster-block-prov-rhel7:v3.11openshift_storage_glusterfs_heketi_image= registry.redhat.io/rhgs3/rhgs-volmanager-rhel7:v3.11

Reference Architecture | Deploying Red Hat OpenShift* Container Platform v3.11 10

The masters and etcd sections include servers acting as OpenShift master nodes (API, controller, scheduler, and web user interface). Master nodes host etcd key-value stores to achieve low-latency traffic between them, but also provide high availability.

[masters]master1.openshift.medium.localmaster2.openshift.medium.localmaster3.openshift.medium.local

[etcd]master1.openshift.medium.localmaster2.openshift.medium.localmaster3.openshift.medium.local

The infras section describes servers that perform an infrastructure role.

[infras]infra1.openshift.medium.local infra2.openshift.medium.localinfra3.openshift.medium.local

The glusterfs _ registry section represents servers hosting an OpenShift Container Storage backend. Gluster file system implementation is located on the infrastructure nodes. Moreover, the glusterfs _ devices parameter describes devices that comprise the storage pool.

[glusterfs_registry]infra1.openshift.medium.local glusterfs_devices=”[‘/dev/sda’]”infra2.openshift.medium.local glusterfs_devices=”[‘/dev/sda’]”infra3.openshift.medium.local glusterfs_devices=”[‘/dev/sda’]”

[glusterfs]app1.openshift.medium.local glusterfs_devices=”[‘/dev/nvme0n1’]”app2.openshift.medium.local glusterfs_devices=”[‘/dev/nvme0n1’]”app3.openshift.medium.local glusterfs_devices=”[‘/dev/nvme0n1’]”

If openshift _ master _ cluster _ method=native, then the lb section indicates a host on which a software load balancer will be installed. In this reference architecture, the host is one of the infrastructure nodes.

[lb]lb1.openshift.medium.local

The nodes section includes all of the servers that are included in the OpenShift cluster.

If required or if already available in the target deployment location, it is recommended that an enterprise external load balancer (F5, NGINX, or other) be leveraged as the ingress point for both the web console and OpenShift Router. This is done for edge request routing and load balancing for applications deployed on the OpenShift cluster. For details, see the “Multiple Masters” section of Red Hat’s “Example Inventory Files” documentation.

[nodes]master1.openshift.medium.local openshift_node_group_name= node-config-mastermaster2.openshift.medium.local openshift_node_group_name= node-config-mastermaster3.openshift.medium.local openshift_node_group_name= node-config-masterinfra1.openshift.medium.local openshift_node_group_name= node-config-infrainfra2.openshift.medium.local openshift_node_group_name= node-config-infrainfra3.openshift.medium.local openshift_node_group_name= node-config-infraapp1.openshift.medium.local openshift_node_group_name= node-config-computeapp2.openshift.medium.local openshift_node_group_name= node-config-computeapp3.openshift.medium.local openshift_node_group_name= node-config-compute

For the large configuration, this node list in extended by additional node-config-compute entries reflecting the total number of application nodes in the cluster.

Intel® Optane™ DC Persistent Memory Module ConfigurationRHEL provides a toolkit to manage devices using the command-line interface. Click here for full information about BIOS options and settings on Intel Xeon Scalable processors.

Topology Recommendations

This reference architecture uses a 2-1-1 memory configuration, meaning three DDR4 modules and one Intel Optane DC persistent memory module are used per CPU memory controller (see Figure 6).

Figure 6. Three DDR4 modules and one Intel® Optane™ DC persistent memory module per CPU memory controller provides fast DRAM availability on every memory channel and increases the likelihood of a memory cache hit.

2-1-1 Memory Configuration

Intel® Xeon®Scalable

Processor

IMC IMC

8 Slots per CPUTrade DDR bandwidth for smaller board real estate on the DIMM slots

Reference Architecture | Deploying Red Hat OpenShift* Container Platform v3.11 11

This topology provides fast DRAM availability on every memory channel, thereby increasing the likelihood of a memory cache hit. The following output represents medium and large cluster topologies for application nodes.

DimmID | MemoryType | Capacity |PhysicalID | DeviceLocator========================================================================== 0x0001 | Logical Non-Volatile Device | 252.4 GiB | 0x0020 | CPU1_DIMM_A2 0x0101 | Logical Non-Volatile Device | 252.4 GiB | 0x002c | CPU1_DIMM_D2 0x1001 | Logical Non-Volatile Device | 252.4 GiB | 0x0038 | CPU2_DIMM_A2 0x1101 | Logical Non-Volatile Device | 252.4 GiB | 0x0044 | CPU2_DIMM_D2 N/A | DDR4 | 32.0 GiB | 0x001e | CPU1_DIMM_A1 N/A | DDR4 | 32.0 GiB | 0x0022 | CPU1_DIMM_B1 N/A | DDR4 | 32.0 GiB | 0x0025 | CPU1_DIMM_C1 N/A | DDR4 | 32.0 GiB | 0x002a | CPU1_DIMM_D1 N/A | DDR4 | 32.0 GiB | 0x002e | CPU1_DIMM_E1 N/A | DDR4 | 32.0 GiB | 0x0031 | CPU1_DIMM_F1 N/A | DDR4 | 32.0 GiB | 0x0036 | CPU2_DIMM_A1 N/A | DDR4 | 32.0 GiB | 0x003a | CPU2_DIMM_B1 N/A | DDR4 | 32.0 GiB | 0x003d | CPU2_DIMM_C1 N/A | DDR4 | 32.0 GiB | 0x0042 | CPU2_DIMM_D1 N/A | DDR4 | 32.0 GiB | 0x0046 | CPU2_DIMM_E1 N/A | DDR4 | 32.0 GiB | 0x0049 | CPU2_DIMM_F1

Refer to the Quick Start Guide to Intel Optane DC persistent memory for more information.

Memory Mode Configuration

System memory consists of the total amount of memory provided by DRAM plus the Intel Optane DC persistent memory modules, allowing higher memory utilization and higher workload density compared to a system without persistent memory. Memory Mode-enabled modules are transparent to an application and are presented in the OS as /dev/nmem devices.

To enable Intel Optane DC persistent memory modules in Memory Mode (2LM), it is required to modify the following settings in the BIOS (see Figures 7 and 8):

Advanced ➔ Memory Configuration n IMC Interleaving = 2-way Interleave n Volatile Memory Mode = 2LM n ARS on Boot = Disabled n Average Power Budget = 18000

Advanced ➔ PCI Configuration UEFI Option ROM Control ➔ Intel Optane DC Persistent Memory Configuration ➔ Regions ➔ Create goal config ➔ Memory Mode [%] = 100

App Direct Mode Configuration

App Direct Mode exposes block devices to the OS as /dev/pmem. Utilization is software dependent. To enable Intel Optane DC persistent memory modules in App Direct Mode, first create a Region (Figure 9):

Advanced ➔ PCI Configuration ➔ UEFI Option ROM Control ➔ Intel Optane DC Persistent Memory Configuration ➔ Region ➔ Create Goal Config ➔ Memory Mode [%] = 0 (apply all capacity to App Direct)

Create Goal Config

Figure 8. Finishing the Memory Mode configuration.

Regions

Figure 9. Creating a region for App Direct Mode.

Memory Configuration

Figure 7. Memory Mode configuration for Intel® Optane™ DC persistent memory.

Reference Architecture | Deploying Red Hat OpenShift* Container Platform v3.11 12

Next, create a Namespace (see Figure 10):

Advanced ➔ PCI Configuration ➔ UEFI Option ROM Control ➔ Intel Optane DC Persistent Memory Configuration ➔ Namespaces ➔ Create Namespace ➔ Create Namespace

Using Apache Spark* with Intel® Optane™ DC Persistent Memory in Memory Mode

The Red Hat OpenShift Container Platform provides an efficient solution for executing analytics workloads. Apache Spark* with the Red Hat OpenShift Container Platform creates a flexible, standalone, and configurable data processing pipeline, while container-native storage exposes a fast, distributed, and resilient data backend.

Inclusion of Intel Optane DC persistent memory in data processing nodes allows greater density of executors and a larger amount of simultaneously processed data. Figure 11 shows an execution flow.

Shared, distributed volumes for application and data distribution are provided by OpenShift Container Storage:

apiVersion: v1kind: PersistentVolumeClaimmetadata: annotations: volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs name: spark-data-claimspec: accessModes: - ReadWriteMany resources: requests: storage: 1000Gi storageClassName: glusterfs-storage

The code snipet below represents an example of this execution flow.

scheduledriver pod

requestexecutor pods

executor podwatch events

OpenShiftMaster

KubernetesAPI Server

KubernetesScheduler

NODE A

Executor

NODE B

Executor

NODE C

Executor

Node A

Spark*Driver

spark-submit

Client

scheduleexecutor pod

CLUSTER

Figure 11. Execution flow for Apache Spark* with the Red Hat OpenShift* Container Platform.

spark-submit \ --master k8s://https://ocp.example.com:8443 \ --deploy-mode cluster \ --name spark-text-sort-count \ --driver-memory 2G \ --executor-memory 20G \ --driver-cores 10 \ --conf spark.executor.instances=70 \ --conf spark.kubernetes.executor.request.cores=10 \ --conf spark.kubernetes.executor.limit.cores=10 \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.kubernetes.namespace=spark \--conf spark.kubernetes.driver.volumes.persistentVolumeClaim.drivervol.mount.path=/spark \ --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.drivervol.options.claimName=spark-data-claim \ --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.drivervol.mount.readOnly=false \ --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.execvol.mount.path=/spark \ --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.execvol.options.claimName=spark-data-claim \ --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.execvol.mount.readOnly=false \ --conf spark.kubernetes.container.image=docker-registry.default.svc:5000/spark/spark-py:latest \ file:///spark/xml_process.py

Create Namespace

Figure 10. Creating a namespace for App Direct Mode.

Reference Architecture | Deploying Red Hat OpenShift* Container Platform v3.11 13

For full list of options related to running Spark with Kubernetes, click here. Note that the Red Hat OpenShift Container Platform incorporates the Kubernetes scheduler, hence the installation process is the same for both Spark and Red Hat OpenShift Platform. No additional steps are required to take advantage of Intel Optane DC persistent memory modules in Memory Mode.

ConclusionIntel® solutions involving the Red Hat OpenShift Container Platform provide an excellent foundation for building a production-ready environment that simplifies the deployment process, provides the latest best practices, and helps ensure stability by running applications in a highly available environment.

The guidance and best practices described in this reference architecture provide system, network, storage, and Red Hat OpenShift Container Platform administrators the blueprints required to create solutions to meet business needs. Administrators may reference this document to simplify and optimize their Red Hat OpenShift Container Platform with Intel® infrastructure components.

Find the solution for your organization. Contact your Intel representative or visit builders.intel.com/intelselectsolutions/hybrid-cloud/intelr-select-solutions-for-red-hat-openshift-container-platform.

1 PortWorx, October 2018, “2018 Container Adoption Survey.” portworx.com/wp-content/uploads/2018/12/Portworx-Container-Adoption-Survey-Report-2018.pdf Notice: This document contains information on products in the design phase of development. The information here is subject to change without notice. Do not

finalize a design with this information. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software, or service activation. Learn more at

intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Intel does not assume any liability for lost or stolen data or systems or any damages resulting from such losses. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect

actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit intel.com/performance.

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit intel.com/benchmarks.

You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps.

Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

Warning: Altering PC clock or memory frequency and/or voltage may (i) reduce system stability and use life of the system, memory and processor; (ii) cause the processor and other system components to fail; (iii) cause reductions in system performance; (iv) cause additional heat or other damage; and (v) affect system data integrity. Intel assumes no responsibility that the memory, included if used with altered clock frequencies and/or voltages, will be fit for any particular purpose. Check with memory manufacturer for warranty and additional details.

Cost reduction scenarios described are intended as examples of how a given Intel- based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction.

Results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance.

Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate.

Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting intel.com/design/literature.htm.

Intel, the Intel logo, Xeon, and Optane are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © Intel Corporation 0919/ACHA/KC/PDF 340710-001US

Learn MoreYou may find the following resources useful:• Red Hat OpenShift* Container Platform• Intel® Optane™ DC persistent memory