Dell EMC Ready Solutions for Oracle with Dell EMC XtremIO ... · Provisioning XtremIO LUN and presentation with a few clicks in XtremIO WebUI Designed to modernize how databases are

Abstract

This validation guide describes the system architecture, design, testing methodology, and testing results of Dell EMC Ready Solutions for Oracle using Dell EMC XtremIO X2 all-flash and Data Domain protection storage.

Validation Guide

Dell EMC Solutions

Dell EMC Ready Solutions for Oracle with Dell EMC XtremIO X2 and Data Domain

Enterprise-Class Storage Provisioning and Data Protection

November 2018

H17447.1

2

Dell EMC Ready Solutions for Oracle with Dell EMC XtremIO X2 and Data Domain Enterprise-Class Storage Provisioning and Data Protection Validation Guide

The information in this publication is provided as is. Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any software described in this publication requires an applicable software license.

Copyright © 2018 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Intel, the Intel logo, the Intel Inside logo and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. Other trademarks may be trademarks of their respective owners. Published in the USA 11/18 Validation Guide H17447.1.

Dell Inc. believes the information in this document is accurate as of its publication date. The information is subject to change without notice.

Contents

3

Dell EMC Ready Solutions for Oracle with Dell EMC XtremIO X2 and Data Domain Enterprise-Class Storage Provisioning and Data Protection

Validation Guide

Contents

Chapter 1 Executive Summary 5

Business challenge ............................................................................................... 6

Benefits of Ready Solutions for Oracle .................................................................. 7

Key results ............................................................................................................ 8

Scope.................................................................................................................. 10

Audience ............................................................................................................. 10

We value your feedback ...................................................................................... 11

Chapter 2 Technology Overview 12

Solution overview ................................................................................................ 13

Solution architecture ........................................................................................... 14

Key components ................................................................................................. 15

Chapter 3 Architecture Overview 19

Logical architecture overview .............................................................................. 20

Physical architecture overview ............................................................................ 24

Data Domain DD6300 systems for database backup .......................................... 29

Chapter 4 Design Considerations 30

Compute design .................................................................................................. 31

Network design ................................................................................................... 37

Storage design .................................................................................................... 45

XtremIO Virtual Copy (XVC) database design ..................................................... 51

Data Domain backup system design ................................................................... 54

Chapter 5 Test Methodology and Results 60

Test objective ...................................................................................................... 61

Test tools and methods ....................................................................................... 62

Use case 1: Ease of storage provisioning with XtremIO ...................................... 62

Use case 2: Baseline – One production OLTP RAC database ............................ 66

Use case 3: Impact of one 11gR2 and one 12cR2 virtualized database on baseline ........................................................................................................ 78

Results Summary: Mixed physical and virtual environments running mixed OLTP and OLAP workloads on Ready Solution for Oracle with XtremIO X2 ........................................................................... 83

Contents

4


Chapter 6 Test Methodology and Results: Data Protection 84

Test objective ...................................................................................................... 85

Use case 1: Full backup of one OLTP RAC database ......................................... 85

Use case 2: Restore and recovery of one OLTP RAC database from full backup ............................................................................. 88

Data protection testing summary ......................................................................... 88

Chapter 7 Conclusion 90

Conclusion .......................................................................................................... 91

Benefits ............................................................................................................... 91

Summary ............................................................................................................ 92

Chapter 8 References 93

Dell EMC documentation..................................................................................... 94

VMware documentation ...................................................................................... 94

Oracle documentation ......................................................................................... 94

HammerDB documentation ................................................................................. 94

Appendix A Configuration Details 95

Database performance data collection ................................................................ 96

Database parameters .......................................................................................... 98

HammerDB configuration parameters ................................................................. 98

OLAP query customization ................................................................................ 100

Chapter 1: Executive Summary

5

Dell EMC Reay Solution for Oracle with Data Protection Enterprise-Class Protection and Provisioning using Dell EMC XtremIO X2 and Data Domain Storage

Validation Guide


Validation Guide

Chapter 1 Executive Summary

This chapter presents the following topics:

Business challenge ............................................................................................. 6

Benefits of Ready Solutions for Oracle ............................................................. 7

Key results........................................................................................................... 8

Scope ................................................................................................................. 10

Audience ............................................................................................................ 10

We value your feedback ................................................................................... 11


6


Business challenge

Oracle databases often support a company’s most complex and critical applications,

frequently enabling Enterprise Resource Planning (ERP) and Customer Relationship

Management (CRM) systems that are responsible for all back-office processes. The

pressure to modernize the database infrastructure means that businesses are looking for

solutions that offer greater agility, operational efficiencies, and resiliency in a single

solution.

Dell EMC Ready Solutions for Oracle are solutions that are designed to boost

performance and operational agility for your database ecosystem. Ready Solutions for

Oracle integrate Dell EMC PowerEdge servers, networking, and storage. They incorporate

proven design, testing, and release phases that might take months or weeks to complete,

to deliver superior performance, significant cost savings, and future-ready scalability. The

engineered Ready Solutions for Oracle provide your business with faster time-to-value to

reach operational readiness.

Dell EMC XtremIO X2 storage, a purpose-built all-flash array, has been added to the

Ready Solutions for Oracle offerings. XtremIO X2 storage offers consistently high

performance with low latency, unmatched storage efficiency with inline, all-the-time data

services, rich application, integrated copy services, and unprecedented management

simplicity.

XtremIO storage is designed and optimized for business applications, databases, and for

DBAs to:

Provide an extremely balanced system across all X-Bricks for compute power,

storage bandwidth, and capacity.

Provide consistent and predictable database performance by balancing data

distribution to eliminate hot spots.

Inline deduplication saves physical data capacity without impacting database

performance.

Only copy and store unique data that is added or changed from the point that the

copy is made due to inline deduplication. This feature helps with data reduction

for Oracle databases.

Provide a flash-based data protection algorithm (XDP) that offers better

performance than RAID-10 with RAID-5 capacity savings.


7


Validation Guide


Validation Guide

Benefits of Ready Solutions for Oracle

Traditionally, application owners worked collaboratively with the IT organization to design

a new database platform. The multivendor process took months and required extensive

research and analysis to ensure that all the components worked together. Without any

assurance that the new system would perform as expected, the endeavor also entailed

significant risk.

Dell EMC Ready Solutions, such as Ready Solutions for Oracle, transform the design,

buy, and build process by providing a fully integrated and tested platform. In designing

Ready Solutions for Oracle, we focus on key priorities such as performance, resiliency,

and automation. Ready Solutions for Oracle with Dell EMC XtremIO X2 fully integrate

these components:

Dell EMC PowerEdge R740 and R940 servers

Red Hat Enterprise Linux 7.4

VMware vSphere 6.5

Dell EMC Networking

Dell EMC XtremIO X2 storage

Complete testing with customer-purchased Oracle 12c Release 2 and Oracle 11g

Release 2 databases

Dell EMC Data Domain 6300 storage

This paper focuses on the configuration that includes R740 and R940 servers, however,

the R840 server is also supported. Ready Solutions for Oracle can also be based on a

VMAX system configuration.

By eliminating the time-consuming and complex process of designing a system, this

pretested, prevalidated solution streamlines the purchase and update cycles for the IT

organization. It also accelerates delivery times of complex mission-critical databases and

applications. Features of this Ready Solution for Oracle include:

Sub-0.85 millisecond latencies for OLTP databases and applications such as ERP

and CRM systems with higher IOPS, network throughput, and transactions per

second

High density of IOPS for greater database consolidation

High throughput for OLTP and mixed workloads

Inline deduplication and compression to increase space savings without impacting

database performance


8


Integration with VMware vSphere for centralized virtualization management

Automated database copies, repurposing, and protection with Dell EMC XtremIO

Virtual Copy (XVC)

Provisioning XtremIO LUN and presentation with a few clicks in XtremIO WebUI

Designed to modernize how databases are managed, Ready Solutions for Oracle are:

Engineered—Compute, networking, and storage integrated with prerequisites and

dependencies have been tested to deliver a seamless solution experience.

Agile—A modern Oracle management experience with simpler and intuitive

provisioning capabilities featuring 3-step database creation provides faster time to

value.

Optimized—Proven and documented consistent performance, simpler

management 3-step storage provisioning, higher data reduction (while creating

faster snapshots) and resiliency best practices ensure a highly effective Oracle

database environment.

Key results

Ready Solutions for Oracle accelerate adoption of a modern database platform. We tested

and validated every component, including servers, storage, and software, with Oracle

databases. We conducted extensive testing to ensure integration, performance, resiliency,

and the development of best practices. Sizing the solution for your Oracle ecosystem is a

streamlined process because extensive testing provides an accurate foundation for

meeting database requirements. As part of the validation process, this guide documents

the best practices that we used to configure and accelerate Oracle databases on an

Oracle Real Application Clusters (RAC) production OLTP, snapshot, and virtualized (both

11g and 12c) database configurations along with the backup and recovery of the RAC

production configurations.

We tested the physical environment or the physical production database setup by running

an Oracle RAC 12cR2 database across two physical servers running Red Hat Enterprise

Linux 7.4.

We created and validated the Oracle 12c production OLTP database on the XtremIO X2

storage with just a few clicks. The key results for this use case are as follows:

DBAs can plan and provision the XtremIO storage simply by using the XtremIO UI

with 3-step provisioning.

The Ready Solution for Oracle sustained 383,000 Transactions per Minute (TPM)

using PowerEdge R940 servers and an XtremIO X2 all-flash array.

Two-node Oracle 12c RAC generated over 120,000 IOPS at sub 0.85 millisecond

storage latencies for the production OLTP database.

Dell EMC achieved a 54% space savings with an Oracle 12c RAC database on the

new Ready Solution with XtremIO. Inline deduplication and compression reduced a

1.16 TB database to 629 GB on XtremIO X2.

Physical

production

database setup

results (use case

2: test 1)


9


Validation Guide


Validation Guide

We tested the physical XtremIO Virtual Copy (XVC) with Oracle databases that were

repurposed from the production database by running the following Oracle databases in

parallel on a single dedicated server:

One production RAC DB (two nodes)

One Online Analytical Processing (OLAP)-like XVC database repurposed from the

OLTP production database

One OLTP XVC developmental database repurposed from the OLTP production

database

The key results for this use case are as follows:

Over 1.4 GB/s was generated by the OLAP workload running in parallel with the

Oracle RAC OLTP database.

The OLAP workload did not significantly impact average storage latencies of the

Oracle RAC OLTP database.

Over 389,000 TPM were achieved by the Oracle RAC database running in parallel

with the OLAP database.

XtremIO overall copy efficiency of 2.9 to 1 shows that two copies of an Oracle

database used a small amount of space. For example, 3.5 TB of Oracle databases

occupied just 1.2 TB on XtremIO X2 due to advanced data reduction features.

We tested the virtualized environment by running the following Oracle databases in

parallel on a single dedicated VMware ESXi host:

One production RAC DB (two nodes)

One virtual machine (VM) running Oracle Database 11g Release 2

A second VM running Oracle Database 12c Release 2

The key results for this use case are as follows:

We generated over 451,000 TPM across five OLTP databases on XtremIO X2

XtremIO scaled from 383,500 to 451,000 TPM, demonstrating the capability to

address rapid application growth

XtremIO X2 sustained over 131,000 IOPS across five Oracle database

Average storage latencies remained under 1 millisecond with five Oracle database

workloads running in parallel.

XtremIO scaled from 120,000 to over 131,000 IOPS using 36 SSD drives in

XtremIO X2 with with sub-millisecond latencies

In this use case, we observed a complete database consolidation with sub 0.85

millisecond latencies, and with increased IOPS, network throughput, and transactions per

minute.

Physical XVC

databases setup

results (use case

2: tests 2 and 3)

Virtualized

databases setup

results (use case

3)


10


We validated and tested the backup and recovery for this configuration of the Ready

Solutions for Oracle by using a Dell EMC Data Domain DD6300 system with Data Domain

Boost (DD Boost) software. We used the following test cases:

Full backup of a 1 TB production OLTP database

Full recovery of a 1 TB production OLTP database

Dell EMC engineering test results show the following performance outcome by using a

Data Domain DD6300 system:

A full backup of a 1 TB Oracle database completed in 36 minutes. The database

size was reduced to 756.7 GB (total compression size) in the Data Domain system

with 28 percent compression (a compression factor of 1.4X) and storage throughput

of 482 MB/sec.

The full recovery of a 1 TB Oracle database completed in 32 minutes with storage

throughput of 571 MB/sec.

Scope

This validation guide describes the Ready Solutions for Oracle with Data Protection on

XtremIO X2 storage, including Data Domain storage. We tested and validated Ready

Solutions for Oracle with various types and sizes of database workloads to ensure

maximum flexibility. This guide discusses the methodology of the testing that we

conducted on the solution and the results of the testing.

Audience

This guide is intended for IT administrators, storage administrators, virtualization

administrators, system administrators, IT managers, and personnel who evaluate, acquire,

manage, maintain, or operate Oracle database environments.

Data protection

test results

(Chapter 6)


11


Validation Guide


Validation Guide

We value your feedback

Dell EMC and the authors of this document welcome your feedback on the solution and

the solution documentation. Contact The Dell EMC Solutions team with your comments.

Authors: Oracle Ready Solutions Engineering team, Indranil Chakrabarti, Sam Lucido,

Reed Tucker

mailto:[email protected]?subject=Dell%20EMC%20Ready%20Bundle%20for%20Oracle%20with%20Data%20Protection%20Validation%20Guide%20(H16631.2)

Chapter 2: Technology Overview

12


Chapter 2 Technology Overview


Solution overview ............................................................................................. 13

Solution architecture ........................................................................................ 14

Key components ............................................................................................... 15


13


Validation Guide


Validation Guide

Solution overview

The ability to run OLAP, OLTP, and test/dev in a single virtualized environment while

safeguarding availability, lowering costs, and increasing productivity yields significant

advantages. This Ready Solution for Oracle was designed to support production OLTP

and OLAP applications and test/dev environments simultaneously without sacrificing

performance or storage space. With these Oracle databases and other applications co-

existing and functioning optimally, you gain a host of benefits, including the following:

Enhanced IOPS with sub-0.85 millisecond response times

Efficient operation of multiple applications in the same rack while saving on

licensing costs

Breakthrough simplicity for deployment, management, and support

Management by DBAs of their own backup and recovery processes without the

assistance of backup administrators

Exceptional performance, reducing backup times maintenance windows

Minimal impact on production workloads while performing backups or recovery,

enabling DBAs to run production simultaneously with backup windows

Integration of the hardware stack, drivers, and other components by the same

common vendor

Reduction of the customer’s TCO due to purchase of an optimal hardware stack

with this solution


14


Solution architecture

The following figure shows the solution architecture.

Figure 1. Ready Solutions for Oracle with XtremIO X2 storage and data protection: Architecture overview


15


Validation Guide


Validation Guide

Key components

Dell EMC PowerEdge 14G servers are built to accommodate databases, storage arrays,

data protection, hyper-converged appliances and racks. They are frequently used in

Ready Solutions and other Dell EMC solutions. These servers are part of a secure,

scalable compute platform that is the ideal foundation for cloud, analytics, and software-

defined data center initiatives.

The PowerEdge R740 server was designed to accelerate application performance by

using accelerator cards and storage scalability. The 2-socket, 2U platform has the

optimum balance of resources to power the most demanding environments.

The PowerEdge R940 server delivers consistent, high performance results for data-

intensive applications and data analytic workloads. With powerful Intel Xeon Scalable

processors and up 112 cores, the PowerEdge R940 server can quickly turn analytics into

insights to drive your business forward faster. Create an optimal configuration of NVMe,

SSD, HDD, and GPU resources to address your most demanding workloads–all in a 2U

chassis.

The Dell EMC XtremIO array is a purpose-built, all-flash array that delivers an innovative

metadata-centric architecture and unprecedented management simplicity. It is designed

with an elegant software-driven architecture that not only provides high performance with

consistently low latency for production volumes, but also creates thousands of copies on

which it runs workloads without impacting performance. The XtremIO X2 array provides

all-the-time, inline data services: data reduction, thin provisioning, flash-optimized data

protection, encryption, metadata-aware replication, and in-memory virtual data copies.

XtremIO Virtual Copy (XVC) helps provision and deploy space-efficient, instant virtual

data copies without impacting system performance. XVCs are created by capturing the

state of data in volumes at a particular point in time and allowing users to access that data

when needed, even when the source volume has been changed or deleted. XVCs are

inherently writeable, but can be created or changed to be read-only to maintain

immutability. Virtual copies can be taken from either the source or any virtual copy of the

source volume, which makes it easy to protect and recover from any operational and

logical corruption. XVCs enable the creation of frequent point-in-time copies (according to

RPO intervals – seconds, minutes, hours) and use them to recover from any data

corruption. An XVC can be kept in the system for as long as needed. Recovery by using

XVC is instantaneous and does not impact system performance.

XVC is an integral part of the XtremIO array’s integrated Copy Data Management (iCDM)

capabilities. It enables administrators to create, refresh, and restore thousands of

production copies, and run workloads on them without any storage overhead, accelerating

business agility. iCDM enables instant XVC creation from the production system or from a

gold master copy with no performance impact. Administrators can provision copies that

are immediately usable by the application, enabling quick deployments and ensuring that

those deployed copies are fully functional. These copies can be repurposed for near real-

time analytics, test/dev, patching, sandbox testing, and the ability to refresh and restore in

all directions, resulting in complete space efficiency. XVCs instantly refresh virtual copies

such as production to any copy, any copy to any copy, and any copy to production.

Dell EMC

PowerEdge 14G

servers

Dell EMC

XtremIO X2

storage

XtremIO Virtual

Copy (XVC)


16


Data Domain systems are disk-based inline deduplication appliances and gateways that

provide data protection and disaster recovery (DR) in the enterprise environment. All

systems run the Dell EMC Data Domain Operating System (DD OS), which provides a

command-line interface (CLI) to perform all system operations. They also run the Dell

EMC Data Domain System Manager (DD System Manager UI) to configure, manage, and

monitor. The Data Domain storage system offers a cost-effective alternative to tape. Data

Domain systems reduce the amount of disk storage to retain and protect data by 10 to 30

times. Because data on disk is available online and onsite for longer retention periods,

restoration is fast and reliable.

DD Boost technology for Recovery Manager (RMAN) optimizes communication between

the database servers and the Data Domain system. It improves backup performance by

reducing the amount of data that is transferred over the network between database

servers and the Data Domain system, as well as the amount of data stored by Data

Domain. By working with RMAN, the DD Boost components consists of:

DD Boost server software that runs on the Data Domain system.

DD Boost database application agents, which are installed on database servers.

The agent works as a plug-in for Oracle RMAN to provide database backup. It has

a DD Boost library for communicating with the DD Boost server running on a Data

Domain system.

DD Boost software extends the Data Domain Data Invulnerability Architecture by

generating checksums on the Oracle database server before RMAN sends the data to the

Data Domain system. The Data Domain system receives the data, computes the new

checksum that is based on the incoming data, and compares the new checksum with the

old checksum sent from the Oracle database server. This process ensures inline

verification of data.

DD Boost distributed segment processing

DD Boost technology includes distributed segment processing (DSP). When this feature is

enabled, the deduplication process is distributed between the DD Boost database

application agent on the database server and the DD Boost server on the Data Domain

system. Because parts of the deduplication process run on the database servers, the DD

Boost library sends only the unique data to the Data Domain system over the network.

With this DD Boost feature enabled, the backup with deduplication process follows these

steps:

1. The backup data stream is broken into variable-length segments and each

segment is identified.

2. The system determines if each segment is unique or if it is already stored in the

Data Domain system.

3. If the segment is unique (not stored in the Data Domain system), it is compressed

and sent over the network to the Data Domain system and written to the disks.

Distributed segment processing provides significant benefits for the Oracle database

backup:

Improves backup throughput because the DD Boost library sends only the unique

data to the Data Domain system. The more deduplicated data that there is in the

Dell EMC Data

Domain systems

Dell EMC DD

Boost technology

for Recovery

Manager


17


Validation Guide


Validation Guide

backup dataset, the higher the backup throughput, which in turn reduces the

backup time.

Reduces the network bandwidth requirement. Because only unique data is sent to

the Data Domain system through the network, less network bandwidth is used.

Reduces the storage capacity required to store database backup images, and

increases the retention period for the database backups.

VMware vSphere is a complete and robust virtualization platform that uses dynamic

resource pools to virtualize business-critical applications with flexibility and reliability. It

transforms a computer's physical resources by virtualizing the CPU, RAM, hard disk, and

network controller. This transformation creates fully functional VMs that run isolated and

encapsulated operating systems and applications.

The vSphere virtualization layer decouples the application from the underlying physical

resources. This decoupling enables greater flexibility in the application layer by eliminating

hardware downtime for maintenance and changes to the physical system without affecting

the hosted applications. In a server-virtualization use case, this layer enables multiple

independent VMs to share the same physical hardware.

Red Hat Enterprise Linux (RHEL) 7.4 offers automation capabilities designed to limit IT

complexity while enhancing workload security and performance for traditional and cloud-

native applications. For more information about Red Hat Enterprise Linux 7.4, go to the

Red Hat website.

Oracle Database 12c delivers industry-leading performance, scalability, security, and

reliability on a choice of clustered or single servers running Microsoft Windows, Linux, or

UNIX. It introduces a new architecture, Oracle Multitenant, where one or more pluggable

databases (PDBs) are created inside a container database (CDB).

The multitenant architecture supports the following configurations:

A single-tenant configuration, with one PDB plugged into a CDB, which is available

for no extra cost in all editions

A multitenant option for up to 252 PDBs per CDB, which is an extra-cost option of

Oracle 12c Enterprise Edition

Oracle Database 11g Release 2 provides a foundation from which IT can successfully

deliver more information with a higher quality of service, making more efficient use of your

IT budget.

Oracle RMAN is a database backup and recovery tool that is built into the Oracle

database server. With Oracle RMAN, the Oracle Database Administrator (DBA) schedules

database backup jobs to back up database files and archive logs to a backup system

routinely. DBAs also use RMAN to restore and recover database files and archive logs

from the backup system.The DD Boost database agent works with RMAN to send the

database backup images to the Data Domain backup system.

VMware

vSphere 6.5

Red Hat

Enterprise

Linux 7.4

Oracle Database

12c

Oracle Database

11g

Oracle RMAN for

database backup

and recovery

https://www.redhat.com/en/about/press-releases/red-hat-bridges-hybrid-multi-cloud-deployments-latest-version-red-hat-enterprise-linux-7

https://www.redhat.com/en/about/press-releases/red-hat-bridges-hybrid-multi-cloud-deployments-latest-version-red-hat-enterprise-linux-7


18


Dell EMC Connectrix switches and directors bring high bandwidth and zero downtime to

your storage network. Connectrix offers a range of enterprise-class directors, medium

density departmental switches, and edge switches for small to large enterprise

environments or applications. Key features include:

Fibre Channel (FC) connectivity of up to 32 Gb/s and speeds of up to 40 GbE

NVMe-ready

Departmental switch scaling from 8 to 96 ports per switch

Redundant components and multipath deployments to ensure high availability and

automated failover

Advanced management tools to simplify the deployment and management of your

storage networking environment automatically

This solution was built with the cost-effective Connectrix B-Series DS-6510B and DS-

6610B switches. Most of the switches in the Connectrix portfolio can be used to build a

storage network for Oracle solutions, as long as the SAN speeds match or exceed the

speed of the devices in the SAN.

The DS-6510B is a 16 Gb switch that scales from 24 ports to 48 ports. The DS-6610B is a

32 Gb switch that scales from 8 ports to 24 ports. By default, the DS-6610B has 16 Gb

SFPs but can be upgraded with 32 Gb SFPs.

S4048-ON ToR switch

The Networking S4048-ON switch is an ultra-low-latency 10/40 GbE top of the rack (ToR)

switch that is built for applications in high-performance data center and computing

environments. By using a non-blocking switching architecture, the S4048-ON switch

delivers line-rate L2 and L3 forwarding capacity with ultra-low-latency to maximize

network performance.

The compact design provides a density of 48 dual-speed 1/10 GbE (SFP+) ports and six

40 GbE QSFP+ uplinks to conserve valuable rack space and simplify migration to 40 Gb/s

in the data center core. Each 40 GbE QSFP+ uplink can also support four 10 GbE ports

with a breakout cable.

In addition, the S4048-ON switch incorporates multiple architectural features that

optimize:

Data center network flexibility

Efficiency and availability

PSU to I/O panel airflow for hot/cold aisle environments

Redundant, hot-swappable power supplies and fans

Networking S3048-ON management switch

The Networking S3048-ON management switch is a low-latency switch that features forty-

eight 1 GbE and four 10 GbE ports, a dense 1U design, and up to 260 Gb/s performance.

Dell EMC

Connectrix

switches

Dell EMC

Networking

switches

Chapter 3: Architecture Overview

19


Validation Guide


Validation Guide

Chapter 3 Architecture Overview


Logical architecture overview .......................................................................... 20

Physical architecture overview ........................................................................ 24

Data Domain DD6300 systems for database backup ...................................... 29


20


Logical architecture overview

The XtremIO X2-based Ready Solution for Oracle is designed to consolidate multiple

types of mixed-workload Oracle databases in a single system. The following types of

Oracle databases have been tested and validated for Ready Solutions for Oracle:

A production database in a physical environment (two nodes)

Two-nodes running Oracle RAC OLTP production database (also referred to as

PROD, OLTP Prod, or Prod database)

XtremIO XVC databases in a physical environment (single node)

One XtremIO XVC database repurposed from the production database for a

OLAP workload (also referred to as OLAP XVC, snpolap, or OLAP Snap)

One OLTP XVC database repurposed from the production database for

development (also referred to as DEV XVC or OLTP Snap)

Oracle databases in a virtual environment (single node)

One VM running Oracle Database 11g Release 2 (also referred to as v11gDB)

A second VM running Oracle Database 12c Release 2 (also referred to as

v12cDB)

The following figure shows the logical architecture of consolidated mixed-workload

databases. It includes the multiple layers of infrastructure components of the Ready

Solutions for Oracle along with Data Protection by using a Data Domain system as the

backup appliance.


21


Validation Guide


Validation Guide

Figure 2. Ready Solutions for Oracle with XtremIO X2 Storage and data protection: Architecture overview


22


The following figure shows the logical layout of the Ready Solution for Oracle with the

XtremIO X2 platform.

Figure 3. Ready Solution for Oracle with XtremIO X2 storage: Logical layout

The server layer consists of:

Two PowerEdge R940 servers host a two-node Oracle 12cR2 RAC database

(PROD ) with one Oracle RAC database instance on each server. Both of these

PowerEdge R940 servers have four 18-core CPUs and 1.5 TB RAM. The software

stack includes Red Hat Linux (RHEL) 7.4 and the Oracle 12cR2 RAC database

environment, which includes Oracle 12cR2 Grid Infrastructure and Oracle 12cR2

RAC.

One PowerEdge R940 server is the ESXi host with ESXI 6.5 U2 hosting two virtual

machines, VM1 and VM2. These machines host one single-node 12c OLTP

database (v12cDB) and one single-node 11gR2 OLTP database (v11gDB). The

PowerEdge R940 server has four 18-core CPUs and 1.5 TB RAM. The virtual

machine VM1 uses RHEL 7.4 as the guest operating system that runs Oracle

12cR2 Automatic Storage Management (ASM) and an Oracle 12cR2 standalone

database. The virtual machine VM2 uses RHEL 6.9 as the guest operating system

that runs Oracle 11gR2 ASM and an Oracle 11gR2 standalone database.

One PowerEdge R740 server hosts the DEV XVC and OLAP XVC snapshot

databases. The PowerEdge R740 server has two 12-core CPUs and 768 GB RAM.

The PowerEdge R740 server stack consists of RHEL 7.4 and Oracle 12cR2 ASM

with Oracle 12cR2 standalone database software. These snapshot databases are

based on an XVC virtual copy of the database volumes of the Oracle 12cR2 OLTP

RAC database (PROD). The DEV XVC database is a snapshot copy of the PROD

database for development and test purposes, while OLAP XVC is for OLAP

reporting. These XVC copies are mounted to the ASM instance in this server to

form corresponding diskgroups where the two snapshot databases are stored. Both

XVC snapshot databases are writable snapshot copies of the PROD database.

Logical layout


23


Validation Guide


Validation Guide

The storage layer consists of an XtremIO X2 two-X-Brick cluster that hosts the storage

volumes’ XVC snapshots of all the databases and Oracle 12cR2 clusterware, as well as

the VM operating system disks. The following table describes the storage layout.

Table 1. Storage size for databases

Database Storage sizes (GB)

DATA REDO TEMP FRA Clusterware1 VM OS Total

OLTP 12cR2 PROD

4x300 2x100 500 200 3x50 N/A 2,250

v12cDB 4x600 2x100 500 2x100 3x100 600 4,200

v11gDB

DEV XVC2 4x300 2x100 100 200 3x50 N/A 250

OLAP XVC2 4x300

Total 3,600 400 1,100 600 600 600 6,700

Notes:

1 Clusterware storage size includes space for the Oracle Cluster Registry (OCR), voting disk, and

Grid Infrastructure Management Repository (GIMR). The VM operating system volume consists of

the virtual disks of the VMM operating system. 2 Snapshot volumes for both XVC databases - 4x300 GB for DATA, 2x100 GB for REDO, ASM and 1x200 GB for FRA - are repurposed copies of the corresponding volumes of OLTP 12cR2 PROD database. These volumes are not counted as extra space beyond the original storage volumes.

ASM provides storage management for all databases. For 12cR2 RAC PROD databases:

4 x 300 GB volumes are grouped to form a 1,200 GB DATA diskgroup for database

files

2 x 100 GB volumes form REDO1 and REDO2 diskgroups for REDO logs

1 x 200 GB volume forms a FRA diskgroup for FRA (Flash Recovery Area)

1 x 500 GB volume forms a TEMP diskgroup for TEMP files

For virtual databases, each volume is divided into two equal VMDKs, each of which is

presented to each of the two VMs to form the corresponding diskgroups. For the XVC

snapshot databases, the XVC snapshots are mounted into the ASM instance of the XVC

database server to form the corresponding diskgroups.


24


Physical architecture overview

The following figure shows the physical architectural layout of the Ready Solution for

Oracle with an XtremIO X2 storage array.

Figure 4. Physical architecture and connectivity overview

Physical layout


25


Validation Guide


Validation Guide

The figure also shows the high level connectivity between the various major components

in the Ready Solutions for Oracle. A single point of failure is eliminated for all the critical

hardware components, as follows:

Each of the database servers consists of highly available LAN and SAN

connectivity with multiple network interface cards (NICs) and multiple host bus

adapters (HBAs).

Two dual-port 10 GbE NICs and two dual-port 16 Gb/s HBAs in each database

server provide sufficient bandwidth and high availability for Oracle database

LAN and SAN traffic.

Redundant network ports are connected to separate 10 GbE switches for HA

and bandwidth in case of failure.

HBA ports on the same HBA adapters are connected to separate FC switches

for HA and bandwidth in case of failure.

Redundant ToR S4048-ON 10 Gb Ethernet switches provide sufficient 10 Gb ports

and bandwidth to share both Oracle database public and private interconnect traffic.

These switches also provide sufficient 40 GbE uplink ports to connect the Ready

Solution for Oracle to the datacenter core switches.

Redundant DS-6510B 16 Gb/s Fibre Channel (FC) switches provide sufficient ports

and bandwidth to share Oracle database SAN traffic between the database servers

and the XtremIO X2 storage controllers.

A dual X-Bricks XtremIO X2 cluster storage array with redundant and active-active

storage controllers provide sufficient capacity, bandwidth, and high availability for all

the Oracle databases tested in the Ready Solution for Oracle.

Each FC port in each XtremIO X2 storage controller is connected to separate

FC switches for high bandwidth and high availability (HA).

A management port from each major hardware component is connected to S3048-

ON 1 GbE management switch.

On each of the database servers:

A dedicated iDRAC network port is connected to the management switch

for out-of-band, bare-metal management of the server

The 1 GbE LAN on motherboard (LOM) network ports is connected to the

management switch for in-band management of the server from within the

operating system


26


The following table lists the major hardware components in the Ready Solution for Oracle

with XtremIO X2 storage.

Table 2. Ready Solution for Oracle with XtremIO X2 storage: Components overview

Component Description

Database servers 2 x Dell EMC 4S PowerEdge R940 servers for physical production databases

1 x Dell EMC 4S PowerEdge R940 server for virtual databases

1 x Dell EMC 2S PowerEdge R740 server for physical XVC databases

LAN switches 2 x Dell EMC Networking S4048-ON 10 Gb Ethernet switches

SAN switches 2 x Dell EMC Connectrix DS6510-B 16 Gb/s FC switches

Management switch 1 x S3048-ON 1 Gb Ethernet switch

Storage array Dual X-Bricks XtremIO X2 cluster

Compute

The following table lists the hardware details of the database servers used for physical

production databases in the Ready Solution for Oracle with XtremeIO X2 storage.

Table 3. Ready Solution for Oracle with XtremIO X2 storage: Physical production database server components


Servers 2 x PowerEdge R940 servers

Chassis 2.5” chassis with up to 8 hard drives

Processor per server 4 x Intel Xeon Gold 6150 18c 2.7 GHz

Memory per server 1,536 GB (24 x 64 GB QR DDR4 2666MT/s LRDIMMs)

Local disks per server 3 x 1.2 TB 10 K SAS 12 Gb/s 2.5 in. HDDs (includes 1 hot spare)

RAID controller PERC H740P/H730P

iDRAC iDRAC9 Enterprise

rNDC Broadcom 5720 QP 1 Gb Base-T rNDC

Add-on NICs per server 2 x Broadcom 57412 DP 10 Gb SFP+ PCIe adapter

HBAs per server 2 x Emulex LPe31002-M6-D DP 16 Gb/s FC HBAs

Power supplies per server 2 x 1,600 W

Hardware

components and

sizing


27


Validation Guide


Validation Guide

Table 4. Ready Solution for Oracle with XtremIO X2 storage: Physical XVC databases server components


Server 1 x PowerEdge R740 server

Chassis 8 x 2.5 in. SAS/SATA hard disk drives (HDDs) for 2 CPU configuration

Processor 2 x Intel Xeon Gold 6136 12c 3.0 GHz

Memory 768 GB (24 x 32 GB DR DDR4 2667 MT/s RDIMMs)

Local disks 3 x 1.2 TB 10 K SAS 12 Gb/s 2.5 in. HDDs (includes 1 hot spare)

RAID controller PERC H740P Adapter


rNDC Broadcom 5720 DP 1Gb + 57412 DP 10Gb NetXtreme- E rNDC

Add-on NICs None

HBAs 2 x Emulex LPe16002B-M6-D DP 16 Gb/s FC HBAs

Power supply 2 x 1,100 W

Table 5. Ready Solution for Oracle with XtremIO X2 storage: Virtualized databases server components


Servers 1 x PowerEdge R940 server

Chassis 2.5” chassis with up to 8 hard drives

Processor 4 x Intel Xeon Gold 6150 18c 2.7 GHz

Memory 1,536 GB (48 x 32 GB DR DDR4 2667 MT/s RDIMMs)

Local disks 8 x 1.6 TB SAS 12 Gb/s 2.5 in. SSDs

RAID controller PERC H740P


rNDC Broadcom 5720 QP 1 Gb Base-T rNDC

Add-on NICs 2 x Intel X710 DP 10 Gb SFP+ NICs

HBAs 2 x QLogic QLE2692 DP 16 Gb/s FC HBAs

Power supply 2 x 2,000 W


28


Storage

The following table lists the hardware details of the storage array used in this Ready

Solution for Oracle with XtremIO X2 storage.

Table 6. Ready Solution for Oracle with XtremIO X2 storage: Storage components

Storage array Dell EMC XtremIO X2 storage

System specification Two X2-S X-Bricks Cluster

Operating system version 6.1.0-99_X2

Active-Active Controllers 4

Front-end FC ports 8

SSD enclosures 2

Number of SSDs 36

Raw/Usable capacity 13.1 TiB / 10 TiB

Infiniband switches 2

Network

The following table lists the network switches used in this Ready Solution for Oracle with

XtremIO X2 storage.

Table 7. Ready Solution for Oracle with XtremIO X2 storage: Network switches

Switch function Switch type

LAN 2 x Dell EMC Networking S4048-ON 10 GbE switches

SAN 2 x Dell EMC Connectrix DS-6510B 48-ports 16 Gb/s switches

Management 1 x Dell EMC Networking S3048-ON 1 GbE switch


29


Validation Guide


Validation Guide

Data Domain DD6300 systems for database backup

We tested the Data Domain DD6300 system as a backup configuration to back up and

restore the databases in this Ready Solution for Oracle. The following table lists the

hardware and software stack of the Data Domain DD6300 system. Chapter 6 provides the

details of our test methodology, tools, and results.

Table 8. Data Domain DD6300 system for database backup

Category Components

Processor 2 x Intel Xeon CPU E5-2680 v3, 2,501 MHz

Memory 96 GB (12 x 8 GB 1,866 MHz)

Number of network ports (in use) 2 x 10 GbE

Number of enclosures (DS60) 1

Active tier disks (in use) 11 x 3.64 TiB (40.03 TiB) SAS HDDs

Active tier disks (spare) 1 x 3.64 TiB SAS HDD

Cache tier disks 2 x 0.728 TiB (1.45 TiB) SAS SSDs

Expandable storage disks1 60 x 2.73 TiB (163.8 TiB) SAS HDDs—56 active (152.9 TiB) + 4 spare (10.9 TiB)

1 The expandable storage disks in DS60 were available but were not used in the testing.

This DD6300-based backup system was configured and tested as the database backup

and recovery solution for this Ready Solution for Oracle.

The following figure shows the high-level backup solution architecture. The Data Domain

DD6300 backup appliance is connected to the public network of this Ready Solution for

Oracle to back up the databases.

Figure 5. Backup solution high-level architecture

Chapter 4: Design Considerations

30


Chapter 4 Design Considerations


Compute design ................................................................................................ 31

Network design ................................................................................................. 37

Storage design .................................................................................................. 45

XtremIO Virtual Copy (XVC) database design ................................................. 51

Data Domain backup system design ............................................................... 54


31


Validation Guide


Validation Guide

Compute design

The database servers tested in this Ready Solution for Oracle—PowerEdge R940-based

physical production database servers, a PowerEdge R740-based physical XVC

databases server, and a PowerEdge R940-based virtual databases server—were

designed and configured with the following best practices:

The PCIe network adapters and HBAs that are used for Oracle database public,

private interconnect, and SAN traffic were populated based on the recommended

PCIe slot priority for optimal power, bandwidth, and thermal performance of the

adapters and the system. The populated and the recommended PCIe slots are

wired to two separate CPUs in their respective servers, allowing for load balancing

of I/Os across the two CPUs.

For optimal performance, in the PowerEdge R940-based databases servers,

the HBAs were populated in PCIe slots 2 and 5 and the NICs were populated in

PCIe slots 3 and 6.

For optimal performance, in the PowerEdge R740-based database server, the

HBAs were populated in PCIe slots 1 and 7 and the NICs were populated in

PCIe slot 3 and the rack Network Daughter Card (rNDC) slot.

The BIOS System Profile was set to Performance.

Memory DIMMs were populated with at least one DIMM per memory channel

across all CPU sockets to maximize the memory throughput with the CPU sockets.

The following table shows the capacity and quantity of memory DIMMs populated in

each of the database servers that are based on the recommended best practices.

Table 9. Database servers: Memory DIMMs capacity and quantities

Use case Server type

Number of CPU sockets

DIMMs per channel populated

DIMMs per socket populated

Per DIMM capacity

Total physical DRAM

OLTP PROD Database

R940 4 1 6 64 GB 4 x 1 x 6 x 64 = 1,536 GB

XVC Databases

R740 2 1 6 32 GB 2 x 1 x 6 x 32 = 768 GB

Virtual Databases

R940 4 2 12 32 GB 4 x 2 x 12 x 32 = 1,536 GB

For additional recommended best practices that were implemented on the physical

database servers see RHEL 7.4 as bare-metal operating system.

Physical servers


32


The virtualized environment in this Ready Solution for Oracle was tested with two virtual

databases deployed on a single R940-based VMware ESXi host.

Figure 6. Single ESXi host with two VMs running two virtual databases

As shown in the figure above, the ESXi host contains:

One virtual machine (VM1) running:

RHEL 6.9 as the guest operating system

Oracle 11gR2 grid infrastructure software

Standalone Oracle Database 11gR2

A second virtual machine (VM2) running:

RHEL 7.4 as the guest operating system

Oracle 12cR2 grid infrastructure software

Standalone Oracle Database 12cR2

The ESXi host and the VMs were configured, monitored, and maintained using VMware

vSphere Web Client and VMware vCenter Server Appliance (VCSA), which was deployed

as a VM on the management server.

ESXi host


33


Validation Guide


Validation Guide

In the PowerEdge R940-based virtual databases server, we deployed ESXi 6.4 U2 as the

hypervisor. We applied best practices in our test environment, as described in the

following sections.

Note: We used the XtremIO Host Configuration Guide to apply best practices. See this guide for

the complete list of best practices for XtremIO storage in a VMware ESXi host environment.

HBA queue depth settings

The following table lists the default and the recommended HBA queue depth settings in

ESXi 6.5 hosts connecting to XtremIO X2 storage arrays:

Table 10. HBA queue depth settings in ESXi based hosts

Parameter Default value Recommended value

LUN Queue Depth QLogic: 64

Emulex: 30

QLogic: 256

Emulex: 128

HBA Queue Depth QLogic: N/A

Emulex: 8192

QLogic: N/A

Emulex: 8192 (maximum)

We set the LUN queue depth to the recommended value of 256 for the QLogic HBAs used

in our virtualized databases server. This setting ensures that the XtremIO X2 storage

arrays handle an optimal number of SCSI commands (including I/O requests).

Note: The QLogic HBA Queue Depth setting is no longer read by vSphere, therefore, it is not

relevant when configuring a vSphere host with QLogic HBAs.

Multipath configuration

The virtual databases ESXi host was configured by using vSphere Native Multipathing

(NMP). The following parameter values are recommended on the host for optimal

performance with XtremIO X2 storage:

Set the native path selection policy on the XtremIO volumes presented to the ESXi

hosts to round-robin.

Set the vSphere NMP round-robin path switching frequency to XtremIO volumes

from the default value (1000 I/O packets) to 1.

Note: In ESXi 6.5, the default path selection policy is round-robin and the default path switching

frequency is 1. Therefore, no change was needed in our virtualized databases server.

Host parameter settings

The following ESXi host parameters were configured:

Disk.SchedNumReqOutstanding—Determines the maximum number of active

storage commands (I/O) allowed at any time at the VMkernel. This parameter value

for each XtremIO volume presented to the ESXi host was set to the recommended

value of 256 by running the following CLI commands on the ESXi 6.5 host:

https://sso.emc.com/cleartrust/iwa/emc/axm_home.asp?CT_ORIG_URL=%2Fauthsso%2Fprotected.jsp&ct_orig_uri=%2Fauthsso%2Fprotected.jsp


34


To get the list of all XtremIO volumes, type:

$> esxcli storage nmp path list | grep XtremIO -B1 | grep "\

naa" | sort | uniq

To set the value for each volume, type:

$> esxcli storage core device set -d <naa.xxx> -O 256

where <naa.xxx> is the XtremIO volume obtained from the previous command.

Disk.SchedQuantum—Determines the maximum number of consecutive

“sequential” I/Os allowed from one VM before switching to another VM. This value

was set from its default value of 8 to the recommended value of 64.

We used the following design principles and best practices to create the VMs in this

Ready Solution for Oracle:

SCSI controllers—We created multiple SCSI controllers to optimize and balance

the I/O for the different database disks, as shown in the following table.

Table 11. SCSI controller properties set in VMs

Controller Purpose SCSI bus sharing

Change type

SCSI 0 Guest OS disk None VMware Paravirtual

SCSI 1 Oracle DATA disks Physical VMware Paravirtual

SCSI 2 Oracle REDO disks Physical VMware Paravirtual

SCSI 3 Oracle OCR, GIMR, FRA, TEMP Physical VMware Paravirtual

Hard disk drives—All database-related virtual disks—for example, DATA, REDO,

FRA, OCR/VD, and TEMP—were assigned the following properties:

Type: ‘Thick provisioned eager zeroed’—Which ensures that the space required

for the virtual disks are allocated at creation time and the data on the physical

device on the storage is zeroed out

Sharing: ‘No sharing’—Selected because the deployed databases are

standalone databases that do not require sharing database virtual disks with

another VM, unlike RAC VMs that share virtual disks between two or more VMs

in the database cluster.

VM vCPU and vMem—The following table lists the distribution of virtual CPU

(vCPU) and virtual memory (vMem) to the two database VMs.

Table 12. VM configuration: vCPU and vMem details

VM Number of vCPUs

vMem

Reservation (GB) Total (GB)

VM1 18 120 140

VM2 18 120 140

Virtual machines


35


Validation Guide


Validation Guide

Enable disk UUID—In each of the VM options, we added and set the configuration

disk.enableUUID parameter to TRUE. This setting ensures that the VMDK

always presents a consistent UUID to the VM.

RHEL 7.4 as bare-metal operating system

In the PowerEdge R940-based production database servers and the PowerEdge R740-

based XVC databases server, we deployed RHEL 7.4 as the bare-metal operating

system.

Note: We used the XtremIO Host Configuration Guide to apply best practices. See this guide for

the complete list of best practices for XtremIO storage in a Linux environment.

HBA queue depth settings

The following table lists the default and recommended HBA queue depth settings for a

Linux environment.

Table 13. HBA queue depth settings in Linux-based servers


LUN Queue Depth QLogic: 32

Emulex: 30

QLogic: Keep default value

Emulex: Keep default value

HBA Queue Depth QLogic: 32

Emulex: 8192

QLogic: 65535 (maximum)

Emulex: 8192 (maximum)

Note: We kept the default queue depth values in our physical database servers because we used

Emulex HBAs.

I/O elevator settings

The recommended I/O elevator setting in the RHEL operating system running in the

database servers connecting to XtremIO X2 storage arrays is either deadline or noop.

The cfg I/O elevator setting is not recommended. In our physical database servers, we

used deadline, which is the default I/O elevator setting in RHEL 7.4.

Multipath configuration

We configured the physical database servers using Linux Native Multipathing available in

the RHEL 7.4 operating system. We created the configuration file for the

/etc/multipath.conf multipath daemon with the following recommended settings:

devices {

device {

vendor XtremIO

product XtremApp

path_grouping_policy multibus

path_checker tur

path_selector "queue-length 0"

rr_min_io_rq 1

user_friendly_names yes

fast_io_fail_tmo 15

failback immediate

}

RHEL operating

system for

Oracle

databases

https://sso.emc.com/cleartrust/iwa/emc/axm_home.asp?CT_ORIG_URL=%2Fauthsso%2Fprotected.jsp&ct_orig_uri=%2Fauthsso%2Fprotected.jsp


36


Partition alignment in Linux

We partitioned the database disks or XtremIO volumes presented to the Linux-based

physical database servers by using fdisk with the default starting sector value of 2,048.

This setting ensures that the starting sector number is a multiple of 16 (16 sectors, at 512

bytes each, is 8 KB). Therefore, each database disk is correctly aligned with the XtremIO

storage LUN striping.

RHEL as VM guest operating system

In this Ready Solution for Oracle, we deployed RHEL 6.9 as the guest operating system in

VM1 running Oracle 11g R2 database and RHEL 7.4 as the guest operating system in

VM2 running Oracle 12c R2 database.

In each guest operating system , we adhered to the following recommended best

practices.

PVSCSI driver

For optimal XtremIO X2 storage performance in a VMware environment, we recommend

PVSCSI controllers and driver in the guest VMs. In this Ready Solution for Oracle, we

ensured that the inbox RHEL vmw_pvscsi driver module was loaded and used in the

guest operating systems.

Note: The PVSCSI driver is used only when the SCSI controller type is set to VMware

Paravirtual in the VM settings.

PVSCSI LUN Queue Depth and ring-pages settings

The following table shows the default and the recommended vmw_pvscsi parameter

settings.

Table 14. PVSCSI parameter settings in guest operating systems


vmw_pvscsi.cmd_per_lun RHEL 6: 64

RHEL 7: 254

RHEL 6: 254

RHEL 7: 254

vmw_pvscsi.ring_pages RHEL 6: 8

RHEL 7: 8

RHEL 6: 32

RHEL 7: 32

The parameters and their respective recommended values in this table were appended to

the kernel boot arguments:

In the /boot/grub.conf file for the RHEL 6-based guest operating system

In the /etc/default/grub file for the RHEL 7-based guest operating system

Other guest operating system configurations

Other guest operating systems were configured as follows:

Multipath was not configured because it is handled at the ESXi host level.

In the 12c database VM, all database storage disks were set up by using Oracle

ASM Filter Driver (ASMFD).


37


Validation Guide


Validation Guide

In the 11g database VM, all database storage disks were set up by using udev

rules.

Network design

Physical network design and connectivity

The following figure shows the solution’s redundant and highly available physical LAN

design and connectivity.

Figure 7. Physical network design and connectivity

LAN setup


38


Note: The ports on the switches shown in the preceding figure, to which the database server

network ports are connected, are for illustration purposes only. Network administrators can choose

any available ports on the switches, as appropriate.

The four database servers tested in this solution were configured with a highly available

network:

In the two physical production RAC database servers represented by R940-Prod-

DB-Node1 and R940-Prod-DB-Node2 in Figure 7.

Each RAC node has two 10 GbE network ports for Oracle public traffic

connected to two separate ToR switches.

Each RAC node has two 10 GbE network ports for Oracle private interconnect

traffic connected to two separate ToR switches.

Each RAC node has two 1 GbE network ports connected to the management

switch. One port is for server management from within the operating system

and one port is for out-of-band management of the server by using iDRAC.

In the single physical XVC database server represented by R740-XVC-DB-Server

in Figure 7:

Two 10 GbE network ports for Oracle public traffic are configured and


Two 1 GbE network ports are configured and connected to the management

switch. One port is for in-band server management from within the operating

system and one port is for out-of-band management of the server by using

iDRAC.

In the single virtualized database server represented by R940-Virtual-DB-Server in

Figure 7:

Two 10 GbE network ports for Oracle public traffic are configured and


Two 1 GbE network ports are configured and connected to the management

switch. One port is for in-band server management using the VMware vCenter

Server Appliance and one port is for out-of-band management of the server by

using iDRAC.

Oracle public networks from all database servers and private interconnect networks from

the two Oracle RAC nodes are configured on the same redundant ToR 10 GbE S4048-ON

switches. However, the network traffic is segregated by using VLANs as shown in the

following table.

Table 15. Sample VLAN configuration on S4048-ON 10 GbE ToR switches

Traffic type VLAN ID

All Oracle public 16

Oracle private interconnect 100


39


Validation Guide


Validation Guide

Note: VLAN IDs used in the table are examples. Network administrators can use VLAN IDs that

conform to their network policies and standards only if the Oracle public and private networks are

on two separate VLANs.

Virtual network design

The virtual environment in this Ready Solution for Oracle was tested by using a single

VMware ESXi host running two VMs. One VM for running a standalone Oracle 11gR2

database and the other VM for running a standalone Oracle 12c R2 database.

Figure 8. Virtual network design on the VMware ESXi database host

The preceding figure shows the virtual or VM network topology implemented in the

virtualized databases ESXi host. It also shows how the virtual layer maps to the physical

layer.

As shown in the figure, the VMware-based virtual network design in the Ready Solution

for Oracle consists of the following virtual switches and port groups:

Public distributed switch—This switch is implemented as a distributed virtual

switch for Oracle public traffic, which is generated by the two virtual databases

running inside two separate VMs. In this distributed switch, we created one

distributed port group and two uplink ports:

Public distributed port group—This group provides the virtual interfaces for

Oracle public traffic for the two Oracle database VMs, which are represented by

the VM names xorb-virt-11g and xorb-virt-12c in the figure. This port

group is tagged with VLAN ID 16, which is the same as the VLAN ID that is

configured on the S4048-ON ToR switch for public traffic.


40


Two physical uplink ports—Two 10 GbE physical ports from two separate

NICs serve as uplink ports to the public distributed switch to provide sufficient

bandwidth and redundancy.

NOTE: Typically, for single ESXi host implementations, standard switches and port groups are

sufficient. Distributed switches and port groups configured in this solution provide easy expansion

to virtual Oracle RAC environments by using multiple ESXi hosts, if needed.

Management standard switch—This switch is implemented as a standard switch

for management traffic. The default VMKernel port in the Management Network port

group, vmk0, is used to manage the ESXi host from VMware vCenter. The VM

Network port group provides the virtual interfaces to access and manage the Guest

VMs. Both these management traffic interfaces use one 1 GbE LAN On

Motherboard (LOM) port in the ESXi host that is connected externally to the S3048-

ON management switch.

NOTE: Though not shown in Figure 8 or implemented during testing of this Ready Solution for

Oracle, we recommend that you configure an additional 1 GbE management network port for

redundancy.

We used the following recommended SAN connectivity and zoning best practices to

configure all the database servers with the XtremIO X2 storage arrays:

Use at least two initiators per ESXi host or physical database servers for load

balance and bandwidth. For high availability, place the two initiators on separate

HBAs.

Implement redundant FC switches for high availability.

Ensure that one FC zone set includes one HBA port or initiator and at least one FC

target port on each XtremIO storage controller.

The following figure shows the recommended physical SAN connections between the

database servers, the FC switches, and the dual X-Brick XtremIO X2 cluster storage

arrays in this Ready Solution for Oracle.

SAN setup


41


Validation Guide


Validation Guide

Figure 9. SAN setup: Physical design and connectivity

Note: The ports on the FC switches shown in the figure to which the database server HBA ports

and the XtremIO FC front-end ports are connected are for illustration purposes only. SAN

administrators can choose any available ports on the switches, as appropriate.

The SAN connectivity and redundancy of the components in the solution ensure that no

single point of failure exists and provides the necessary bandwidth. As shown in the

figure, the SAN setup in the Ready Solution for Oracle with XtremIO X2 storage consists

of:


42


HBAs in the database servers:

Two dual-port 16 Gb/s HBAs per database server with a total bandwidth of 8

GB/s per server. Two dual-port HBAs per node provide HA and bandwidth if

one HBA fails.

Each port on the same HBA adapter is connected to two separate FC switches

to provide redundant paths and HA if one FC port or FC switch fails.

Redundant FC switches:

Two 16 Gb/s FC switches provide redundant paths for bandwidth and

connectivity between the database servers and the storage controllers if one

FC switch fails.

Front-end FC ports in the XtremIO storage controllers:

Two 16 Gb/s FC front-end ports per XtremIO I/O storage controller with a total

bandwidth of 8 GB/s per controller.

Each FC front-end port per controller is connected to two separate FC switches

to provide redundant paths and HA if one or more FC front-end ports, one or

more storage controllers, or one FC switch fails.

Zoning overview

The following figure shows the logical view of the Oracle RAC production database

servers after the recommended zoning configurations are created on the redundant FC

switches. The same design and configuration is used for the XVC database and the

virtualized database servers. Zoning is configured so that each host initiator in the

database server is zoned to four target front-end ports that are located on four separate

XtremIO storage controllers. This configuration provides a total of 16 paths per database

server and ensures sufficient bandwidth and availability for the Oracle database servers to

reach the storage if one or more ports or HBAs, a switch, or storage FC ports or

controllers fail.


43


Validation Guide


Validation Guide

Figure 10. SAN setup: Zoning logical view

To test the backup and recovery solution, we connected the DD6300 system as the

backup appliance to the Ready Solution for Oracle with XtremIO X2 storage. As shown in

the preceding figure, we connected two 10 GbE ports from two separate NICs on the

DD6300 system to two separate S4048-ON 10 GbE switches. S4048-ON switches serve

as the ToR Ethernet switches for Oracle database public and private interconnect network

traffic in this Ready Solution for Oracle.

DD6300 network

design


44


Figure 11. Data Domain DD6300 backup and recovery solution: IP network connections

For the databases in this Ready Solution for Oracle to communicate with the DD6300

backup appliance, the two DD6300 network ports on the S4048-ON switches were added

as untagged 10 GbE ports to VLAN (configuration) ID 16. This VLAN serves as the public

VLAN in this Ready Solution for Oracle. We connected the management port on the


45


Validation Guide


Validation Guide

DD6300 system to the S3048-ON 1 GbE switch that serves as the management switch in

the Ready Solution for Oracle with XtremIO X2 storage.

In the DD6300 system, we configured the two network ports with static IP addresses that

belong to the same subnet as the Oracle database public network in this Ready Solution

for Oracle. These two network interfaces were added to the default interface group under

the DD Boost protocols configuration.

XtremIO storage volume design

In this solution, we used the following design principles to implement the storage volumes

for the Oracle databases:

Create three volumes for OCR

Create at least four volumes for DATA

Create two volumes for REDO

Create one volume each for FRA and TEMP

The following figure shows the storage design of these five databases.

Figure 12. Storage design and database consolidation on XtremIO X2 storage

In this solution, the Oracle RAC production database is running across two physical nodes

or servers. We created separate volumes for OCR, DATA, REDO, FRA, and TEMP in the

XtremIO system and mapped them to both physical hosts. The following table shows the

volume design for the production database

Storage design

overview

Production

database storage

design


46


Table 16. Production database volume design

Volume name

Number of volumes

Size per volume (GB)

Total size (GB)

C2-OCR 3 50 150

C2-DATA 4 300 1,200

C2-REDO 2 100 200

C2-FRA 1 200 200

C2-TEMP 1 500 500

Total 11 2,250

Two snapshot databases, DEV XVC and OLAP XVC, are hosted in a single R740-

XVC_DB_Server. As shown in Figure 12, these databases use the common OCR

volumes for the clusterware and use a single TEMP volume for the temporary tablespace.

Each snapshot database has its own DATA volumes, REDO volumes, and FRA volume.

While OCR volumes and the TEMP volume are the storage volumes, the four DATA

volumes, two REDO volumes, and FRA volume are the XVC snapshot copies of the

corresponding volumes of the OLTP RAC database (PROD). The following table lists the

XVC snapshot copies and storage volumes for the two XVC databases.

Table 17. List of storage volumes

Name Number of volumes

Snapshot/volume

Snapshot database

Source volumes


Total size (GB)

DATA_DEV 4 snapshots DEV XVC DATA 300 1,200

REDO_DEV 2 snapshots DEV XVC REDO 100 200

FRA_DEV 1 snapshot DEV XVC FRA 200 200

DATA_OLAP 4 snapshots OLAP XVC DATA 300 1,200

REDO_OLAP 2 snapshots OLAP XVC REDO 100 200

FRA_OLAP 1 snapshot OLAP XVC FRA 200 200

TEMP 1 volume Shared both databases

N/A 500 500

OCR 3 volumes Clusterware N/A 50 150

In the virtualized environment, we have two single-instance databases in two separate

VMs. We created shared volumes for OS, OCR, DATA, REDO, FRA and TEMP. The

following table shows the volume design for the virtualized databases.

XVC databases

storage design

Virtualized

databases

storage and ESXi

datastore design


47


Validation Guide


Validation Guide

Table 18. Virtualized database volume design

Volume Name

Number of volumes


Total size (GB)

C3-VM-OS 1 600 600

C3-OCR 3 100 300

C3-DATA 4 600 2,400

C3-REDO 2 100 200

C3-FRA 1 200 200

C3-TEMP 1 500 500

Total 12 4,200

VM datatore design

The VM datastore design on vSphere follows the XtremIO storage volume design.

Therefore, for each volume created on the XtremIO storage, we create an equivalent

vSphere datastore formatted with the VMFS 6 file system with a single GPT partition that

spans the entire disk.

Similar to the XtremIO storage volume design, the VM datastore design for the virtualized

database configuration involves a separate datastore for each of the volumes as shown in the

following table.

The table shows the datastore design for the virtualized databases. During the VM

configuration of each database, we selected these dedicated datastores as the location of

each of the HDDs that we created for each of the database volume types.

Table 19. vSphere VM datastore XtremIO X2 storage volume design for OLTP 11gR2 and 12cR2 virtualized databases

Datastore name Datastore size (GB)

Purpose

C3-VM-OS 600 1 x operating system datastore for the two guest

operating systems (xorb-virt-11g and

xorb-virt-12c). Each guest operating system

Virtual Machine Disk (VMDK) is 250 GB.

C3-OCR1 100 3 x OCR datastores for normal redundancy OCR/voting disk in virtualized database.

Each OCR VMDK is 48 GB. C3-OCR2 100

C3-OCR3 100

C3-DATA1 600 4 x DATA datastores for Oracle DATA disks in the virtualized databases.

Each Data VMDK is 298 GB. C3-DATA2 600

C3-DATA3 600

C3-DATA4 600

C3-REDO1 100


48


Datastore name Datastore size (GB)

Purpose

C3-REDO2 100 2 x REDO datastores for Oracle REDO disks in the virtualized databases.

Each REDO VMDK is 48 GB.

C3-FRA 200 1 x FRA datastore for Oracle FRA disks in the virtualized database

Each FRA VMDK is 98 GB

C3-TEMP

500

1 x TEMP datastore for Oracle TEMP disks in the virtualized databases

Each TEMP VMDK is 248 GB

Total datastores: 12 4,200

Database ASM storage design

The details of the storage design for Oracle RAC databases are based on the design

introduced in Production database storage design and Virtualized databases storage and

ESXi datastore design.

The following table provides the details of the storage volumes that are provisioned for the

production database in this Ready Solution for Oracle.

Table 20. Storage volumes configured for the production Oracle database

Size (GB)

Oracle ASM disk

Oracle ASM disk group

ASM striping

Oracle datafile

50 C2_OCR1 +OCR (normal redundancy)

Coarse striping

OCR files and voting disk files

50 C2_OCR2

50 C2_OCR3

300 C2_DATA11 +DATA (external redundancy)

Coarse striping

Data files, temp files, control files, undo tablespace 300 C2_DATA12

300 C2_DATA13

300 C2_DATA14

100 C2_REDO1 +REDO2 (external redundancy)

Fine-grain striping

Online redo log files.

100 C2_REDO2 +REDO 2 (external redundancy)

Fine-grain striping


200 C2_FRA +FRA (external redundancy)

Coarse striping

Archived redo logs

500 C2_TEMP +TEMP(external redundancy)

Fine-grain striping

Temp files


49


Validation Guide


Validation Guide

The XVC database shares the same storage volumes as the production Oracle database

as shown in the following tables. The only difference is that, for an XVC database such as

XVC DEV, the four DATA volumes (DATA_11_DEV, DATA_12_DEV, DATA_13_DEV,

DATA_14_DEV), two REDO volumes (REDO1_DEV and REDO2_DEV), and one FRA

volume (FRA_DEV) are the XVC snapshot copies of the corresponding volumes of the

OLTP RAC database (PROD). OCR1-3 and TEMP are the regular volumes, not the

snapshot volumes. The same principal applies to the second snapshot database XVC

OLAP.

Table 21. Storage volumes configured for XVC DEV Oracle database

Size (GB)

Oracle ASM disk


ASM striping

Oracle datafile

50 OCR1 +OCR (normal redundancy)

Coarse striping


50 OCR2

50 OCR3

300 DATA11_DEV +DATA_DEV (external redundancy)

Coarse striping

Data files, temp files, control files, undo tablespace 300 DATA12_DEV

300 DATA13_DEV

300 DATA14_DEV

100 REDO1_DEV +REDO2_DEV (external redundancy)

Fine-grain striping


100 REDO2_DEV +REDO 2_DEV (external redundancy)

Fine-grain striping


200 FRA_DEV +FRA_DEV (external redundancy)

Coarse striping

Archived redo logs

500 TEMP +TEMP(external redundancy)

Fine-grain striping

Temp files

Table 22. Storage volumes configured for XVC OLAP Oracle database

Size (GB)

Oracle ASM disk Oracle ASM disk group

ASM striping

Oracle datafile

50 OCR1 +OCR (normal redundancy)

Coarse striping


50 OCR2

50 OCR3

300 DATA11_OLAP +DATA_OLAP (external redundancy)

Coarse striping

Data files, temp files, control files, undo tablespace

300 DATA12_OLAP

300 DATA13_OLAP

300 DATA14_OLAP


50


Size (GB)

Oracle ASM disk Oracle ASM disk group

ASM striping

Oracle datafile

100 REDO1_OLAP +REDO2_OLAP (external redundancy)

Fine-grain striping


100 REDO2_OLAP +REDO1_OLAP (external redundancy)

Fine-grain striping


200 FRA_OLAP +FRA_OLAP (external redundancy)

Coarse striping

Archived redo logs

500 TEMP +TEMP(external redundancy)

Fine-grain striping

Temp files

The following table provides the details of the storage volumes that are provisioned for

each of the virtualized databases in this Ready Solution for Oracle.

Table 23. Storage volumes configured for virtualized Oracle databases

VMware virtual disk

Size (GB)

Oracle ASM disk


ASM striping Oracle datafile

Disk2 298 DATA1 +DATA (external redundancy)

Coarse striping Data files, control files, undo tablespace

Disk3 298 DATA2

Disk4 298 DATA3

Disk5 298 DATA4

Disk6 48 OCR1 +OCR (normal redundancy)

Coarse striping OCR files and voting disk files

Disk7 48 OCR2

Disk8 48 OCR3

Disk9 48 REDO1 +REDO1 (external redundancy)

Fine-grain striping

Online redo log files

Disk10 48 REDO2 +REDO2 (external redundancy)

Fine-grain striping

Online redo log files

Disk11 98 FRA +FRA (external redundancy)

Coarse striping Archived redo logs

Disk12 248 TEMP +TEMP1 (external redundancy)

Fine-grain striping

Temp files


51


Validation Guide


Validation Guide

XtremIO Virtual Copy (XVC) database design

XVC copies from the production database

XtremIO pioneered the concept of integrated Copy Data Management (iCDM) which

allows consolidation of both the primary database and its associated snapshot copies on

the same scale-out all-flash array. With this feature, we can group snapshots of the

volumes of the production database to form multiple copies of the production database.

XtremIO Virtual Copies (XVCs) is XtremIO’s implementation of the snapshot copy

functionality. XVCs are created by capturing the state of data in volumes at a particular

point in time. An XVC snapshot provides the users to access the data in the volume at the

time when the XVC snapshot was taken. To take the cross-consistent XVC copy of the

volumes of the PROD database, we placed all four DATA volumes, two REDO volumes,

and one FRA volume of the PROD database in a consistency group called C2_CAP as

shown in the following figure.

Figure 13. Consistency Group C2_CAP

In this architecture, we took two XVC snapshots of the consistency group C2_CAP: DEV

XVC and OLAP XVC. Creating an XVC snapshot is quick, has no impact on the PROD

database, and imposes no PROD database downtime. The snapshots do not consume

any capacity at the point when created. We create these two XVC copies for different

purposes. DEV XVC forms a development database and OLAP XVC forms an analytics

application database. We created an XVC snapshot copy of the C2_CAP consistency

group by using the Create Repurpose Copy process in the XtremIO Manager UI, as

shown in the following two figures.


52


Figure 14. Create an XVC copy of consistency group GC_CAP

Figure 15. Consistency group C2_CAP and its XVC copies C2_CAP_DEV and C2_CAP_OAP

Snapshots database based on XVC copies of the production database

When the C2_CAP DEV or C2_CAP_OLAP XVC copies are created, we use them to

create the DEV_XVC and DEV_OLAP snapshot databases by mounting these XVC

copies to a database server. The following procedure provides the major steps to create

snapshot databases with the C2_CAP_DEV or C2_CAP_OLAP XVC snapshot copies.

1. Present the XVC snapshot copies to the XVC server and mount them as ASM

disks, as shown in the following table. TEMP, OCR1, OCR2 and OCR3 are the

original volumes, not the snapshots.

Table 24. Volumes and snapshots presented to the XVC server

Snapshot or volume name

Snapshot source volume

Snapshot

/volume Diskgroup Database/clusterware

DATA11_DEV C2_DATA11 Snapshots DATA DEV XVC Database

DATA12_DEV C2_DATA12



REDO1_DEV C2_REDO1 REDO1

REDO2_DEV C2_REDO2 REDO2

FRA_DEV C2_FRA FRA

DATA11_OLAP C2_DATA11 Snapshots DATA OLAP XVC Database

DATA12_OLAP C2_DATA12



53


Validation Guide


Validation Guide

Snapshot or volume name

Snapshot source volume

Snapshot

/volume Diskgroup Database/clusterware


REDO1_OLAP C2_REDO1 REDO1

REDO2_OLAP C2_REDO2 REDO2

FRA_OLAP C2_FRA FRA

TEMP N/A Volume TEMP Shared by DEV XVC and OLAP XVC

OCR1 N/A Volume OCR Clusterware

OCR2 N/A Volume

OCR3 N/A Volume

Note: The entries marked as “Snapshots” are the XVC copies of the PROD database and are

mapped to the XVC server. The entries marked as “Volume” are the actual storage volumes

mapped to the XVC servers. These volumes are TEMP and OCR1, OCR2, OCR3, which are not

based on snapshot copies.

2. Provide new diskgroup names. When the XVC copy takes an XVC snapshot, the

XVC snapshot has the ASM disk name and the ASM diskgroup name as its

ancestor volume. When mounting an XVC snapshot to form a new database,

change its ASM diskgroup name and ASM disk name to avoid duplicating ASM

diskgroup names. The following table shows the new ASM diskgroup names for

these XVC snapshots.

Table 25. ASM diskgroup renaming map

Volume name Original diskgroup name New diskgroup name Snapshot database

DATA11_DEV DATA DATA_DEV DEV XVC

DATA12_DEV

DATA13_DEV

DATA14_DEV

REDO1_DEV REDO1 REDO1_DEV

REDO2_DEV REDO1 REDO1_DEV

FRA_DEV FRA FRA_DEV

DATA11_OLAP DATA DATA_OLAP OLAP XVC

DATA12_OLAP

DATA13_OLAP

DATA14_OLAP

REDO1_OLAP REDO1 REDO1_OLAP

REDO2_OLAP REDO1 REDO1_OLAP


54


Volume name Original diskgroup name New diskgroup name Snapshot database

FRA_OLAP FRA FRA_OLAP

3. Change all the ASM diskgroup names in all the file paths for all the database files,

control files,and redo logs. Also change the ASM diskgroup names in all

references to file paths and in the destinations setting in the spfile. This

modification is necessary because all these file paths refer to the old diskgroup

names that are no longer valid because the ASM diskgroup name has changed..

For example, the following SQL command renames the database file:

SQL>alter database rename file

'+DATA/DBCAP/DATAFILE/users.261.986035293' to

'+DATA_DEV/DBCAP/DATAFILE/users.261.986035293';

4. Change the database name and the DBID with the DBNEWID utility. The

DBNEWID utility enables you to assign the new database name and the new

database ID to replace the original production database name from the primary

database (PROD). Then, change the dbname in the spfile.

5. To open the new database, use the alter database reset logs command.

Disable the archive log mode if the new database is for development and an

archive log is not required.

For detailed steps about how to create the snapshot databases by using XVC snapshot

copies from the PROD database, see the Dell EMC Ready Solutions for Oracle with Data

Protection Deployment Guide.

Data Domain backup system design

During the database backup operation with Oracle RMAN, the Oracle database sends

backups to the Data Domain system through the network, which can be Fibre Channel or

Ethernet. We selected DD Boost over Ethernet protocol to take advantage of the proven

performance and deduplication features of DD Boost technology. In this configuration,

both the DD Boost feature and the distributed segment processing (DSP) are enabled. DD

Boost software runs on both the Oracle database server and the Data Domain system. As

shown in the following figure, for each backed-up segment, the DD Boost software

determines if the segment is unique (has not been previously stored in the Data Domain

system). When DD Boost confirms that the segment is unique, the segment is

compressed and transferred over the network and stored on the Data Domain system.

The deduplication and compression processes ensure that only unique data is

compressed and sent over the network and stored in the Data Domain system.

DD Boost

technology


55


Validation Guide


Validation Guide

Figure 16. Oracle RMAN backup to the Data Domain system with DD Boost software

During the first full database backup, because no data from this database was stored in

the Data Domain system, all the data segments from the backup are unique. Therefore,

each data segment from the first full backup is compressed, sent over the network, and

stored in the Data Domain system. Starting with the second full backup, the DD Boost

software backs up only those unique data segments that were not previously stored in the

Data Domain system.

Figure 17. Show compression output

Two related factors are used to measure the effectiveness of the DD Boost deduplication

and compression features:

Total-Comp Factor = 1054.5/784.2 = 1.3x


56


Reduction % = ((Pre-Comp-Post-Comp)/Pre-Comp)* 100 = (1054.5-784.2)/

1054.5 = 25.63%,

The first result, 1.3x, is the total compression factor achieved by DD Boost deduplication

and compression. The second result, 25.63%, shows that DD Boost technology reduced

the storage usage and network bandwidth by 25.63 percent during the subsequent

backup.

The Data Domain system includes a set of disks that store database backups. During the

initial Data Domain configuration, these disks are assigned to disk groups to be used to

create file systems for storing database backups. For example, the DD6300 system has

one head unit with 14 disks plus one additional disk enclosure (DS60) with 60 disks. As a

default configuration, the disk group dg0 as a base unit is created with 12 disks from the

head unit. That is, 11 disks (1.1–1.10, 1.12) plus one disk (1.11) as a spare disk, for a

total of 40 TiB usable storage capacity that can be used to store the database backup

images.

For more storage, 60 additional storage disks in the disk enclosure DS60 can be added

into DD6300.

During Data Domain system initialization, a file-system-enabling command on the Data

Domain system command line enables the file system. The following command shows

the current space usage of the file system in a DD6300.

Figure 18. Data Domain DD6300 file system space status

When the file system is enabled, the storage space from the disks becomes available. To

perform operations such as backup, restore, or remote replication, create logical volumes

in the Data domain system.

The logical disk volumes created in the Data domain system are called as storage units.

The Data Domain system exposes volumes called as storage units to a backup server

enabled with the DD Boost software.

Create one or more storage units on the Data Domain system to use with the database

application agent on the database server to back up the database files, as shown in the

following example:

Storage and file

system

Mtree and

storage unit


57


Validation Guide


Validation Guide

Figure 19. Storage units

These storage units are shown as a logical partition of the Mtree file system:

Figure 20. Mtrees for the storage units

To implement the Oracle optimized deduplication feature in a Data Domain system, use

the following command to set the value of the app_optimized-compression option to

oracle1 on the Mtree:

mtree option set app-optimized-compression oracle1 mtree

<storage_unit_name>

For example, run these commands in the command line on the Data Domain system for

storage unit slob_unit_7 and slob_unit_10:


/data/col1/slob_unit_7


/data/col1/slob_unit_10

A Data Domain system connects to the Oracle Ready Solution configuration through an

Ethernet network as a backup appliance. The physical connectivity between the Data

Domain system and the Oracle Ready Solution configuration is based on two ports of 10

GbE network interface controllers (NICs) that are installed on the DD6300 system and

four ports of two 10 GbE NICs that are installed on the DD9300 system. For a detailed

description about the connectivity design, see Network design.

We created a network interface group on the Data Domain system by adding these

interfaces to this group.

The following figure below shows that two network interfaces, 172.16.191.30 and

172.16.191.31, were added to the default interface group in the DD6300 system.

IP network

design


58


Figure 21. Interface group in the Data Domain DD6300 system

To register and connect the database server as a client with the Data Domain system,

select the static IP address assigned to one of the interfaces on the Data Domain. By

internally enabling load balance and failover capability among the network interfaces

configured in a group, the interface group configuration provides a high network

bandwidth and a highly available backup network between the database servers and the

Data Domain system.

To increase the RMAN backup/restore throughput, establish a number of parallel backup

channels with the RMAN backup or restore. On an Oracle RAC database system, we can

take advantage of the multiple-instances architecture of the RAC database to scale the

RMAN backup workload by distributing multiple parallel backup channels over multiple

RAC database instances. Multiple backup channels direct the connections to each

instance of an Oracle RAC database.

The PARALLELISM setting in the RMAN backup and restore script defines the total

number of parallel RMAN backup or restore channels. The setting varies depending on

the database size, CPU utilization, backup throughput, and backup time. In general, more

parallel channels can lead to a higher backup throughput with a shorter backup time, but

also require a higher CPU utilization and more network bandwidth.

For example, we used a total of four channels in the first full backup performance test with

two channels connecting to each RAC database instance, as shown in the following

figure.

Multiple channel

backup and

restore with

Oracle RAC


59


Validation Guide


Validation Guide

Figure 22. Multiple channels for Oracle RAC Database backup and restore

Configure Oracle RMAN backup and restore by providing specific parameter settings. For

commercial backup with the DD6300 system, we used the following settings in our backup

and restore tests for all the databases.

Table 26. RMAN backup and restore parameter settings

Operation Parameter Setting

Prod database backup PARALLELISM 4

Prod database backup SECTION SIZE 4 G

Prod database backup BLKSIZE 1048576

Prod database restore PARALLELISM 4

Prod database restore BLKSIZE 1048576

RMAN backup

and restore

parameters

Chapter 5: Test Methodology and Results

60


Chapter 5 Test Methodology and Results


Test objective .................................................................................................... 61

Test tools and methods .................................................................................... 62

Use case 1: Ease of storage provisioning with XtremIO storage .................. 62

Use case 2: Baseline—One production OLTP RAC database ........................ 66

Use case 3: Impact of one 11gR2 and one 12cR2 virtualized database on the baseline ........................................................................... 78

Results Summary: Mixed physical and virtual environments running mixed OLTP and OLAP workloads on Ready Solution for Oracle with XtremIO X2 ......................................................... 83


61


Validation Guide


Validation Guide

Test objective

This Ready Solution for Oracle on XtremIO X2 storage provides simple, easy-to-use

management through an interface that allows storage administrators to provision storage

with little setup or planning. Also, it provides predictable and consistent low-latency

performance with sub-0.85 millisecond response times regardless of the workload and

environment that can include production, QA, test, or development in typical enterprise

applications. Multiple copies might be required for test/development, reporting, or online

analytics.

DBAs and test/dev engineers often spend hours managing database creation and

refreshing the environments while often being limited by capacity, performance, and

number of copies. XtremIO's Integrated Copy Data Management (iCDM) enables you to

create XtremIO Virtual Copies (XVCs) from production instantly with no performance

impact. These copies can be repurposed for near-real-time analytics, test/dev, or any

other use case—all with complete space efficiency.

Protecting the database is easy with XtremIO. There is no need to consider the

complexities of RAID type, data file capacity, or load balancing. The data is protected with

the XtremIO Data Protection (XDP) proprietary flash-optimized algorithm. XDP is different

from RAID in several ways. Because XDP is always working in an all-flash storage array,

several criteria were important in the design of this protection scheme. XDP benefits

include ultra-low capacity overhead, high levels of data protection in case of double SSD

failure, rapid rebuild times, flash endurance, and extreme performance.

In this Ready Solution for Oracle, we use Data Domain to back up and recover an Oracle

database. Data Domain benefits are the same as XDP benefits. XtremIO virtual copies

provide easy protection and recovery from any operational and logical corruption. XVCs

enable the creation of frequent point-in-time copies (according to RPO intervals –

seconds, minutes, hours) and use them to recover from any data corruption. An XVC can

be kept in the system for as long as it is needed. Recovery using an XtremIO virtual copy

is instantaneous and does not impact system performance.

To demonstrate the benefits of this solution, Dell EMC has designed a series of

performance tests using OLTP and OLAP databases. OLTP database usage is

characterized by small requests for information, such as looking up an inventory item or

checking a customer account, and supporting mission-critical back-office applications.

ERP and CRM systems can support thousands of users who generate millions of

database transactions and require fast response times. In this case, response time is the

total amount for the database to respond to a request. For our OLTP tests, we used an

aggressive response time goal of under 0.85 milliseconds to measure success.


62


Test tools and methods

To simulate both OLTP and OLAP database workloads, we used the HammerDB tool

version 3.1. OLTP workload was generated through standard TPC-C like queries from

HammerDB, and the OLAP workload was generated through a custom query run on the

OLAP XVC database. Our tests featured 70 virtual users on a production database and 10

virtual users each on OLTP XVC databases and virtualized databases. We created a

HammerDB dataset with 9,000 warehouses. The OLTP workloads were made up of 65

percent reads and 35 percent writes. The OLAP custom reporting query performs read-

only operations. For detailed HammerDB parameter settings, refer to the HammerDB

configuration parameters section in Appendix A.

Use case 1: Ease of storage provisioning with XtremIO storage

This use case describes the ease of configuring XtremIO X2 storage for Oracle Database.

We created required volumes from XtremIO storage for the Oracle 12c standalone

database installation by considering these design principles:

Created four volumes for DATA to optimize queuing

Created three volumes for OCR for redundancy

Created separate volumes for REDOLOGS for ease in monitoring performance

Created separate volume for FRA because it is not needed for repurposing a

database

Created separate volume for TEMP because it is not needed for repurposing a

database

All these volumes were created from the XtremIO X2 management console. Volume

provisioning in XtremIO X2 storage is simple and straightforward. You can create volumes

and present them to the Oracle database servers with just a few clicks from the XtremIO

X2 management console. To create volumes and map them to the servers:

1. Log in to the XtremIO X2 web management console by providing credentials:

http://www.tpc.org/tpcc/detail.asp


63


Validation Guide


Validation Guide

Figure 23. XtremIO Management login screen

2. In the XtremIO X2 management console, click Configuration, click Volumes,

and then click Create New Volume.

Figure 24. Creating storage configuration volumes

3. Enter the number of volumes, the volume name prefix, the size of the volumes,

and then click APPLY.


64


Figure 25. Enter new storage volume parameters

4. Repeat step 1 through step 3 to create volumes for the Oracle 12c database, as

listed in the following table.

Table 27. Storage volume parameters

Volume Name

Number of volumes


Total size (GB)

C1-OCR 3 50 150

C1-DATA 4 250 1,000

C1-REDO 2 100 200

C1-FRA 1 100 100

C1-TEMP 1 100 100

Total 11 1,550

5. Select all the newly created volumes and click Mapping.


65


Validation Guide


Validation Guide

Figure 26. Select volumes for mapping

6. Select the Initiator Group (Server) to which the volumes will be mapped and click

NEXT.

Figure 27. Select initiator group


66


7. Verify mapping confirmation and click APPLY.

Figure 28. Mapping confirmation

8. After mapping newly created volumes that are mapped to the initiators, execute

rescan-scsi-bus.sh on all hosts to display volumes.

This use case demonstrates the ease of provisioning XtremIO volumes for the Oracle 12c

database and explains step-by-step provisioning of volumes to Oracle 12c database

servers. It takes just a few mouse clicks to provision a volume from XtremIO storage to

the Oracle 12c database.

Use case 2: Baseline—One production OLTP RAC database

In the second use case, we created two Oracle 12c Release 2 RAC databases across two

PowerEdge 940 servers, as shown in the figure below. We used the HammerDB test tool

to generate an OLTP workload with a 65/35 read/write mixture. We created the databases

with an 8 KB block size and with ASM in a coarse-striped and externally redundant

configuration. In this use case, we performed multiple stress tests to understand the

performance and other capacity numbers that are generated under different workloads

such as:

2 nodes RAC Production DB (Test 1)

Test 1 + two SNAP DBs generated from RAC DBs (Test 2)

Test 1 + Test 2 + one 11g Virtualized DB + 12c Virtualized DB (Test 3)


67


Validation Guide


Validation Guide

Test 1

XtremIO X2 storage is designed to unlock flash technology's instant performance potential

by uniquely leveraging the characteristics of SSDs. It uses advanced inline data reduction

methods to reduce the physical data that must be stored. XtremIO's storage system uses

industry-standard components and custom-designed intelligent software to deliver

unparalleled levels of performance, achieving consistently low latency for up to millions of

IOPS.

The XtremIO Management Server (XMS) delivers an HTML5 user interface that is simple

and easy to use for storage administrators. XMS enables storage administrators to

provision storage with little setup or planning as shown in use case 1. In this use case we

created two Oracle 12c Release 2 RAC databases across two PowerEdge 940 servers,

as shown in the following figure. We used HammerDB to create an OLTP workload with a

65/35 read/write mixture. We created the databases with an 8 KB block size and with

ASM in a coarse-striped and externally redundant configuration. The detailed

configuration of the two 12c RAC OLTP databases are documented in use case 1. In this

use case, we performed a series of tests and we will discuss the performance results with

RAC, SNAP, and virtualized databases that are either repurposed or newly created. In

test 1, we already created the RAC OLTP DB as shown in the following figure.

Figure 29. Test 1 architecture

The following table shows the high-level configuration of the two production Oracle RAC

databases.

Table 28. Production Oracle RAC database configuration

Category Specification/setting PROD configuration

Operating system VM guest OS RHEL 7.4

Database configuration Database version 12c R2 RAC

Database size 1 TB

Performance

impact of two

snapshot

databases


68


Category Specification/setting PROD configuration

db_block_size 8 KB

db_file_multiblock_read_count 4

sga_max_size 64 GB

pga_aggregate_target 200 GB

SLOB I/O configuration Read/write ratio 65/35

We ran the two production Oracle RAC databases on dedicated PowerEdge R940 servers

and a dedicated XtremIO X2 array. The goal of this test was to develop and validate

implementation best practices for running Oracle databases on this platform. We

monitored performance. Because the two Oracle 12c RAC databases had dedicated

servers and storage, performance measurements do not reflect the consolidation

capabilities of the database platform. Most customers consolidate databases to achieve

greater capital and operation expenditure savings and gain more value from their

investment in licensing the databases. By generating an OLTP workload, the goal was not

to maximize performance but to create a realistic production workload. Results of the

testing are shown in the following figure.

The average CPU utilization across this test is minimal, which provides significant room

for growth. The Oracle RAC database generated over 120,828 IOPS, demonstrating a

significant production workload.

120,828

Total IOPS

Test 1 - IOPS


69


Validation Guide


Validation Guide

The average latency for the Oracle 12c RAC was .39 for reads and .66 for writes and well

under our target threshold of .85 milliseconds. Results show the XtremIO X2 array

delivered strong storage performance for a production-like workload of 120,828 IOPS.

Transactions per Minute (TPM) measures the number of completed transactions the SQL

Server database is able to complete in one minute and is an indicator that DBAs can use

to evaluate database activity. The two-node Oracle 12c RAC database was able to

process 383,591 TPM in this test, showing the capability to support a large OLTP

workload.

0.39

0.66

Average read latency Average write latency

Test 1 - Latency

383,591

TEST 1 - TPM


70


The following table shows how space savings work. The production database was

provisioned with 1,200 GB of disk volumes on the XtremIO X2 array, and the database

used 628.98 GB physical space after compression. The XtremIO XVC processed the

production copies for OLTP. XtremIO compression in this test has generated the Data

Reduction Ratio of 1.8:1. The overall efficiency ratio is 1.85:1.

Table 29. Different XtremIO ratios depicting the power of deduplication and compression

Parameter Value

Test case number 2

Description Production database with 720 GB data

Number of volumes 4

Databases Production DB

Total volume size (GB) 1,200

Host accessible size (GB) 1,166

Logical used (GB) 1,165.35

Unique physical space (GB) 628.98

Data reduction ratio 1.8:1

Copy efficiency 1.0:1

Thin provisioning ratio 0.97:1

Overall efficiency ratio 1.85:1

Test 2

IT organizations are under increased pressure to update and add new features to

applications quicker and more frequently. To address business demands for faster

updates and new features, many IT organizations look for opportunities to increase

efficiencies by using automation. In this use case, we used the XtremIO XVC feature to

create production copies and repurpose them for development. XtremIO's Integrated

Copy Data Management (iCDM) allows for instant XtremIO Virtual Copies (XVCs) to be

created from production with no performance impact. These copies can be repurposed for

near real-time analytics, test/dev, and any other use case—all with complete space

efficiency, reducing the time it takes to provision a test or development database.

Application and database administrators can repurpose a production copy on demand.

Using XtremIO Virtual Copies (XVCs) accelerates repurposing production copies and

enables the IT organization to fully automate the complex and time-consuming process.

The following figure depicts the architecture used for test 2 of use case 2.

https://www.emc.com/collateral/solution-overview/h12453-xtremio-integrated-data-reduction-so.pdf


https://www.emc.com/collateral/whitepaper/h16753-dell-emc-xtremio-x2-oracle-db-compression-wp.pdf


71


Validation Guide


Validation Guide

Figure 30. Architecture of test 2 of use case 2

For a detailed description of snapshot databases and their parameters, see


72


XtremIO Virtual Copy (XVC) database design. Using the XtremIO XVC to create snapshot

databases, our testing yielded the performance numbers shown in the following figure.

OLAP workloads are used for data mining in which a DBA or developer examines large

data sets to generate new informational insights that assist the business with decisions.

OLAP workloads are characterized by reading large sets of historical data that makes

storage throughput the key metric for OLAP workloads. Both the production OLTP Oracle

RAC database and the Oracle OLAP database where running in parallel. Even with the

increase in workload, the XtremIO X2 array was able drive 1.43 GB/s of throughput.

The production OLTP RAC database performance experienced a minor loss in IOPS.

However, the key metric of latency was consistently strong.


73


Validation Guide


Validation Guide

The overall average storage latency was .44 ms for reads and .65 ms for writes. The

minor increase in read latency is due to the large database table scales in the OLAP

database. The write latency is interesting, as the OLAP workload did not perform many

writes. So the write latency was actually better than in test 1, our baseline. The consistent

latency performance means that application users continue to have good application

response times.

Test 3

In this test, we created one OLTP snapshot database from the existing OLTP production

database. Details of the snapshot OLAP and OLTP databases architecture are shown in

the following figure.

0.44

0.65

Read Latency Write Latency

Overall Average Storage Latencies


74


Figure 31. Architecture of running an OLAP and OLTP snapshot with the PROD RAC database

Based on the architecture shown in the figure, we derive the following performance

statistics, as shown in the following figure.


75


Validation Guide


Validation Guide

In test 3, we added another OLTP workload by repurposing a production snapshot. Our

original production OLTP, OLAP, and a second OLTP workload were running in parallel,

which is a demanding workload for an XtremIO X2 array with 36 flash drives.

120,828

125,305

Test 1 Test 3

Total IOPS

120,828

125,305

Test 1 Test 3

TOTAL IOPS


76


Overall, the XtremIO X2 array delivered 6.4 percent more IOPS from test 1 to test 2.

However, IOPS alone do not provide the full performance picture.

Overall average storage latency was sub-millisecond with .62 ms for reads and .81 ms for

writes. Test results for average write latency slightly exceeded our goal of .85

milliseconds, which was well under the gold standard of 1 ms for all-flash arrays. The

difference in .06 milliseconds (.81 - .85 milliseconds ) most likely does not impact the

application experience for the end user. Therefore, it is our opinion that these latencies

are good considering the incremental increase in workloads and the XtremIO X2

configuration.

0.66

0.81

1 2

Overall Average Storage Latency

Average OLTP Write Latency

383,591

389,341

Test 1 Test 3

Total TPM


77


Validation Guide


Validation Guide

The total Transactions per Minute (TPM) also increased by 1.50 percent from test 1 to test

3. In test 3, the XtremIO X2 array supported 389,341 TPM, showing the capability of the

array to drive greater workloads with a minimal configuration.

These incremental workload tests are designed to push the small XtremIO X2 array to its

limits, while challenging the array to meet or exceed the aggressive goal of .85

milliseconds of latency or less while supporting a mixed workload of OLTP databases and

one OLAP workload. Test results show that the XtremIO array has continually delivered

.85 sub-millisecond latencies with the exception of test 3.

Results show that IOPS and TPM have increased, meaning that the XtremIO array can

consolidate challenging workloads and deliver on performance. Customers can

standardize their database ecosystems on XtremIO X2 storage starting with a small

configuration as a means of minimizing their initial investment. Then, over time, they can

add to the XtremIO configuration to address the continued growth of the database

ecosystem.

This following table shows that when the two snapshots are repurposed, the overall

efficiency, copy efficiency, and data reduction ratio improved considerably. We can see

that by adding two snapshots, the unique physical space usage increased by a small

amount (631.27-628.98 = 2.29 GB). This space is mainly used for storing the Oracle ASM

diskgroup metadata, new control files, and spfiles for the snapshot databases. This result

confirms that creating XtremIO snapshots does not take additional physical space in the

XtremIO storage.



https://www.dellemc.com/en-us/storage/xtremio-all-flash.htm



78


Table 30. Improvement in the XtremIO ratios when two snapshots are added to the production RAC database

Parameter Value Value

Test case number 2 3

Description Production database with 720 GB data

Production database + two snapshot databases with 720 GB data

Number of volumes 4 12

Databases Production DB Production database and two snapshot databases

Total volume size (GB) 1,200 3,600

Host accessible size (GB) 1,166 3,535

Logical used (GB) 1,165.35 1,206.48

Unique physical space (GB) 628.98 631.27

Data reduction ratio 1.8:1 1.9:1

Copy efficiency 1.0:1 2.9:1

Thin provisioning ratio 0.97:1 0.335:1

Overall efficiency 1.85:1 5.6:1


79


Validation Guide


Validation Guide

Use case 3: Impact of one 11gR2 and one 12cR2 virtualized database on the baseline

In this use case, all the OLTP workloads were combined across two PowerEdge R940

servers and one PowerEdge R740 server. The advantages of combining all OLTP

workloads include greater capital and operation expenditure savings, consolidation, and

ease of management.

As shown in the following figure, we ran one production RAC database in parallel with six

snapshot development and OLAP databases plus two virtual 11g and 12c databases for a

total of five mixed workload databases. We used HammerDB to create an OLTP workload

with a 65/35 read/write mixture. We created all the OLTP databases with an 8 KB block

size and with ASM in a coarse-striped and externally redundant configuration. The

following figure shows the details of the architecture of use case 3.

Figure 32. Architecture of use case 3

The OLTP and OLAP VMs were similar in configuration, with 18 more vCPUs and 72

pCPU. At the database configuration level, each development database was configured

with an sga_max_size of 12 GB and the production database was configured with a larger

sga_max_size of 32 GB. The following figure shows the performance of this use case.

For the final and most challenging incremental workload test, we added two virtualized

Oracle databases. One virtualized database was 11gR2 and the other was 12cR2. Both

virtualized databases generated an OLTP workload. Adding virtualized databases to the

workload extends our mixed workload testing to include a mixed platform that includes

both physical and virtual workloads.


80


The test results show a steady increase in IOPS with test 4 generating 131,764 IOPS.

This result is a 5.15 percent increase in IOPS from test 3 to test 4. Recall that in test 3,

our latency for writes exceeded the goal of .85 milliseconds .

125,305

131,764

Test 3 Test 4

Total IOPS


81


Validation Guide


Validation Guide

The overall average read latency was .66 ms and the write was .77 which is slightly higher

than our goal of .85 milliseconds . Both DBAs and storage administrators understand that

workloads fluctuate over time and there is an exceptional range that meets business

SLAs. In this case, there is a minor difference of .02 ms (.77 - .85 milliseconds ) for our

write latency. Latency results across all our tests show consistently low latency times.

0.66

0.77

Average OLTP Read Latency Average OLTP Write Latency


389,341

451,561

Test 3 Test 4

Total TPM


82


The total TPM increased most significantly in performance from test 3 to test 4. In test 4,

we achieved 451,561 TPM, which is a 15 percent increase from the prior test. This test

result shows that XtremIO X2 storage is a strong platform for consolidating OLTP

workloads. Performance trends across all our tests show that IOPS consistently increased

as we added more OLTP workload.

IOPS consistently increased as we added more OLTP workload. The XtremIO X2 array

was able to scale with our database workloads, showing its capability as a database

consolidation platform.

In the following chart, the blue bars indicate test 1, the green bars indicate test 3, and the

purple bars indicate test 4. Overall average storage latency across the three tests was

under .66 ms for reads and .81 milliseconds for writes. More importantly, the XtremIO X2

array delivered consistent sub-millisecond latencies under increasing workloads,

strengthening the conclusion that this array is ideal for database consolidation.

120,828

125,305

131,764

Test 1 Test 3 Test 4

Total IOPS


83


Validation Guide


Validation Guide

0.39

0.660.62

0.81

0.66

0.77

Average OLTP Read Latency Average OLTP Write Latency


383,591

389,341

451,561

Test 1 Test 3 Test 4

Total TPM


84


Finally, the TPM results show the most positive gain for supporting increasing volumes of

transaction. The XtremIO X2 array delivered a significant increase in TPM from test 1

through test 4.

All the results show how XtremIO X2 storage has been designed to support demanding

database workloads. In particular, this XtremIO configuration had 36 flash drives but

supported over 130,000 IOPS at sub-millisecond latencies. From a transactional workload

view, this small all-flash configuration drove over 450,000 TPM.

Results Summary: Mixed physical and virtual environments running mixed OLTP and OLAP workloads on Ready Solution for Oracle with XtremIO X2

We tested mixed physical and virtual environments running mixed OLTP and OLAP

workloads on Ready Solution for Oracle with XtremIO X2 storage. The tests performed in

these use cases demonstrate that you can consolidate all types of Oracle databases with

sub-millisecond latencies and strong throughput. Virtualization combined with XtremIO X2

inline deduplication and compression enable greater consolidation and disk space saving

on this database platform.

Chapter 6: Test Methodology and Results: Data Protection

85


Validation Guide


Validation Guide

Chapter 6 Test Methodology and Results: Data Protection


Test objective .................................................................................................... 85

Use case 1: Full backup of one OLTP RAC database ..................................... 85

Use case 2: Restore and recovery of one OLTP RAC database from full backup ......................................................................................... 88

Data protection testing summary .................................................................... 88


86


Test objective

Traditional ERP and CRM business applications are under constant pressure to protect

and recover data in the shortest possible time. Organizations are greatly interested in

achieving high throughput, low CPU utilization, fast backup and recovery time, low storage

I/O response time, and so on. However, Oracle DBAs and other data center staff are also

interested in achieving high data compression for Oracle backup sets, high network and

storage throughputs, and high storage IOPS during the backup and restore times.

Dell EMC has designed a series of backup and recovery tests by using RAC OLTP

databases on a Data Domain appliance. The backup and recovery of OLTP databases is

of great importance as data stored in OLTP is loaded with a huge volume of organization-

wide transactional or inventory data that supports mission-critical back-office applications.

Faster response and recovery of this data is of particular importance, especially during

longer database downtime.

During this testing we used the Data Domain DD6300 system with DD Boost. This

configuration has yielded superior performance for over 10 years in many challenging

situations.

The Data Domain system with DD Boost software enhances the speed of backup and

recovery with high levels of deduplication and compression. This method accelerates

speed and offers space savings with increased reliability of data restoration and recovery.

In our test environment, we performed multiple tests in different use cases and achieved

impressive results.

Our benchmark for success requires that the backup and recovery time is below 40

minutes for 1 TB of data with higher storage throughput under all circumstances, that

space savings is 28 percent or more, and that the storage throughput is 450 MB/sec or

more.

Use case 1: Full backup of one OLTP RAC database

We performed a full backup of a 1 TB Oracle database by using DD Boost software. DD

Boost software integrates with RMAN and enables host-based deduplication of database

backups to the Data Domain appliance. A full backup eliminates reliance on other

backups, simplifying the management of backups and simplifying restoration after an

unplanned failure.

In this use case, we used the DD Boost appliance to perform the full backup of the

production database. In the tested configuration, we used a LAN connection to the Data

Domain appliance, as shown in the following figure.


87


Validation Guide


Validation Guide

Figure 33. Use case 1 architecture diagram

The first full backup of an Oracle database is entirely unique; thus, all the data is protected

on the Data Domain system. Therefore, the DD Boost software only sends a small subset

of information to the Data Domain system for protection. Although the first full backup is

unique, when the data has been protected on the Data Domain system, it then is

compressed, as shown in the following figure.

Figure 34. Data compression statistics after the backup in Data Domain Appliance

A Data Domain system uses a local compression algorithm developed specifically to

maximize throughput as data is written to disk. The default algorithm (lz) allows shorter

backup windows for backup jobs but uses more space. Two other types of local

compression are available, gzfast and gz. Both provide increased compression over lz,

but at the cost of additional CPU load. Local compression options provide a trade-off

between slower performance and space usage. It is also possible to disable local

compression.


88


The following figure demonstrates a local compression factor savings that is based on the

default algorithm (maximized throughput) on the Data Domain system. There is a

relationship between the amount of unique data and the local compression factor: the

greater the amount of unique data, the more opportunity for compression and the higher

the compression factor. For example, the first backup consists of entirely unique data and

has the largest compression factor. The following figure illustrates the compression.

Figure 35. Database server size and Data Domain backup comparison

This compression saves significant space on the Data Domain system. Dell EMC

engineering test results show that the compression factor was 1.4X: that is a 40 percent

space savings for the full backup. The 1 TB Oracle RAC database was backed up in 36

minutes to the Data Domain system. Because this backup was the first full backup and all

the data is considered unique, the backup time shows the capability of the business to

protect databases quickly with the Data Domain system.


89


Validation Guide


Validation Guide

Use case 2: Restore and recovery of one OLTP RAC database from full backup

Backing up and protecting databases enables recovery from an unplanned failure.

Unplanned failures can represent significant risk to the business by stopping back-office

operations, thus impacting revenue. In this test, we performed a restore from the Data

Domain system backed up to the PowerEdge R740 servers. The goal of this test is to

show a fast restore time of a 1 TB Oracle RAC database that has been protected in the

Data Domain system.

Figure 36. Use case 2 architecture diagram

In this use case, the total recovery time includes restoring the database from Data Domain

using RMAN and opening the database for processing. Restore time alone does not

represent that the database is open and available to the business. In this test, we showed

that a 1 TB Oracle RAC database can be fully recovered from backup in less than 33

minutes.

Data protection testing summary

In summary, this final test shows that the data protection benefits Oracle DBAs by freeing

up CPU resources. In the two use cases described in this chapter, we see that the time of

backup and restore are 36 and 32 minutes and 40 seconds respectively, which are

reasonably low. Also, we see that the storage throughput in these use cases is 482

MB/sec and 571 MB/sec respectively. The following table shows the details.

Table 31. Comparative analysis of backup and restore of Oracle database on XtremIO storage

Parameter name Backup Restore

Size (TB) 1 1

Time (Mins) 36 32.67


90


Parameter name Backup Restore

IOPS 8,060 9,149

Storage throughput (MB/sec) 482 571

Compression factor 1.4X NA

Chapter 7: Conclusion

91


Validation Guide


Validation Guide

Chapter 7 Conclusion


Conclusion ........................................................................................................ 91

Benefits .............................................................................................................. 91

Summary ........................................................................................................... 92


92


Conclusion

Database systems contain the most critical data for companies, therefore, these complex

systems remain in the data center. However, enterprises want solutions with cloud

characteristics such as scalability, performance, consolidation, automation, centralized

management, and protection. Extensive testing of Ready Solutions for Oracle has stress-

tested every component of the system to validate that this database platform delivers

value. From an owner or administrator perspective, management is simplified because

Dell EMC delivers and supports the entire stack. This solution works for every Oracle

ecosystem, and those IT organizations using VMware virtualization or Dell EMC

infrastructure will find this to be a complementary solution that integrates quickly into the

existing data center.

Benefits

Testing of the Ready Solutions for Oracle with XtremIO X2 and Data Domain shows that

the database solution scales well, supports multiple workload types, and enables

aggressive consolidation of the enterprise’s ecosystem. Scalability is essential as

databases grow in size and number over time. In all use cases, the testing shows that

systems can start with just a few databases and support multiple workloads without

impacting the overall database performance.

Today’s consolidated data centers must demonstrate the ability to support multiple types

of workloads. The capability to consolidate types of workloads enables the business to

remove dedicated silos that increase complexity and costs. Testing of the Ready

Solutions for Oracle has proven that mixed workloads can be easily consolidated on this

solution.

Consolidation of databases to fewer servers results in significant savings for the business.

Lower operating and capital expenditures are two possible savings vehicles. Ready

Solutions for Oracle have been verified to support RAC, Snap, and Virtualized databases

in a heterogeneous workload environment. Even the most aggressive testing resulted in

unused resources that could host even more databases.

The use cases included two PowerEdge R940 servers, one PowerEdge R740 server and

an XtremIO X2 array. The following is a review of the results of use cases, where the

workload spans from one production OLTP RAC database to OLTP RAC database, one

each SNAP development OLTP and OLAP standalone databases, and two OLTP

standalone virtualized databases running in parallel:

Simplified and intuitive volume provisioning on the XtremIO X2 array with:

Over 450,000 Transactions per Minute (TPM)

Over 130,000 IOPS with sub-millisecond latencies

Because of the processing power of the PowerEdge servers, CPU utilization was

minimal, leaving room for more databases or for failover of VMs from one ESXi host

to another.

The XtremIO X2 array with inline deduplication and compression saved 1.8X the

flash space, using only 629 GB of capacity for 1.2 GB of logical usable data.


93


Validation Guide


Validation Guide

A snapshot database was created by using just a few clicks with minimal storage

addition and no performance impact. Greater space savings of 1.9X were achieved

through the use of the inline deduplication and compression features of the XtremIO

X2 array.

A full backup of a 1 TB Oracle database took 36 minutes to complete by using a

Data Domain 6300 system. The database size was reduced to 756.7 GB (total

compression size) in the Data Domain system with 28 percent compression (a

compression factor of 1.4X) and storage throughput of 482 MB/sec.

The full recovery of a 1 TB Oracle database took 32 minutes to complete by using

the Data Domain 6300 system with storage throughput of 571 MB/sec.

Summary

This solution is an integrated, validated, and tested database solution. Guesswork,

complexity, and risk are exchanged for faster time-to-value, ease of management and

support, and an engineered system that is specifically designed for Oracle databases. The

solution’s PowerEdge R740 and R940 servers support large database workloads and still

have more than 75 percent unused capacity.

Solution test results show that the XtremIO X2 storage array delivers fast response times,

with latencies under 0.85 milliseconds and throughput to satisfy demanding OLAP and

OLTP databases. Repurposing copies of production to development by using the XtremIO

XVC inline deduplication and compression features delivered 1.8X flash space savings

with the overall efficiency of 5.6. In addition, features such as replication, which are not

discussed in this guide, can provide protection from all types of disasters for Oracle

databases.

Automation is the key to reducing the time devoted to routine database provisioning tasks.

With AppSync software, you can automate the repurposing and protection of databases.

You can repurpose databases on-demand or on a schedule. Either way, the time that is

saved by automating the work can then be invested in more valuable activities.

For data protection of the RAC databases running on XtremIO X2 storage, this solution

attains goals in terms of CPU utilization, database backup/recovery time, and network

throughput. The solution uses inline deduplication and compression to accelerate backup

and recovery activity while reducing bandwidth utilization and backup/recovery time, and

increasing storage/network throughput. The DD Boost software prevents duplicating

backups of similar data, thus reducing the load on the database, storage, and backup

host. The DD Boost software also reduces the frequency of full backups, improves RPO

and RTO, and reduces load on the data center infrastructure.

Dell EMC has an Oracle team that is devoted to customers who are interested in Ready

Solutions for Oracle. Many of these database experts have been working with Oracle for

more than 10 years and understand all the dependencies to ensure your success. Dell

EMC’s Oracle Specialists can size and configure Ready Solutions for Oracle to meet the

needs of your business.

https://www.emc.com/collateral/white-papers/h13163-xtremio-microsoft-sql-server-wp.pdf

Chapter 8: References

94


Chapter 8 References


Dell EMC documentation .................................................................................. 94

VMware documentation .................................................................................... 94

Oracle documentation ...................................................................................... 94

HammerDB documentation .............................................................................. 94

Chapter 8: References

95


Validation Guide


Validation Guide

Dell EMC documentation

The following documentation on DellEMC.com or Dell EMC Online Support provides

additional and relevant information. Access to these documents depends on your login

credentials. If you do not have access to a document, contact your Dell EMC

representative.

Dell EMC Ready Solutions for Oracle with Dell EMC XtremIO X2 and Data Domain

Deployment Guide

Introduction to Dell EMC XtremIO X2 Storage Array

Best Practices for Running Oracle on Dell EMC XtremIO X2

Dell EMC Data Domain Deduplication Storage Systems Spec Sheet

XtremIO Snapshot Refresh for Oracle Database Production, Development and

TEST

VMware documentation

The following documentation on the VMware website provides additional and relevant

information:

VMware ESXi 6.5 Installation and Setup

VMware vSphere 6.5 Installation and Setup

Oracle Databases on VMware Best Practices Guide

Oracle documentation

The following documentation on the Oracle website provides additional and relevant

information:

Oracle Database 12c Release 2 Installation Guide

Oracle Real Application Clusters 12c Release 2 Installation Guide

Oracle Grid Infrastructure Installation and Upgrade Guide

HammerDB documentation

The following documentation on the Hammer website provides additional and relevant

information

Installing and Starting HammerDB on Linux

Configuring Schema Build Options

How to Run an OLTP Workload

http://www.emc.com/

https://support.emc.com/

https://www.dellemc.com/resources/en-us/asset/white-papers/products/storage-2/h16444-introduction-xtremio-x2-storage-array-wp.pdf?isKoreaPage=false&domainUrlForCanonical=https%3A%2F%2Findia.emc.com

https://www.dellemc.com/resources/en-us/asset/white-papers/products/storage-2/h16444-introduction-xtremio-x2-storage-array-wp.pdf?isKoreaPage=false&domainUrlForCanonical=https%3A%2F%2Findia.emc.com

https://www.emc.com/collateral/white-papers/h16919-best-practices-for-running-oracle-on-xtremio-x2.pdf

https://www.emc.com/collateral/specification-sheet/h11340-datadomain-ss.pdf?isKoreaPage=false&domainUrlForCanonical=https://www.emc.com

https://www.emc.com/collateral/white-papers/h14485-xtremio-snapshot-refresh-oracle-databases.pdf

https://www.emc.com/collateral/white-papers/h14485-xtremio-snapshot-refresh-oracle-databases.pdf

http://www.vmware.com/

https://docs.vmware.com/en/VMware-vSphere/6.5/vsphere-esxi-vcenter-server-651-installation-setup-guide.pdf

https://docs.vmware.com/en/VMware-vSphere/6.5/vsphere-esxi-vcenter-server-65-installation-setup-guide.pdf

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/oracle/oracle-databases-vmware-best-practices-guide.pdf

https://docs.oracle.com/en/database/oracle/oracle-database/index.html

https://docs.oracle.com/database/122/LADBI/LADBI.pdf

https://docs.oracle.com/database/122/RILIN/RILIN.pdf

https://docs.oracle.com/database/122/CWLIN/CWLIN.pdf

https://www.hammerdb.com/document.html

https://www.hammerdb.com/docs/ch01s04.html

https://www.hammerdb.com/docs/ch04s03.html

https://www.hammerdb.com/docs/ch04.html

Appendix A: Configuration Details

96


Appendix A Configuration Details

This appendix presents the following topics:

Database performance data collection ............................................................ 96

Database parameters ........................................................................................ 98

HammerDB configuration parameters ............................................................. 98

OLAP query customization ............................................................................ 100


97


Validation Guide

Database performance data collection

We collected the following database performance data through the Oracle database AWR

report.

Based on the 20-minute AWR reports of a test case, the IOPS is the sum of physical

read total I/O requests, per Second, and physical write total I/O requests, per

Second, as shown in the following figure.

Figure 37. Sample of IOPS measurement from AWR report

For an Oracle OLTP-style I/O workload, db file sequential read, the User I/O class wait

is always the top wait event, accounting for most of the wait time. In this example, wait

time averaged 0.443 milliseconds, as shown in the preceding figure.

Note: The db file sequential read events account for single block random I/O calls to the

operating system.

In addition to db file sequential read wait event, transaction redo logging write is another

key performance indicator for Oracle OLTP-style I/O workloads. The following figure

shows the top five timed events section of the AWR report from one of the OLTP

production databases while the workload ran. In this example, wait time averaged 0.244

milliseconds.

IOPS

I/O latency


98


Figure 38. Example I/O Latency measurement from AWR report

For an Oracle OLAP-style I/O workload, the I/O MB/s throughput can be calculated as the

physical read total I/O bytes per second, as shown in the following figure. In this example,

the I/O throughput is 591,947,981.72 bytes per second, or 564.52 MB/s.

Figure 39. Sample of I/O throughput in MB/s from AWR report

The CPU utilization of the database nodes is shown in the OS Statistics By Instance

field of the AWR report, as shown in the following figure.

Figure 40. Example CPU Utilization measurement from AWR report

I/O MB/s

throughput

CPU utilization


99


Validation Guide

Database parameters

To perform an ideal performance comparison, we used the following database

configuration on different types of databases. We used these database parameter settings

to configure production, XVC, and virtualized databases before running the test

workloads.

Table 32. Database parameter settings for OLTP PROD, DEV XVC, OLAP XVC, and virtual databases

Parameter Production DEV XVC OLAP XVC Virtualized (11gR2 and 12cR2)

Database block size 8KB 8KB 8KB 8KB

sga_target 16GB 16GB 16GB 64GB

sga_max_size 32GB 32GB 32GB 64GB

pga_aggregate_target 200GB 200GB 200GB 16GB

open_cursors 1000 1000 1000 1000

filesystemio_options setall setall setall setall

use_large_pages TRUE TRUE TRUE TRUE

resource_manager_plan null null null null

db_file_multiblock_read_count 4 4 16 4

HammerDB configuration parameters

During each test, we kicked off as many HammerDB UI instances as the number of OLTP

databases that were running during that test. During test 1 which involved only the OLTP

database (production), we ran only one HammerDB workload generator UI. During test 3,

when we tested one OLAP database (OLAP XVC) and two OLTP databases (one OLTP

production and one OLTP DEV XVC) in parallel, we ran two separate instances of the

HammerDB workload generator UI to generate load on the two OLTP databases in

parallel.

Note: The OLAP workload on the OLAP XVC database was generated by using a custom OLAP

query that is run directly on the XVC database server. HammerDB was not used to generate the

OLAP workload.

The number of concurrent HammerDB virtual users for each test is shown in the following

table:


100


Table 33. HammerDB Virtual User settings for each OLTP database during each test case

Test case #

OLTP workload OLAP workload

Databases running (names)

Number of HammerDB Virtual Users

Database running (name)

Number of OLAP users

1 db2tp 70 None -

2 db2tp 70 snpolap 1

3 db2tp, snpdev 70, 10 snpolap 1

4 db2tp, snpdev, v11gdb, v12cdb

70, 10, 10, 10 snpolap 1

During each test case, all other HammerDB driver script and Virtual User options were set

as shown in the following table:

Table 34. Common HammerDB parameter settings for all OLTP databases

Parameter Value

Oracle TPC-C Driver Script Options

Total Transactions per User 100,000

TPC-C Driver Script Timed Driver Script (checked)

Exit on Oracle Error TRUE (checked)

Keying and Thinking Time FALSE (unchecked)

Checkpoint when complete TRUE (checked)

Minutes of Rampup Time 5

Minutes for Test Duration 15

Use All Warehouses TRUE (checked)

Time Profile TRUE (checked)

Virtual User Options

User Delay(ms) 500

Repeat Delay(ms) 500

Iterations 1

Show Output TRUE (checked)

Log Output to Temp TRUE (checked)

Use Unique Log Name TRUE (checked)

No Log Buffer TRUE (checked)

Log Timestamps TRUE (checked)


101


Validation Guide

OLAP query customization

To generate an OLAP workload, we used the following query that was run directly on the

XVC database server:

select max(H_AMOUNT),min(H_AMOUNT) from HISTORY where H_DATE >=

'01-JUN-81'

Table 35. Configuration parameters for the OLAP workload

Parameter Value

User Count 1

Run Time 20 min

Note: The custom query did not take 20 minutess to finish. The query was run in a loop for 20

minutes.

Documents

Dell EMC Ready Solutions for Oracle with Dell EMC XtremIO ... · Provisioning XtremIO LUN and presentation with a few clicks in XtremIO WebUI Designed to modernize how databases are