Oracle Best Practices on Datera Storage

TECHNICAL REPORTORACLE BEST PRACTICES ON DATERA STORAGE

1GET A FREE CONSULTATION. Contact Us | Visit datera.io | Email [email protected]

©2020 Datera, Inc. All Rights Reserved. Datera is a trademark of Datera, Inc. All other trademarks belong to their respective owners. Date: July 2020

Oracle Best Practices on Datera Storage

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Target Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Document Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

General Oracle Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Storage layout considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Datera Guidance and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Logical Volume Managers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Volume Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Linux OS Tuning Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14IRQ Vector Affinity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Set CPU Frequency Governor to “Performance” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Set Block Device Queue Scheduler to “noop” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Set Block Device Queue MAX_SECTORS_KB SIZE TO “1024” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Set Network ARP Settings ARP_ANNOUNCE and ARP_IGNORE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Next Please Run the Following for the Changes to Take Effect: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Finally, You Can Verify The Changes With: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Overview of the demonstration environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

AUGUST 2020

https://datera.io/contact-us/

http://www.datera.io

mailto:sales%40datera.io?subject=My%20Free%20Consultation




AbstractOracle is a longstanding global presence in enterprise databases. Datera is a leader in Tier 1, block based, continuously available Software Defined Storage. The Datera architecture is designed provide low latency, and 100% availability. Together, Oracle and Datera provide Global 1000 businesses world class flexibility and scale.

In this document, we introduce, configure, and optimize Oracle Unbreakable Linux for performance and availability with Oracle RAC 19.3 in a VMware environment with Datera storage templates.

Executive SummaryThis document focuses on configuring Oracle RDBMS utilizing Datera software defined storage as the platform for all Oracle required volumes. This paper describes the volume requirements and steps to configure a four node Oracle RAC server. The configuration differences for a one-node Oracle server are also noted when required.

The Datera Data Services Platform runs on cost effective x86 servers and provides high performance, scale-out, application-driven enterprise software-defined block and S3/Object storage for physical servers, virtual machines, and containers. It is architected from the ground up to power the most demanding primary workloads at any scale and deliver a dynamic and autonomous data infrastructure.

Datera can seamlessly and concurrently manage multiple generations of servers and multiple classes of media (HDD, SATA Flash, NVMe Flash, Persistent Memory, such as Optane™, and future technologies) from certified vendors, and it automatically composes data infrastructure that optimizes the requirements of each application by simply letting application owners specify their corresponding application service level objectives (SLOs). When application requirements change, or new server or media technology becomes available, Datera can “live-migrate” data across servers and media, to deliver data infrastructure “as-a-service” with the best performance and economics for every application – across the datacenter, multiple data centers, or ultimately, across private and public clouds.

As a result, through a complete new architectural paradigm, the Datera Data Services Platforms delivers a high performance, dynamic and autonomous data infrastructure that radically simplifies operations and provides unparalleled operational efficiency, flexibility, and economics.

Target AudienceThis guide is intended for Oracle database administrators (DBAs), solution architects, storage engineers, system administrators, and IT managers who analyze, design, and maintain Oracle database environments on Datera Storage Solutions and assumes a minimum working knowledge of Oracle databases, Linux operating systems, and operational understanding.







Document PurposeThis deployment considerations guide describes best practices for deploying and using Oracle databases on Datera Storage Solutions. It offers considerations for database instance tuning and storage layout, and it provides information about OS tuning, clustering, virtualization, copy management, and data protection. This provides proven methods to successfully manage Oracle environments on Datera Storage environments. Following these guidelines will provide the best results when running Oracle RAC on top of the Datera storage solution.

General Oracle ConsiderationsStandard best practices for Oracle with respect to intelligent storage should be fully applied. Additional considerations for running Oracle Real Application Clusters (RAC) databases are provided. Datera uniquely lends itself to supporting the Oracle data bases’ need to have extremely high performance. The intelligent storage enables DBA’s and Storage Administrators to layout the data with automated policies that ensure that the environment is both cost effective, dynamic and highly performant. The guidance for laying out the storage based on the performance requirements vs cost is suggested Figure 1

Figure 1: Configurable Datera Node Types by Performance and Capacity







Storage layout considerationsThe storage layout and considerations are related to the use and context of the service that is being supported by the Database. This document provides specific details about the types of volumes that make up a normal Oracle Database and the configuration and policies that provide the best performance and availability for generic Oracle environments. Please consider your specific environment and cost constraints when building your specific solution.

One of the benefits of using an enterprise solution for the storage Is the visibility and manageability that ensure the best performance and availability for the environment. The following picture provides the universal view that Is provided by the Datera dashboard.

Figure 2: The Datera Dashboard







The following tables describe the different standard Oracle volume types. It also provides an overview of the type of I/O based on normal workloads for the specific type of database. Your database will be different based on the use and the load, but this provides good initial guidelines.

The following picture shows the layout of the Oracle storage as displayed by the Datera dashboard.

The volume set for Oracle will be different based on the specific use case and business needs, but as shown, It Is easy to see the layout and state of the environment using the dashboard.

Table 1 provides the recommended performance levels based on the overall impact to the running system. It is always possible to use the fastest and most expensive storage for all the volumes, but if there are different classes of storage, this provides a roadmap of how to lay out the volumes.

Figure 3: The Oracle Volumes







Table 1: OLTP Database Storage Priorities

Types of Database Files

1st Tier Datera Storage

2nd Tier Datera Storage

3rd Tier Datera Storage

I/O Types and Intensity

Oracle Software 3rd choice 2nd Choice 1st Choice Medium - Exclusively Reads

Grid Software 3rd choice 2nd choice 1st Choice Medium - Exclusively Reads

Control Files 1st Choice with Cache

2nd choice with Cache

3rd Choice with Cache

Medium to High I/O

On-Line Redo Log Files

1st Choice with Cache



High - Sequential Writes

Users Tablespace 1st Choice with Cache



Medium-High Random I/O

Undo Tablespace 1st Choice with Cache



Medium to High I/O

TMP Tablespace 2nd Choice 1st choice 3rd Choice Medium to High I/O

SysAux Tablespace 2nd Choice 1st choice 3rd Choice Medium to High I/O

Other Datafiles on Other Tablespaces




Medium to High I/O

System Tablespace 1st Choice with Cache



Medium to High I/O

Archive Files 1st Choice with Cache



Medium to High I/O

Index Tablespaces 1st Choice with Cache



Medium to High I/O

Other Object Tablespaces




Medium to High I/O

FRA 1st Choice with Cache



Medium to High I/O

RMAN (for Backup and restores)




Medium to High I/O

Backup Stored for Compliance

2nd Choice 3rd Choice 1st Choice Low







Table 2: DSS Database Volumes provide guidelines for DSS Databases: Characterized by a very low volume of transactions but extremely high volumes of very complex queries, returning large amounts of data. OLAP and DSS have sequential I/O reads, demanding a huge throughput. In the Oracle environment, usually a table scan search is used instead of an index.

Table 2: DSS Database Volumes






Oracle Software 3rd Choice 2nd Choice 1st Choice Low to Medium

Grid Software 3rd Choice 2nd Choice 1st Choice Low to Medium

Control Files 3rd Choice 2nd Choice 1st Choice Low

On-Line Redo Log Files 3rd Choice 2nd Choice 1st Choice Low


2nd Choice with Cache


Low to Medium, high throughput for complex queries

Undo Tablespace 3rd Choice 2nd Choice 1st Choice Low to Medium

TMP Tablespace 1st Choice with Cache




SysAux Tablespace 1st Choice with Cache



Medium to High

Datafiles 1st Choice with Cache








Archive Files 3rd Choice 2nd Choice 1st Choice Low to Medium

Index Tablespaces 3rd Choice 2nd Choice 1st Choice Low to Medium






FRA 3rd Choice 2nd Choice 1st Choice Low



2nc Choice with Cache


Low


Yes Yes Low







Table 3 provides guidelines for a database that is a combination of OLTP and DSS.

Mixed Workloads: Characterized by a mixture of OLTP and DSS services. Hot Spots must be identified and placed on the storage with the best performance.

Table 3: Mixed Workload Database Volume Placement






Oracle Software Yes Yes Low to Medium

Grid Software Yes Yes Low to Medium

Control Files 1st Choice with Cache



Low to Medium

On-Line Redo Log Files 1st Choice with Cache



Low to High I/O (High I/O for peak OLTP transactions





Undo Tablespace 1st Choice with Cache




TMP Tablespace 1st Choice with Cache



Low to High (High depending on throughput for the Complex Queries)

SysAux Tablespace 1st Choice with Cache



low to High depending on type of applications served by the TS

Datafiles 1st Choice with Cache



Low to High- If TS is used for OLTP, then Fast Storage Policies are recommended




Low to High












Archive Files 1st Choice with Cache



Medium to High (High for the OLTP workloads)

Index Tablespaces 1st Choice with Cache



Low to High (Fast Storage Policies for OLTP worloads)





Low to High (Hot Spots for OLTP must be identified and fast Storage Policies must be deployed for these tablespaces)

FRA 1st Choice with Cache



low to high (storgage policy will be based on SLAs, usually High for OLTP)





high (large amount of Data that must be backed up within a particular backup window)


Yes Yes Low

(Table 3 Continued)







Table 4 provides a summary of the above. The goal is to maximize IOPS and Minimize Latency for OLTPs concurrent Random I/Os; High Throughput for Sequential I/Os.

Table 4: Summary Table for the volumes

Environment Datafiles Types Storage Policy I/O Types and Intensity

OLTP Index Fast Drives High Concurrent Random I/OSTables

Undo

OLRL/Control

Archive

FRA

TMP Moderate Speed Drives Low to High

Backups Fast Drives High

DSS Index Low to High

Tables Sequential I/Os, high Throughput relying on Cache

Undo Low to Medium

OLRL/Control Low to Medium

Archive Low to Medium

FRA Low to Medium

TMP High

Backups High

Mixed Workloads Identify Files that have Heavy I/O

Heavy Random I/O for OLTP, follow OLTP policies, for DSS/OLAP follow OLAP policies

Low to High







Datera Guidance and ConfigurationThe following section provides the overall guidelines for configuring Datera for Oracle RAC. The appendix contains examples of the actual json files used to configure the storage. What is depicted is a screen shot from the sample configuration showing the volumes.

Figure 4: A screen shot of a large Oracle RAC configuration







The following screen shot is of the details about the shared volumes. These volumes contain the files that are specific to the cluster, like the data. All system will have access to these structures. All the shared data is being accessed across the IP network.

Figure 5: Datera Shared Volumes







Logical Volume ManagersThere are volume managers that can be used, but the testing and demonstration related in this document are based on ASM.

Volume InformationThe volume configuration is based on the I/O characteristics and requirements. Table 5 provides an overview of the normal operational characteristics for the specific volume functions.

Table 5: Information about the amount of volumes

Basic Number of Volumes for Oracle Deployment: Storing the data on different locations for ease of administration by the DBAs

# of Volumes for Single Instance # of Volumes RAC

1 Volume for Oracle Binaries 1 local Volume per instance for the Oracle binaries

1 Volume for Grid Installation 1 for the binaries

2 Volumes 1each per multiplexed copies for Control files and Online redo log groups

At least 2 Volumes - 1each per multiplexed copies for Control files and 1 per Online redo log thread per instance

1 per undo Tablespaces 1 per undo Tablespaces per instance

1 for TMP tablespaces 1 for TMP tablespaces

1 for System Tablespace 1 for System Tablespace

at least one for Indexes at least 1 for indexes

at least one for SysAux 1 for SsysAux

at least 1 for Tablespaces to hold tables at least 1 for Tablespaces to hold tables

1 volume for FRA 1 volume for FRA

at least 2 Volumes for Archive Destination

at least 2 volumes for Archive Destination

1 Volume for GIMR

3 volumes for the OCR and Voting Disks







Linux OS Tuning ParametersWhile Datera can work with default parameters inside the Linux kernel, the following attributes greatly increase optimization for throughput and latency.

The Oracle RAC environment Is made up of four Linux nodes. The following diagram provides the overview of the RAC environment that Is using he underlying Datera storage. Again, It Illustrates the visibility provided by the Datera dashboard.

Each of the systems will need to have the underlying OS configuration values adjusted for best performance.

Adjusting the following values tunes the underlying OS to the best value for Datera:

IRQ Vector AffinityFor clients using the Intel 10GbE iSCSI NIC, we recommend setting affinity for interrupts to all cores.

1. Disable irqbalance on the system; SERVICE IRQBALANCE STOP

2. Run the set_irq_affinity script (also provided in the Intel driver package) to explicitly set IRQ vector affinity to all CPU cores

This allows the interrupt requests to be distributed across all available cores.

Figure 6: Oracle RAC 19.3 Nodes







Set CPU Frequency Governor to “Performance”For 3.x kernels:

Install the cpupowerutils package and run: cpupower frequence-set --governor performance

For 2.6 kernels, you can do the same by interacting with the sysfs filesystem, for example:

1. echo 360000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling _ min _ freq

2. sudo sh -c “echo 1 > /sys/devices/system/cpu/cpu0/cpufreq/ondemand/ignore _ nice _ load”

For more information, refer: http://unix.stackexchange.com/questions/77410/centos-conservative-governor-nice-error

This instructs the CPU to use the highest frequency within the scaling parameters. This is appropriate for production systems.

Set Block Device Queue Scheduler to “noop”For all multipath and iscsi block devices, in /sys/block, set the value in /queue/schedule to noop, for example: for i in `ls -d /sys/block/sd*| grep -v “sd\a” ;̀ do cat $i/queue/scheduler; done for i in `ls -d /sys/block/dm-* ;̀ do echo “noop” > $i/queue/scheduler; done

Verify: for i in `ls -d /sys/block/sd* ;̀ do echo $i; cat $i/queue/scheduler ; done

This sets the I/O to first in first out. This tends to be the best for the software storage based solutions.

Set Block Device Queue MAX_SECTORS_KB SIZE TO “1024”Similarly, for all iscsi and multipath devices, set max_sectors_kb size to “1024”: for i in `ls -d /sys/block/sd*| grep -v “sd\a” ;̀ do echo “1024” > $i/queue/max _ sectors _ kb ; done for i in `ls -d /sys/block/dm-* ;̀ do echo “1024” > $i/queue/max _ sectors _ kb ; done

Verify: for i in `ls -d /sys/block/dm-* ;̀ do echo $i; cat $i/queue/max _ sectors _ kb ; done

This is doubling the default KB size of 512. This is a good generic number.




http://unix.stackexchange.com/questions/77410/centos-conservative-governor-nice-error




Set Network ARP Settings ARP_ANNOUNCE and ARP_IGNOREEdit the contents of /etc/sysctl.conf

Add or modify the following entries: net.ipv4.conf.all.arp _ announce = 2 net.ipv4.conf.all.arp _ ignore = 1

This allows the system to ignore ARP requests

Next Please Run the Following for the Changes to Take Effect:sysctl -p /etc/sysctl.conf

Finally, You Can Verify The Changes With:sysctl -a | grep ipv4.conf.all.arp

VirtualizationThe test systems were configured on a VMware cluster. This configuration will work with any of the other clustering services for virtual systems.

ClusteringThe test environment that is used to validate the recommendations in this document is built using Oracle RAC 19.3.

Overview of the demonstration environmentThis document is based on the configuration and operation of standard Oracle environment. The following are the specifics for the configuration.

ConclusionThis document provides guidance and details about running Oracle RAC on top of the Datera storage solution. Oracle RAC is a complex data base solution, and care and planning are required for success. The performance of the data base can change dramatically based on the performance and availability of the underlying storage. Datera provides the perfect platform to allow for the optimum price to performance and availability storage solution for Oracle RAC. While any implementation will have unique characteristics, this guide provides the overall path to making sure that the Oracle RAC implementation with Datera will be successful.







AppendixExample of JSON for configuration of the volumes

1

{

"configurable": true, "consumable _ from": [ "/root/Marketing"

],

"tenant": "/root/Marketing", "path": "/app _ templates/Oracle-Local",

"name": "Oracle-Local", "descr": "", "storage _ templates": [ {

"path": "/app _ templates/Oracle-Local/storage _ templates/storage-1",

"name": "storage-1", "volume _ templates": [ {

"path": "/app _ templates/Oracle-Local/storage _ templates/storage-1/volume _ templates/volume-3",

"name": "volume-3", "replica _ count": 2, "size": 32, "snapshot _ policies": [], "placement _ mode": "other", "placement _ policy": "/placement _ policies/Platinum"

},

{


"name": "volume-2", "replica _ count": 3, "size": 500, "snapshot _ policies": [], "placement _ mode": "other", "placement _ policy": "/placement _ policies/Silver"

},

{



}

],

"auth": { "path": "/app _ templates/Oracle-Local/storage _ templates/storage-1/auth",

"type": "none", "initiator _ user _ name": "", "initiator _ pswd": "(hidden)",







"target _ user _ name": "", "target _ pswd": "(hidden)", "access _ key": "", "secret _ key": "(hidden)" },

"ip _ pool": "/access _ network _ ip _ pools/default",

"service _ configuration": "iscsi" }

Example of JSON for Oracle RAC Shared Volumes

1 {

"configurable": true, "consumable _ from": [ "/root/Marketing"

],

"tenant": "/root/Marketing", "path": "/app _ templates/Oracle-Shared",

"name": "Oracle-Shared", "descr": "", "storage _ templates": [ {

"path": "/app _ templates/Oracle-Shared/storage _ templates/RedoVol-15",

"name": "RedoVol-15", "volume _ templates": [ {

"path": "/app _ templates/Oracle-Shared/storage _ templates/RedoVol-15/volume _ templates/volume-1",


}

],

"auth": { "path": "/app _ templates/Oracle-Shared/storage _ templates/RedoVol-15/auth",

"type": "none", "initiator _ user _ name": "", "initiator _ pswd": "(hidden)", "target _ user _ name": "", "target _ pswd": "(hidden)", "access _ key": "", "secret _ key": "(hidden)" },


"service _ configuration": "iscsi" },

{

"path": "/app _ templates/Oracle-Shared/storage _ templates/FRA-4",

"name": "FRA-4", "volume _ templates": [







{

"path": "/app _ templates/Oracle-Shared/storage _ templates/FRA-4/volume _ templates/volume-1",

"name": "volume-1", "replica _ count": 2, "size": 32, "snapshot _ policies": [], "placement _ mode": "other", "placement _ policy": "/placement _ policies/Gold"

}

],

"auth": { "path": "/app _ templates/Oracle-Shared/storage _ templates/FRA-4/auth",




{


"name": "FRA-3", "volume _ templates": [ {



}

],





{











}

],





{





}

],











{





}

],





{

"path": "/app _ templates/Oracle-Shared/storage _ templates/EnvVol-1",

"name": "EnvVol-1", "volume _ templates": [ {

"path": "/app _ templates/Oracle-Shared/storage _ templates/EnvVol-1/volume _ templates/volume-1",


}

],

"auth": { "path": "/app _ templates/Oracle-Shared/storage _ templates/EnvVol-1/auth",










{





}

],





{





}

],


"type": "none", "initiator _ user _ name": "", "initiator _ pswd": "(hidden)", "target _ user _ name": "", "target _ pswd": "(hidden)", "access _ key": "", "secret _ key": "(hidden)"







},



{





}

],





{





}

],


"type": "none", "initiator _ user _ name": "", "initiator _ pswd": "(hidden)", "target _ user _ name": "", "target _ pswd": "(hidden)",







"access _ key": "", "secret _ key": "(hidden)" },



{

"path": "/app _ templates/Oracle-Shared/storage _ templates/DataVol-6",

"name": "DataVol-6", "volume _ templates": [ {

"path": "/app _ templates/Oracle-Shared/storage _ templates/DataVol-6/volume _ templates/volume-1",


}

],

"auth": { "path": "/app _ templates/Oracle-Shared/storage _ templates/DataVol-6/auth",




{





}

],


"type": "none", "initiator _ user _ name": "", "initiator _ pswd": "(hidden)",







"target _ user _ name": "", "target _ pswd": "(hidden)", "access _ key": "", "secret _ key": "(hidden)" },



{





}

],





{





}

],


"type": "none",







"initiator _ user _ name": "", "initiator _ pswd": "(hidden)", "target _ user _ name": "", "target _ pswd": "(hidden)", "access _ key": "", "secret _ key": "(hidden)" },



{





}

],





{





}

],

"auth": {







"path": "/app _ templates/Oracle-Shared/storage _ templates/RedoVol-7/auth",




{





}

],





{





}







],





{





}

],





{




"name": "volume-1", "replica _ count": 2, "size": 20, "snapshot _ policies": [], "placement _ mode": "other",







"placement _ policy": "/placement _ policies/Silver"

}

],





{





}

],





{




"name": "volume-1", "replica _ count": 2, "size": 20,







"snapshot _ policies": [], "placement _ mode": "other", "placement _ policy": "/placement _ policies/Silver"

}

],





{





}

],





{




"name": "volume-1",







"replica _ count": 2, "size": 300, "snapshot _ policies": [], "placement _ mode": "other", "placement _ policy": "/placement _ policies/Silver"

}

],





{





}

],





{











}

],





{





}

],





{


"name": "RedoVol-1",







"volume _ templates": [ {



}

],





{





}

],





{











}

],





{





}

],




"service _ configuration": "iscsi"







},

{





}

],





{





}

],











{





}

],





{





}

],


"type": "none", "initiator _ user _ name": "", "initiator _ pswd": "(hidden)", "target _ user _ name": "", "target _ pswd": "(hidden)", "access _ key": "",







"secret _ key": "(hidden)" },



{





}

],





{





}

],


"type": "none", "initiator _ user _ name": "", "initiator _ pswd": "(hidden)", "target _ user _ name": "",







"target _ pswd": "(hidden)", "access _ key": "", "secret _ key": "(hidden)" },



{





}

],





{





}

],


"type": "none", "initiator _ user _ name": "",







"initiator _ pswd": "(hidden)", "target _ user _ name": "", "target _ pswd": "(hidden)", "access _ key": "", "secret _ key": "(hidden)" },



{





}

],





{





}

],










"service _ configuration": "iscsi" }

]

}




Documents

Oracle Best Practices on Datera Storage