Oracle Cloud Referance Design

Copyright 2015 Mokum Solutions, Inc. All rights reserved.Distribution of the Oracle Cloud Cookbook or derivative of the work in any form is prohibited unless prior permission is obtained from the Copyright holder.

About Mokum Solutions, Inc.Founded in March 2011, Mokum Solutions, Inc. specializes in the implementation, delivery and support of Oracle technologies in private and public clouds. Mokum corporate headquarters are located in San Francisco, CA http://mokumsolutions.comor call 1 415 252 9164

About the AuthorThe author of the Oracle Cloud Cookbook is none other than the owner of Mokum Solutions, Inc., Roddy Rodstein. Roddy is one of the most respected Oracle Cloud Computing experts, having designed and managed many of the worlds largest and most complex Oracle private clouds. Before establishing Mokum in March 2011, Roddy spent three years at Oracle on the Oracle VMand Oracle Linux team designing and supporting Oracle's largest and most complex customer environments. Before Oracle, Roddy spent six years at Citrix,designing and supporting Citrix's largest and most complex customer environments,Including Oracle's. With Mr. Rodsteins rich background and knowledge, there can be no better resource for revealing the Oracle Cloud recipe.

AudienceThe Oracle Cloud Cookbook is a comprehensive, field tested reference design thatguides you through each step to move to your Oracle software portfolio to an elasticOracle cloud using the Oracle VM product line, Oracle Linux, Oracle Engineered Systems managed by Oracle Enterprise Manager 12c, with total control over Oracle processor licensing.

Mokum is the only full-time Oracle virtualization integrator with the expertise to help you virtualize your Production, Test and DR Oracle workloads.

[email protected]

Last update: 02/13/2015This chapter of the Oracle Cloud Cookbook presents Mokum's Oracle Cloud reference design. The Oracle Cloud reference designs encompass the software, hardware, storage, network,orchestration and management components required to deploy a scalable, secure, and supportable internal or external Oracle cloud.

Table of ContentsThe Oracle Cloud Reference Design IntroductionThe Oracle Cloud Reference Design Implementation OverviewThe Oracle Cloud Reference Design Support InfrastructureOracle Cloud Architectural DesignOracle VM Hardware ArchitectureOracle VM Server Pool DesignOracle VM Security StandardsOracle VM Manager Security ControlsOracle VM Server Security ControlsVirtual Machine Operating System StandardsOracle VM Disaster RecoveryOracle VM Application IntegrationChange Log

Share on Twitter Share on Linkedin

The Oracle Cloud Reference Design IntroductionThe Oracle Cloud reference design is a eld-tested best-practice standard, designed with simplicity, reproducibility, usability, scalability, supportability and security. The Oracle Cloud reference designs represent a complete Oracle Cloudstandard that can be leveraged as a vanilla solution or modied to more accurately reect organization-specic needs. The Oracle Cloud reference design includes the following categories:

Cloud Delivery Model Converged Infrastructure Management, Orchestration, and Analytics

Infrastructure as a Service IaaSVirtual Machines

Oracle VM for x86x86 64 Servers

Storage & Network Services

Oracle Enterprise ManagerOracle VM Manager

Open SourceOpenStack | Puppet | Katello | ManageIQ | GrayLog2

Note: A detailed explanation of each category and solution in the Oracle Cloud reference design is presented in the architectural overview section.

The Oracle Cloud Reference Design Implementation OverviewThe Oracle Cloud reference design provides a well dened starting point for each Oracle Cloud implementation. It also serves as a baseline upon which all solution additions, revisions, and toolswill be based. As such, there is an increasing value to Oracle Cloud reference design in keeping implementations as close to the reference design as possible.Prior to implementing an Oracle Cloud, its important that an infrastructure assessment (IA) and gap analysis (GA) be performed. During the IA/GA, the architecture of the solution will match thecustomers business needs while maintaining the integrity of the Oracle Cloud reference design. Implementation and support will follow the analysis phase after careful consideration has beengiven to any specic design modications that deviate from the Oracle Cloud reference design.This document outlines the decision points necessary for implementing the Oracle Cloud reference design. For decisions that rely on preexisting factors or specic organizational needs, theappropriate best practice will be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based onorganizational needs, existing architecture, and budget resource availability.The Oracle Cloud reference design is designed to be scalable and resilient for ease of implementation, high availability, and ease of maintenance for internal and external Oracle clouds. Thecomplete solution is made up of multiple architectural components that work together to provide exibility and options for self-service Infrastructure as a Service delivery with broad network access,resource pooling, elasticity, measured service, high availability, security and ease of maintenance. Infrastructure as a Service is the capability to provision and deliver fundamental computing resources as a service tothe consumer (consumer = end users). The Oracle Cloud reference design outlines the decision points necessary for implementing an Oracle VM cloud infrastructure to deliver self-service Infrastructure as a Service usingpre-congured virtual machine templates from the Oracle Enterprise Manager Cloud Control 12c self service portal.

The next Figure shows a high-level overview of the Oracle Cloud reference design components.

The Oracle Cloud reference design isolates Oracle VM server pools into the following four security domains:Controlled: A controlled security domain is used to restrict access between security domains. A controlled security domain could contain groups of users with their network equipment or ademilitarized zone (DMZ).Uncontrolled: An uncontrolled security domain refers to any network not in control of an organization, such as the Internet.Restricted: A restricted security domain can represent an organizations production, test and development networks. Access is restricted to authorized personnel, and there is no directaccess from the Internet.Secured: A secured security domain is a network that is only accessible to a small group of highly trusted users, such as administrators and auditors.

Note: The classication of security domains is very similar to data classications. FIPS PUB 199 is the Standards for Security Categorization of Federal Information and Information Systems. FIPSPUB 199 can be used to determine the security category of systems and within which security domain systems should reside.

The Oracle Cloud Reference Design Support InfrastructureSupport is an integral part of the Oracle Cloud reference design and includes a combination of Oracle support agreements and on-site and o-site support from the implementing party.Administrators will have several options for support, including live assistance, phone support, and forums.This table outlines the decision points for the support infrastructure for the Oracle Cloud reference design. For decisions that rely on preexisting factors or specic organizational needs, theappropriate best practice will be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based onorganizational needs, existing architecture, and budget resource availability.Decision Point Decision Justication

http://mokumsolutions.com

Mokum Solutions, Inc. +1 415 252-9164 4 of 14

Oracle Support Agreements Oracle Support Agreements for the Oracle technologies will be active andup to date.

Support is an integral part of every successful IT project. Oracle supportagreements are necessary to be able to create and manage service requestsas well as to be able to receive software patches and updates from OracleEnterprise Manager and My Oracle Support.

On-site and O-site support On-site and o-site support from the implementing party will be used formaintenance, site reviews, upgrades, and security audits.

On-site and o-site support from the implementing party for problemresolution, system maintenance, site reviews, upgrades, and security auditsaugments the Oracle support agreement and internal IT operations sta.

Oracle Cloud Architectural DesignThe following sections provides the decision matrices for the Oracle Cloud reference design. Implementers of the Oracle Cloud reference design can use the decision matrices as quick referenceguide to identify settings and conguration decisions to be implemented in the environment. These decisions should be carefully analyzed during a gap analysis phase.

Oracle VM Hardware ArchitectureThe server hardware for your Oracle VM environment is a critical component in the success of your Oracle cloud project. The rst step in selecting an Oracle VM hardware platform is to size theserver hardware, followed by calculating the total number of servers required to be in each Oracle VM server pool. The formula to calculate Oracle VM server sizing is: The total aggregate virtualmachine CPU, RAM and Storage requirements plus your N+x availability requirements provides the total server count along with the hardware requirements.Oracle VM server sizing is calculated by adding the aggregate CPU, RAM and storage requirements for all of the virtual machines that could run on an Oracle VM server, and then selecting serverhardware with ample CPU, RAM and storage resources. Once the server hardware has been selected, the number of servers in a server pool is calculated by selecting enough servers to supportthe aggregate CPU, RAM and storage requirements of all of the virtual machines within a server pool, including the number of additional servers for availability, i.e. HA, Live Migration andDistributed Resource Scheduling (DRS). Oracle VM server pools that use HA, Live Migration and DRS must have excess CPU and RAM capacity for hardware failures and virtual machinemigrations. The number of network interfaces for an Oracle VM server is determined by the network switch VLAN setup and the total number of Oracle VM management network ports, and thevirtual machine network ports for your environment.Oracle VM server can be installed on an x86 64 bit server with up to 900 CPU cores or threads, up to 6 TB of RAM, with no limit on the number of network ports. Oracle VM server can be installedon as little as a 3.2 G partition or disk. Since Oracle VM server can be installed on as little as 3.2 GB of disk, many customers use small ash storage modules or boot from SAN to reduce costs andcomplexity.Tip: I have had the opportunity to support and benchmark Oracle VM server installations on slow single 4 GB SSD Drives (18 MB/second Read Transfer Rate,17 MB/second Write Transfer Rate)as well as Oracle VM server installations using local 7k, 10k and 15k disks. The read and write performance from either type of Oracle VM server installation disk on the virtual machine repositirystorage (SAN, NFS, iSCSI, or local disk) from the Oracle VM server and the virtual machines was identical. The disk speed from the Oracle VM server installation does not aect the virtual machinerepositiry storage (SAN, NFS, iSCSI, or local disk) read and write performance.The next table shows the maximum number of CPUs, RAM and NICs for Oracle VM server release 3.2.x, and 3.3.x.Item 3.2 Maximums 3.3 MaximumsCPU Cores or Threads 160 900RAM 4 TB 6 TBNICs 40 No limit

Oracle VM Server CPU, RAM and storage hardware sizing is calculated by determining the total number of virtual machines CPU, RAM, and storage (I/O and disk) requirements per Oracle VMserver. For example, if a single virtual machine with 16 CPUs, 128 GB RAM, 1 TB of disk space with 1500 IOPS will run on one Oracle VM server, the Oracle VM server hardware should have atleast 16 CPU cores or threads, 130 GB RAM, 1 TB of disk space and the ability to support 1500 IOPS with local or remote storage. If two virtual machines each with 16 CPUs, 128 GB RAM, 1 TB ofdisk space with 1500 IOPS will run on one Oracle VM server, the Oracle VM server hardware must have at least 32 CPU cores or threads, 300G RAM, 2 TB of disk space and the ability to support3000 IOPS with local or remote storage.A single Oracle VM 3.2.x server can support up to 160 CPU cores or threads, 4 TB of memory with local or remote storage. An Oracle VM server with 4 TB of RAM and 160 CPU cores or threadscould allocate the majority of the 4 TB of RAM and more than 160 CPU cores or threads to running virtual machines. Oracle VM server supports CPU oversubscription. CPU oversubscriptionmeans that an Oracle VM server with 160 CPU cores could overallocate the total number of CPU cores to virtual machines. Oracle VM server does not support memory oversubscription, whichmeans that an Oracle VM server with 4 TB of RAM cannot overallocate RAM to virtual machines. By default, each Oracle VM server reserves 512 MB of RAM for Oracle VM server (dom0). Theaverage memory overhead for each running virtual machine on an Oracle VM server is approximately 20 MB plus 1% of each virtual machine' memory allocation. The remaining RAM can beallocated to virtual machines.A best practice is to avoid oversubscribing CPU-bound workloads such as the Oracle Database. CPU oversubscription with CPU-bound workloads negatively aects the performance andavailability of an Oracle VM server along with all of the virtual machines running on the server. CPU oversubscription for non-CPU-bound workloads, such as Oracle Fusion Middleware products,is highly recommended. It is common to oversubscribe CPU cores 3-to-1 with non-CPU-bound workloads. For example, one CPU core could be allocated to 3 virtual CPUs for non-CPU-boundworkloads without a performance penalty.Note: Virtual machines cannot aggregate CPU and memory resources from more than one Oracle VM server. That is, a virtual machine consumes resources only from the Oracle VM server wherethe virtual machine is running.Oracle VM has two high-availability features, HA and Live Migration. Oracle VM HA and Live Migration along with Distributed Resource Scheduling (DRS) must be considered to calculate thetotal number of servers required to respond to hardware failures and virtual machine migrations.The next Figure shows Oracle VM server pool designed with excess CPU and RAM capacity to be able to use HA, DRS and Live Migration. Excess CPU and RAM capacity is a requirement for HA,DRS and Live Migration.

This image shows anOracle VM server poolwith excess capacity ableto use HA, Live Migrationand DRS.

This image shows anOracle VM serverpool responding to a HAevent, with DRS and/orLive Migration movingrunning virtual machines.

This image shows anOracle VM serverpool migrating runningvirtual machines using DRSand/or Live Migration.

Oracle VM HA automatically restarts virtual machines when an Oracle VM pool member fails or restarts. Oracle VM HA minimizes unplanned downtime by restarting virtual machines when anOracle VM server fails or restarts. Live Migration is used to eliminate planned downtime by migrating running virtual machines from one Oracle VM pool member to another during a maintenanceevent, for example, for repairs or an upgrade. DRS is an Oracle VM feature which provides policy based real-time utilization monitoring of Oracle VM servers with the goal to distribute virtualmachine loads across a server pool. DRS migrates virtual machines from heavily utilized Oracle VM servers to less utilized Oracle VM servers. Both HA, Live Migration and DRS require a serverpool with at least three servers with excess CPU and RAM capacity to be able to run and migrate virtual machines across the server the pool even if one Oracle VM servers fails.



Decision Point Decision JusticationCertication The server hardware must be jointly supported by the hardware vendor and

Oracle.Note: The following link is the Oracle' hardware certication page.http://linux.oracle.com/pls/apex/f?p=117:1:5773793518142288::NO:RP::

Only jointly supported hardware product receive vendor support whenproblems occur and service tickets are created. The server hardware mustbe jointly supported by the hardware vendor and Oracle.

CPU Server hardware will be ordered with two socket Intel or AMD multiple-coreCPUs for small and medium workloads and four socket multiple-core CPUsfor large CPU-bound workloads.

The Maximum Number of CPU cores or threads an Oracle VM server cansupport is 160. Oracle VM server maps a virtual CPU to a hardware threadon a CPU core in a CPU socket.Oracle VM Server supports CPU oversubscription. CPU oversubscriptionallows an Oracle VM Server with 160 CPU cores to overallocate the totalnumber of CPU cores to virtual machines. For example, a server with anIntel Xeon processor 5600-series CPU with hyperthreading can have up tosix cores and twelve threads per socket. A two socket server with an IntelXeon processor 5600-series CPU could allocate twenty four virtual CPUswithout oversubscribing the physical CPUs.CPU-bound workloads, such as Oracle Databases, should not be on OracleVM Servers with oversubscribed CPUs.

RAM Server hardware will be ordered with the maximum amount of physicalmemory.Note: Oracle VM Server supports up to 4TB of RAM.

Oracle VM Server does not support memory oversubscription. For example,an Oracle VM Server with 1TB of RAM cannot overallocate RAM to virtualmachines. By default, each Oracle VM Server reserves 512MB of RAM fordom0. The average memory overhead for each running guest on a dom0 isapproximately 20MB plus 1% of the guests memory size. The remainingphysical RAM can be allocated to guests.An Oracle VM Server in a server pool with Live Migration, DRS, DPM and/orHA must have excess RAM capacity to accept virtual machines from a LiveMigration, DRS, DPM and/or HA operation. Oracle VM pool memberswithout available RAM can not support Live Migration, DRS, DPM and/orHA. Having available RAM on each server provides exibility in terms ofadding new virtual machines to the server pool, and to allow Live Migration,DRS, DPM and/or HA within a server pool.

Storage Unless the Oracle VM server is booting from SAN, redundant SSD internalhard drives are recomended.Virtual machine image and conguration les are hosted on shared SAN,iSCSI, or NFS repositories.

Oracle VM Server requires only 3 GB of local storage for the entire OracleVM Server installation. The design goal for Oracle VM is to support multiplenode Oracle VM Server pools with shared bre channel SAN, iSCSI and/orNFS storage.Oracle VM supports local storage without HA or Live Migration. With localstorage, the OCFS2 virtual machine le system must be on a dedicated nonSAS hard dirve. For example, a partition on same disk as Oracle VM serverinstallation is not supported. Local SAS storage for virtual machines is notsupported.

Network Interface Cards A minimum of one Ethernet network interface (NIC) card is required just toinstall Oracle VM Server, although at least four or more 10G NICs isstrongly recommended. NIC bonding with port-based VLANsand/or 802.1Q tag-based VLANs are supported and congured post OracleVM Server installation with Oracle VM Manager or Enterprise Manager.

Oracle VM 3.0.1 through 3.1.1 supports two NIC ports per networkbond, and a total of ve network bonds per Oracle VM Server. Oracle VM 3.2.x supports four NIC ports per network bond, and a totalof ten network bonds per Oracle VM Server. Oracle VM 3.3.x supports an unlimited number of NICs, and bonds.

The exact number of network interfaces for an Oracle VM Server entirelydepends on your organizations business requirements, server hardware,and network and storage infrastructure. For example, there are no NIClimitations with a Cisco UCS hardware, in contract to legacy hardware withphysical NICs. Cisco UCS supports provisioning as many HA enabled vNICsas necessary to meet the most demanding Oracle VM networkrequirements, in contrast to legacy hardware that could require up to 6 10GNICs, or 12 or more 1G ports. It is hard to succeed without a plan. Plan yourOracle VM project in advance before ordering hardware, and deploying

Both 802.3AD NIC bonds, port-based VLANs and/or 802.1Q tag-basedVLANs are supported and congured post Oracle VM Server installationwith Oracle VM Manager. Network redundancy, i.e. 802.3AD NIC bondingdoubles the number of required NICs.Oracle VM uses a total of ve discrete networks; Server Management,Cluster Heartbeat, live Migration, Storage and Virtual Machines. All venetworks can be supported using one or more 802.1Q tag-based VLANs (2NICs) or using up to ve 802.3AD bond (10 NICs).Each Oracle VM server pool should have a discrete network for the ServerManagement, Cluster Heartbeat, Live Migration, Storage and VirtualMachines. Isolating the Server Management, Cluster Heartbeat, LiveMigration and Storage networks protect the server pool from unexpectedserver reboots by eliminating OCSF2 heartbeat interruptions that couldcause a pool member to loose network connectivity, fence from the pool andreboot.Each Oracle VM Server will be assigned a unique IP address on the ServerManagement, Cluster Heartbeat, live Migration and Storage network.

Tip: There is a known limitation with OCFS2 two node cluster and network failures that cause the node with the higher node number to self-fence. For example, with a two node Oracle VM serverpool, if one node has a network failure that triggers a HA event, both Oracle VM server will reboot. A best practice is to use a minimum of three Oracle VM servers for a server pool to eliminatethe two node OCFS2 limitation.Oracle VM HA monitors the status of each server pool member using a network and storage heartbeat. If a server pool member fails to update or respond to network and/or storage heartbeatsdue to hardware failure, the server pool member is fenced from the pool, promptly reboots, then all HA-enabled virtual machines are restarted on a live node in the pool. Oracle VM does notsupport memory oversubscription, which means that an Oracle VM server pool must have sucient RAM capacity to be able to respond to a hardware failure using HA, or to support virtualmachine migrations.The Oracle VM Live Migration and DRS move running virtual machines between server pool members across a LAN without loss of availability. Live Migration and DRS have two primary usecases. The rst use case is to eliminate planned downtime by Live Migrating running virtual machines from one server pool member to another during planned maintenance events. The seconduse case is to use DRS policies to load balance running virtual machines from heavily utilized Oracle VM servers to less utilized Oracle VM servers. Since Oracle VM does not support memoryoversubscription, an Oracle VM server pool must have available RAM capacity to be able to migrate virtual machines between servers.DRS is an Oracle VM feature which provides policy based real-time utilization monitoring of Oracle VM servers with the goal to distribute virtual machine loads across a server pool. DRS migratesvirtual machines from heavily utilized Oracle VM servers to less utilized Oracle VM servers.The exact number of network interfaces for an Oracle VM server is determined by the network switch VLAN setup and the number of Oracle VM management and virtual machine network ports.Oracle VM supports both 802.1Q trunk port VLANs as well as port based VLANs, with Linux bonding Modes 1 (Active-Backup), 4 (802.3ad) and 6 (Adaptive load balancing). 802.1q trunk ports canhave two or more VLANs per port, in contrast to port based VLANS that are limited to one VLAN per port or port channel. 802.1Q uses fewer network switch ports and fewer Oracle VM serverNICs compared to port based VLANs that require a dedicated switch port and NIC per network. A network switch VLAN conguration must rst be selected to be able to calculate the exactnumber of network switch ports and NICs for your Oracle VM servers.Oracle VM uses a total of ve discrete networks for the Oracle VM server management functions; server management, cluster heartbeat, live migration, storage (only for NFS and iSCSI) andvirtual machines. Each Oracle VM server pool should have a discrete network for each of the ve aforementioned server management networks, as well as a discrete network for each virtualmachine network. For example, an Oracle VM Server on a 1-gigabit copper network with NFS or iSCSI storage could easily use 12 or more bonded NICs with access ports just for the servermanagement networks and one virtual machine network. In contrast to the latter 1-gigabit copper network example, an Oracle VM Server on a 10-gigabit ber network using 802.1q trunk portswith NFS or iSCSI storage could easily use up to 4 bonded ports just for the server management and 2 bonded ports for the virtual machine networks.Tip: In an clustered Oracle VM server pool, the loss of network connectivity for the Oracle VM cluster heartbeat network will causes a HA event. When a HA event occurs, the Oracle VM serverthat loses cluster heartbeat connectivity is fenced from the server pool and reboots, then all HA-enabled guests are restarted on a live Oracle VM pool member.Prior to implementing an Oracle Cloud, its important that an infrastructure assessment (IA) and gap analysis (GA) be performed. During the IA/GA, the hardware specications will be matched tothe customers business needs.This table outlines the decision points for the for Oracle VM for x86 server hardware. For decisions that rely on preexisting factors or specic organizational needs, the appropriate best practicewill be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on organizational needs,existing architecture, and budget resource availability.



Oracle VM.NAME Rate(bit/s) Rate(byte/s)Gigabit Ethernet 1 Gbit/s 125 MB/s10 Gigabit Ethernet 10 Gbit/s 1.25 GB/sInniband DDR 16 Gbit/s 2 GB/sTip: One thing to consider is NIC rmware levels between bonded internalNIC ports and PCI NIC ports. Consider only bonding internal NICs withinternal NICs and PCI NICs with PCI NICs.

Host Bus Adapter Cards SAN Storage: At least 2 Host Bus Adapter Cards (HBAs).NAME Line-Rate Throughput MBps 4GFC 4.25 800 8GFC 8.5 1600 10GFC 10.52 2550 16GFC 14.025 320020GFC 21.04 5100

2 HBAs are used to eliminate a single point of failure.

Decision Point Decision JusticationOracle VM Server Pool Design Prior to implementing an Oracle Cloud, its important that an infrastructure

assessment (IA) and gap analysis (GA) be performed. During the IA/GA, thearchitecture of the solution will be matched to the customers businessneeds.

Server pool design is a strategic, architectural security decision. Serverpools can be used to control Oracle licensing costs (hard and softpartitioning) and as a way to implement security domains, defense in depth,the principle of least privilege and compartmentalization of information.

Oracle VM Manager The Oracle VM Manager installer provides two installation options. OracleVM 3.0.1 up to 3.1.1 oers a Demo or Production installation. Oracle VM3.2.1 and above oers a Simple or Custom installation. Oracle VM Manager will be installed in Production, Simple or Custom modeon a dedicated physical or virtual server. Production and Custom mode usesa local or external Oracle 11g Enterprise or RAC database on a dedicatedphysical or virtual server. Simple mode uses a local MySQL database. The Oracle VM Manager Database repository will not be shared with otherproduction or test databases on the same server.The Oracle Enterprise Manager Agent and the Virtualization plug-in will beinstalled to enable Oracle Enterprise Manager integration.

For large environments (>33 hosts), the Oracle VM Manager Databaserepository should be on dedicated virtual or physical servers. If your OracleVM environment starts out small and scales out, make sure to have a plan toscale up Oracle VM Manager with more RAM and CPUs and scale out theOracle VM Manager Database repository on dedicated virtual or physicalservers with RAC.For the Oracle VM Manager Database repository, scaling out means movingfrom a single server Database to a multi node RAC cluster. An importantconsideration when scaling out an Oracle VM Manager environment is todetermine if the underlying hardware where the Oracle VM ManagerDatabase repository runs is capable to transition to RAC. If the hardware isnot capable to transition to RAC, it is possible to move and/or export theOracle VM Manager Database repository to a dierent system with moreresources.

Monitoring and Alerting The Oracle VM product family; Oracle VM Server, Oracle VM Manager,virtual machines, Oracle VM Templates and Assemblies can be managed andmonitored with Oracle VM Manager and Oracle Enterprise Manager 12c.Unlike Oracle VM 2.x, which could only be managed by Oracle VM Manageror Oracle Enterprise Manager, not both, Oracle VM 3 and above can bemanaged simultaneously by Oracle VM Manager along with OracleEnterprise Manager 12c Cloud Control. Oracle VM Manager is a stand-alone management solution for Oracle VM,with limited monitoring and alerting functionality. Oracle VM is adefault Oracle Enterprise Manager 12c feature that provides Infrastructureas a Service (IaaS), Database as a Service (DaaS), Platform as a Service(PaaS) and Testing as a Service (TaaS) provisioning with a self-serviceportal. Oracle VM should be enabled in Cloud Control by installing anOracle Management Agent with the Virtualization plug-in on a managedLinux target with Oracle VM Manager. Once Oracle VM is enabled in CloudControl, Oracle VM Manager, Oracle VM Servers, and all the virtualmachines can be managed, and setup with performance monitoring prolesand alerts that can be used for root cause and statistical analysis.A central log host should be congured to capture the Oracle VM Server,the Oracle VM Manager, and the virtual machine operating systems logles.

When things go wrong within an Oracle VM server pool, being able toquickly determine the root cause of an issue can eliminate or reduce downtime. The most eective way to identify problems with an Oracle VM serverpool is to analyze the Oracle VM Manager, the Oracle VM Servers, and thevirtual machines performance statistics, and log les using OracleEnterprise Manager, SNMP based monitoring solutions, and a central loghost.

Network Time Protocol (NTP) With Oracle VM, accurate time is essential to maintain system stability dueto time-sensitive cluster transactions between Oracle VM Servers. Withoutaccurate time, Oracle VM clusters can be brought to a complete standstill.By default, Oracle VM Servers (up to Release 3.1.1) that are discovered byOracle VM Manager are congured to use the Oracle VM Manager host asthe upstream NTP time host. A best practice is to set-up the Oracle VMManager hosts as the upstream NTP time host to synchronize with upstreamCoordinated Universal Time (UTC) sources as well as provide time servicesto Oracle VM Servers.

With Oracle VM, accurate time is essential to maintain system stability dueto time-sensitive cluster transactions between Oracle VM Servers. Withoutaccurate time, Oracle VM clusters can be brought to a complete standstill.A best practice is to set-up the Oracle VM Manager hosts as the upstreamNTP time host to synchronize with upstream Coordinated Universal Time(UTC) sources as well as provide time services to Oracle VM Servers.

Oracle VM Server Pool DesignOracle VM uses the concept of a "server pool" to group together and centrally manage one or more server pools with up to 32 Oracle VM servers. If more than one location exists, Oracle VMserver pools may be dispersed to dierent locations. Oracle VM Manager with Oracle Enterprise Manager 12c provide a single point of administration for one or more dispersed Oracle VM serverpools.Oracle VM server pools can accommodates organization-specic needs, i.e., Oracle technology license management (hard and soft partitioning) , defense in depth, the principle of least privilege,compartmentalization of information, security domains and dierent applications and their performance, authentication, and security requirements.The next Figure shows a high-level overview of how server pools can be used to implement security domains, defense in depth, the principle of least privilege and compartmentalization ofinformation.

This table outlines the decision points for an Oracle VM server pool. For decisions that rely on preexisting factors or specic organizational needs, the appropriate best practice will be discoveredin the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on organizational needs, existing architecture,and budget resource availability.



Oracle Linux and Red Hat Enterprise Linux ship with a default /etc/ntp.confle that points to three of Red Hat's upstream public UTC time sources. Abest practice is to have two internal NTP servers on your local network toprovide time services for internal systems and devices. Using internal timeservers normalizes system event time-stamps across the Enterprise as wellas reduces NTP Internet bandwidth usage.

Oracle VM Server Agent Roles Oracle VM Manager facilitates centralized management of server pools andtheir resources using an agent-based architecture. When an Oracle VMserver is added to a server pool, up to three Oracle VM agent roles can beenabled. There are a total of three Oracle VM agent roles; 1) Master Server,2) Utility Server and 3) VM Server. When an Oracle VM server is added to aserver pool, it can be assigned one, two, or all three of the agent roles.

Master ServerThe Master Server is the principal server pool role within a serverpool. The Master Server is the server that communicates with OracleVM Manager. The Master Server dispatches commands received fromOracle VM Manager to other servers within a server pool. There canbe only one Master Server in a server pool at any instant. The VirtualIP feature is a mandatory server pool property that detect the loss ofthe server pool master agent and responds with automatic failover tothe rst pool member that can lock the pool le systsm. The serverpool Virtual IP feature removes the single point of failure (SPOF) forthe server pool master agent role.

Utility ServerThe Utility Server role is responsible for I/O-intensive operations suchas virtual machine creation and removal, as well for as creating,deleting, modifying, copying and moving virtual machine les.The Master Server dispatches operations to Utility Servers. There canbe one or more Utility Servers in a server pool. When there aremultiple Utility Servers in a pool, the server Master Server will selectthe least loaded utility server to conduct a task.Tip: Oracle VM environments that dynamically grow should have oneor more dedicated Utility Servers to isolate I/O jobs to Utility Servers.For example, an I/O intensive that runs on an Oracle VM Server withthe VM Server and Utility Server role will impact the performance ofall of the virtual machines running on the Oracle VM Server.

VM ServerServers with the VM Server role are responsible for allocating CPU,memory, and disk resources to the virtual machines in a server pool.There can be one up to 32 VM Servers in a server pool.

Master ServerBy default each clustered server pool has one Master Server with theVirtual IP feature enabled. Utility ServerThe Utility Server role is responsible for I/O-intensive operations such asvirtual machine creation and removal, as well for as creating, deleting,modifying, copying and moving virtual machine les. Enabling the UtilityServer agent role with the VM Server role on the same Oracle VM servermay negatively aect running virtual machines during Utility Serveroperations. Server pools that are not static and support the self serviceportal in Oracle Enterprise Manager Cloud Control 12c should have one ormore dedicated utility servers to isolate the impact of I/O intensiveoperations to Utility Servers.VM ServerUnless a server pool is static, VM Servers should only have the VM Serverrole enabled to be able to dedicate CPU, RAM and I/O resources to runningvirtual machines, eliminating the eect of Utility Server operations.

Storage Back-end storageEach Oracle VM server pool uses one dedicated OCFS2 12G mount point(OCFS2 or NFS) for the server pool's cluster congurations and one or moreshared OCFS2 or NFS repositories to host virtual machine congurationles and images.Front-end storageThe virtual machine layer is where the storage is presented to virtualmachines as either a at le (UUID.img), as RAW disks (LUN), or as acombination of at les and RAW disks.

An Oracle VM storage solution consists of three distinct layers. Each layerhas its own unique requirements, congurations, dependencies andfeatures.The rst layer is the storage array, which is referred to as back-endstorage. Oracle VM supports Fibre Channel and iSCSI SAN and NFSback-end storage.The second layer is the server layer consisting of the Oracle VM Server'sDevice-Mapper Multipath congurations and the shared Oracle Cluster FileSystem 2 (OCFS2) or NFS virtual machine le system.Note: OCFS does not factor disk space exhaustion including space forvirtual machine les as well as volume metadata. OCFS2 metadata canconsume over 6% of an OCFS2 volumes free disk space. The third layer is the guest front-end storage consisting of multiple gueststorage (le and RAW) and driver options. RAW disks have the bestperformance of the two front-end storage storage options. In most cases,RAW disks are the best option for high I/O workloads like Oracle Databases.

Networks Each Oracle VM server pool will have isolated Oracle VM managementnetworks and isolated virtual machine networks.Oracle VM uses a total of ve discrete networks; Server Management,Cluster Heartbeat, live Migration, Storage and Virtual Machines. The exact number of network interfaces for an Oracle VM Server entirelydepends on your organizations business requirements and network andstorage infrastructure capabilities. For example, an Oracle VM Server withfour 10G NICs, congured with two 802.1Q bonds could support the mostdemanding network and storage requirements, with only four 10G NICs. Bycontrast, an Oracle VM Server using access ports/port-based VLANsor 802.1Q tag-based VLANS on a 1G copper network, could easily use themaximum number of supported NIC ports (= 3.2 =40 ports) to meet the minimum network requirements.

Each Oracle VM server pool should have a discrete network for the ServerManagement, Cluster Heartbeat, live Migration, Storage and VirtualMachines. Isolating the Server Management, Cluster Heartbeat, liveMigration and Storage networks protect the server pool from unexpectedserver reboots by eliminating OCSF2 heartbeat interruptions that causepool members to fence from the pool and reboot.Each Oracle VM Server will be assigned a unique IP address on the ServerManagement, Cluster Heartbeat, live Migration and Storage network.Note: The heartbeat trac is TCP on port 7777. Each Oracle VM server ina pool must be able to communicate to all of the pool members over TCP onport 7777.

RAM The server pool must be designed with excess RAM capacity toaccommodate the memory requirements of virtual machines that couldmigrate or start on any pool member.

Oracle VM server does not support memory oversubscription, which meansthat an Oracle VM server cannot accept a DRS, Live Migration or HArequests unless the server has available RAM for the virtual machines.Having excess RAM on each Oracle VM server provides exibility in termsof adding new virtual machines to the server pool, and to allow DRS, LiveMigration and HA to operate within a server pool.

NUMA Contemporary CPUs from Intel and AMD have NUMA architectures. NUMAstands for Non-Uniform Memory Access. With NUMA each physical CPU(pCPU) will be assigned its own local memory. An assigned processor-memory pair is called a NUMA node. Local memory access from CPUs onthe same socket will have signicantly lower latency than remote memoryaccess from CPUs on a dierent socket. Oracle VM supports NUMA using a Xen feature called NUMA awarescheduling. NUMA aware scheduling will assign a virtual machine's vCPUs(virtual CPUs) to a NUMA node as a NUMA client. If a virtual machine hasmultiple vCPUs, the NUMA scheduler will always assign the virtualmachine's vCPUs to a single NUMA node to maintain memory locality. Forexample, an Oracle Database virtual machine with 32 vCPUs allocated to asingle NUMA node with 20 threads would be oversubscribed. CPU-bound

If your supporting virtual machines with more vCPUs than its NUMA node, disable NUMA. Forexample, Xen NUMA aware scheduling will place a virtual machine with 32 vCPUs on a singleNUMA node, even if the node does not have 32 cores or threads.



workloads, such as Oracle Databases, should not be on Oracle VM Serverswith oversubscribed CPUs.

Oracle VM Security StandardsThe security controls used to secure Oracle VM are similar to the security controls used to protect your existing physical and virtual IT resources. As with physical and virtual IT resources,securing Oracle VM is dependent on the security posture of each of its components, from the design, hardware, hypervisor, network, and storage to the virtual machine operating systems andinstalled applications. In short, if the organization has a security policy for virtualization, networking, storage, operating systems and applications, the security policies could be applied to OracleVM.Security controls should be employed using industry standard frameworks and standards in the context of the organization's Enterprise Architecture (EA). Organizations turn to their EnterpriseArchitecture to understand how Oracle VM ts within their information system. An Enterprise Architecture is articulated in diagrams and written policies that dene organizational standards andbest practices to plan, build, run, and monitor technologies, including Oracle VM.Enterprise Architecture has well dened principles and processes and an approach that generates a comprehensive, layered policy infrastructure used to communicate managements goals,instructions, procedures, and response to laws and regulatory mandates. A policy infrastructure consists of written tier 1, tier 2, and tier 3 policies that encompass people, systems, data, andinformation. Policies are broken down into high level policies and lower level standards, procedures, baselines, and guidelines.Oracle VM policies typically fall within the layered policy infrastructure of the platform architecture domain. Platform architecture policies are the foundation used to manage the entire lifecycleof an Oracle VM environment.This table outlines the decision points for Oracle VM Manager security controls. For decisions that rely on preexisting factors or specic organizational needs, the appropriate best practice will bediscovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on organizational needs, existingarchitecture, and budget resource availability.

Oracle VM Manager Security ControlsDecision Point Decision JusticationOracle VM Manager and DMZs The Oracle VM Manager application was not designed to be an Internet

facing application. If Internet access is a requirement for Oracle VMManager, VPN access should be used to access the Oracle VM Manager GUI.

The Oracle VM Manager application was not designed to be an Internetfacing application.

Network Time Protocol (NTP) With Oracle VM, accurate time is essential to maintain system stability dueto time-sensitive cluster transactions between Oracle VM Servers. Withoutaccurate time, Oracle VM clusters can be brought to a complete standstill.By default, Oracle VM Servers (up to Release 3.1.1) that are discovered byOracle VM Manager are congured to use the Oracle VM Manager host asthe upstream NTP time host. A best practice is to set-up the Oracle VMManager hosts as the upstream NTP time host to synchronize with upstreamCoordinated Universal Time (UTC) sources as well as provide time servicesto Oracle VM Servers.Oracle Linux and Red Hat Enterprise Linux ship with a default /etc/ntp.confle that points to three of Red Hat's upstream public UTC time sources. Abest practice is to have two internal NTP servers on your local network toprovide time services for internal systems and devices. Using internal timeservers normalizes system event time-stamps across the Enterprise as wellas reduces NTP Internet bandwidth usage.

With Oracle VM, accurate time is essential to maintain system stability dueto time-sensitive cluster transactions between Oracle VM Servers. Withoutaccurate time, Oracle VM clusters can be brought to a complete standstill.A best practice is to set-up the Oracle VM Manager hosts as the upstreamNTP time host to synchronize with upstream Coordinated Universal Time(UTC) sources as well as provide time services to Oracle VM Servers.

Virtual Machine Console Access Oracle VM uses the RAS proxy (Remote Access Service) java applet to proxyvirtual machine console trac from Oracle VM Manager to theadministrator's Client PC. An Oracle VM Manager administrative account isa requirement to access a virtual machine's console. Any rewall betweenOracle VM Manager and the administrator's Client PC conecting to a virtualmachine console must have TCP port 15901 open for the RAS proxy.Oracle VM Manager does not support role based access control. Alladministrative users with access to the Oracle VM Manager GUI have rootadministrative access to all of the objects managed by Oracle VM Manager,including all of the virtual machine consoles. If an Oracle VM Manageraccount is not an option for a user, for example for DBAs, Opertaions. orapplication administators, Oracle VM role based access control can becongured using Enterprise Manager Cloud Control. With Cloud Control,each object managed by Oracle VM Manager can be congured with rolebased access control, including each virtual machine console.

All Oracle VM administrative users have root access to all of the objectsmanaged by Oracle VM Manager. Virtual machine end users such as DBAsand application administrators should only have access to thier virtualmachines, not root access to all of the objects managed by Oracle VMManager. End user access to virtual machines can be controled usingEnterprise Manager Cloud Control. With Cloud Control, each objectmanaged by Oracle VM Manager can be congured with role based accesscontrol, including each virtual machine console.

Host rewallThe iptables service should be enabled on each Oracle VM Manager hostusing a ruleset managed in /etc/syscong/iptables. In order to use OracleVM Manager, the Core API and the Oracle Management Agent with iptables,it is necessary to open tcp ports 7001, 7002, tcp-54321 or tcps-54322, 15901and 3872 as well as UDP 123.

Host rewalls, for example iptables, are a fundamental part of informationsecurity that protect hosts from attacks and intrusions.

Host rewall failed connectionlogging

Iptables failed connection logging should be enabled on each Oracle VMManager host.The following two lines will be added prior to the last REJECT line in the/etc/syscong/iptables le:-A RH-Firewall-1-INPUT -m limit --limit 15/minute -j LOG--log-prex "FW Drop:"-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-hostprohibited

Failed connect logging is a fundamental part of information security thatallows detection of attacks and intrusions.

Root ssh access and sudo

Systems administrators should access the Oracle VM Manager hostwith non-root individual user accounts and use sudo to perform selectedadministrative tasks. Sudo stands for either "substitute user do" or "superuser do". Root ssh access should be disabled on the Oracle VM Manager host. Sudoshould should be used to congure ne-grained permissions to allowadministrative users to perform selected administrative tasks with logging.Disables Root Access:To disable root ssh access, edit the default /etc/ssh/sshd_cong le anduncomment thethe #PermitRootLogin yes line and change the yes to no; that is,PermitRootLogin no. Next, restart the sshd service by typing servicesshd restart to enable the change.The visudo command is used to edit the /etc/sudoers le. Consult thesudoers man page for sudo conguration details.

By default, Oracle Linux permit ssh access using the root super useraccount.One of the most important security measure that can be taken with OracleVM is to prevent unauthorized access to the root user account by disablingroot ssh access. A best practice is to only provision non-root individual useraccounts that can be audited, disabled, expired and managed using sudo.Note: All sudo user access will be tracked and logged in the /var/log/securele.

SSH login banners

Pre and post SSH login banners should be congured on each Oracle VMManager host.Pre-login banner:Edit the /etc/ssh/sshd_cong and add the following directive:

To be able to successfully prosecute individuals who improperly use acomputer, the computer must have a warning banner displayed at all accesspoints.SSH login banners presents a denitive warning or disclaimer to all users



Banner /etc/banner.netNext, create the /etc/banner.net le and add your login banner text, i.e.This system is restricted to authorized access only. All activities on thissystem are recorded and logged. Unauthorized access will be fullyinvestigated and reported to the appropriate law enforcement agencies.Once the le has been created and the banner text is added and saved,restart the sshd by typing:# service sshd restartPost login banner:Edit /etc/motd and add your login banner text, i.e.This system is restricted to authorized access only. All activities on thissystem are recorded and logged. Unauthorized access will be fullyinvestigated and reported to the appropriate law enforcement agencies.Once the le has been edited and saved, restart the sshd by typing:# service sshd restart

that wish to access your systems using SSH. SSH login banners shouldclarify which types of activities are illegal as well as advise legitimate usersof their obligations relating to the acceptable use of the system.

Central log host A central log host should be used to log all user logins and iptablesconnection failures.Centralized logging for user logins and iptables connection failuressimplies security management for the detection of attacks and intrusions.

This table outlines the decision points for Oracle VM Server security controls. For decisions that rely on preexisting factors or specic organizational needs, the appropriate best practice will bediscovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on organizational needs, existingarchitecture, and budget resource availability.

Oracle VM Server Security ControlsDecision Point Decision JusticationOracle VM Server and DMZs Oracle VM Servers hosting Internet facing virtual machines can be placed in

a DMZ without connectivity to the Internet or internal network segments toreduce the attack surface. TCP/8899 is necessary to and from the Oracle VMServers to Oracle VM Manager to enable centralized management usingOracle VM Manager.

Oracle VM Servers in a DMZ should be restricted from inbound andoutbound Internet connectivity to reduce the attack surface.

Build Process Before any Oracle VM Servers are placed on the production network, astandard build processes should be executed to ensure that all Oracle VMServers are installed, congured and maintained in a manner that preventsunauthorized access, unauthorized use and disruptions in service.

An Oracle VM Server build document provides employees with an approvedprocedure to install and congure Oracle VM Server. An Oracle VM Serverbuild document is used with other IT infrastructure policies to addressinteroperability and security of Oracle VM in the context of the entireinformation system.

Patch Management A key component of a successful Oracle VM deployment is acquiring andvetting new releases, patches and updates for production systems. NewOracle VM releases, patches and updates must be researched to identifywhich release, patches and updates are applicable to your environment.Newly released versions, patches and updates should be vetted before beingdeployed into production. A best practice is to run the latest stable releaseof Oracle VM.Oracle VM Servers should be congured to use local custom yumrepositories. Local yum repositories with point-in-time static channel foreach supported Oracle VM release ensures all like Oracle VM server arepatched in a consistent manner across the organization.All patches should be regression tested in the lab environment before theyare deployed on production systems. High-priority patches, security xes,and upgrades will be applied as needed in accordance with s Change Management Policy.All production systems should undergo security audits in accordance withs Change Management Policy to validate congurationand patch compliance.

A patch management program is an integral component of an organization'sinformation security program used to mitigate the risk from securityvulnerabilities (bugs) that are inherent in all operating systems andapplications. A key component of patch management is acquiring and vetting patches forproduction systems. Patches must be researched to identify which patches,security xes, and updates are applicable to your environment. Newlyreleased patches, security updates, and application updates will be testedbefore being deployed in to production using time stamped local customrepositories.Pre- and post-production audits will be conducted in accordance withs Change Management Policy to validate congurationand patch compliance.

Host rewall The iptables service will be enabled on each Oracle VM server using thedefault policy and ruleset in /etc/syscong/iptables.

Host rewalls, for example iptables, are a fundamental part of informationsecurity that protect hosts from attacks and intrusions.

Host rewall failed connectionlogging

Iptables failed connection logging will be enabled on the Oracle VMManager host and each Oracle VM server.The following two lines will be added prior to the last REJECT line in the/etc/syscong/iptables le:-A RH-Firewall-1-INPUT -m limit --limit 15/minute -j LOG--log-prex "FW Drop:"-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-hostprohibited

Failed connect logging is a fundamental part of information security thatallows detection of attacks and intrusions.

Root ssh access and sudo

Systems administrators should access Oracle VM Servers with non-rootindividual user accounts and use sudo to perform selected administrativetasks. Sudo stands for either "substitute user do" or "super user do". Root ssh access should be disabled on the each Oracle VM servers. Sudoshould should be used to congure ne-grained permissions to allowadministrative users to perform selected administrative tasks with logging.Disables Root Access:To disable root ssh access, edit the default /etc/ssh/sshd_cong le anduncomment thethe #PermitRootLogin yes line and change the yes to no; that is,PermitRootLogin no. Next, restart the sshd service by typing servicesshd restart to enable the change.Note: To enable sudo on Oracle VM Servers, it is neccessary to installthe ovs-support-tools meta-package that includes sudo.The visudo command is used to edit the /etc/sudoers le. Consult thesudoers man page for conguration details.

By default, Oracle VM Server permit ssh access using the root super useraccount.One of the most important security measure that can be taken with OracleVM is to prevent unauthorized access to the root user account by disablingroot ssh access. A best practice is to only provision non-root individual useraccounts, that can be audited, disabled, expired and managed using sudo.Note: All sudo user access will be tracked and logged in the /var/log/securele.

SSH login banners

Pre and post SSH login banners should be congured on each Oracle VMManager host and Oracle VM Server.Pre-login banner:Edit the /etc/ssh/sshd_cong and add the following directive:Banner /etc/banner.netNext, create the /etc/banner.net le and add your login banner text, i.e.

To be able to successfully prosecute individuals who improperly use acomputer, the computer must have a warning banner displayed at all accesspoints.SSH login banners presents a denitive warning or disclaimer to all usersthat wish to access your systems using SSH. SSH login banners shouldclarify which types of activities are illegal as well as advise legitimate usersof their obligations relating to the acceptable use of the system.



This system is restricted to authorized access only. All activities on thissystem are recorded and logged. Unauthorized access will be fullyinvestigated and reported to the appropriate law enforcement agencies.Once the le has been created and the banner text is added and saved,restart the sshd by typing:# service sshd restartPost login banner:Edit /etc/motd and add your login banner text, i.e.This system is restricted to authorized access only. All activities on thissystem are recorded and logged. Unauthorized access will be fullyinvestigated and reported to the appropriate law enforcement agencies.Once the le has been edited and saved, restart the sshd by typing:# service sshd restartNote: By default Oracle VM Server's /etc/motd le displays the followingwarning message: Warning: making manual modications in themanagement domainmight cause inconsistencies between Oracle VM Manager and the server.

Rootkit prevention and monitoring Wikipedia describes a rootkit as A rootkit is software that enablescontinued privileged access to a computer while actively hiding its presencefrom administrators by subverting standard operating system functionalityor other applications.A Hypervisor (Oracle VM Server) may be one of the most sensitive operatingsystems in the data center because it controls the hardware as well as all ofthe virtual machines on it. If the hypervisor is compromised direct access tothe hardware and all of the virtual machines is possible, and other codecould be monitored and controlled by the attacker.

Monitoring the hypervisor (Oracle VM Server) for rootkits is fundamentalpart of information security used to detect rootkits to prevent attacks andintrusions. Each Oracle VM Server should have a rootkit prevention solutionin place, such as chkrootkit, that monitors the host for rootkits.

Central log host A central log host will be used to log all user logins and iptables connectionfailures.Centralized logging for user logins and iptables connection failuressimplies security management for the detection of attacks and intrusions.

Virtual Machine Operating System StandardsThis table outlines the decision points for the for virtual machine operating systems hosted on Oracle VM. For decisions that rely on preexisting factors or specic organizational needs, theappropriate best practice will be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based onorganizational needs, existing architecture, and budget resource availability.Decision Point Decision Justication

Virtual Machine Operating SystemsA small number of virtual machine operating systems should be be used. Forexample, stadardizing on Oracle Linux 6 latest, in contract to supportingOracle Linux 5U2, 5U3, 5U4, 5U5, 5U6, 5U6, 5U8, 5U9, 5U10, 6, 6U1, 6U2,6U3, 6U4, etc...

Standardizing on a small number of virtual machine operating systemsstreamlines operations and increases the level of le duplication onproduction and archival virtual machine data. This design reducescomplexity and increases operational eciency by limiting the number ofsupported operating systems.

Virtual Machine Operating SystemVersioning

In accordance with s Application Software Policy andApplication Software Standards, applications will determine the operatingsystem type and version.

Each application has an operating system support matrix that lists thesupported operating systems, patch levels, and software prerequisites.In accordance with s Application Software Policy andApplication Software Standards, applications will determine the operatingsystem type and version.

Virtual Machine Operating SystemDeployments

All new virtual machine operating systems will be deployed using a virtualmachine template in accordance with s Server Policy,and Server Security Policy.

A virtual machine template is a self-contained, precongured virtualmachine with an operating system and optionally an application installed inaccordance with s Server Policy, Server Security Policy,and Operating System Installation Guidelines.Each time a new virtual machine is deployed using a virtual machinetemplate, s standards are applied to each new virtualmachine.

Patch Management

Linux virtual machines will be congured to use local custom yumrepositories.All patches will be regression tested in the lab environment before they aredeployed on production systems.High-priority patches, security xes, and application upgrades updates willbe applied as needed in accordance with s ChangeManagement Policy.Noncritical xes will be applied on a Quarterly basis in accordance withs Change Management Policy.All production systems will undergo security audits in accordance withs Change Management Policy to validate congurationand patch compliance.

A key component of patch management is acquiring and vetting patches forproduction systems. Patches must be researched to identify which patches,security xes, and application updates are applicable to your environment.Newly released patches, security updates, and application updates will betested before being deployed in to production using time stamped localcustom repositories.Local yum repositories will be maintained for patch testing and productionusing a point-in-time static channel for each supported operating system toensure all like operating systems are patched in a consistent manner acrossthe organization.Pre- and post-production audits will be conducted in accordance withs Change Management Policy to validate congurationand patch compliance.

Oracle VM Disaster RecoveryAn Oracle VM disaster recovery architecture includes the design and process to maintain business continuity following a disastrous event aecting the availability of an organization's primarysite. Failover to a disaster recovery site is prompted by the results of a disaster assessment. The failover process is the restoration of the primary site's services at the disaster recovery site.Note: Disaster recovery requirements are calculated using Service-level Agreements (SLA), Recovery Time Objectives (RTOs) and Recovery Point Objectives (RPOs) objectives. SLA, RPO and RTOobjectives and budget inuence the disaster recovery architecture and design. Oracle VM uses the concept of a server pool to group together and manage one or more clustered Oracle VM servers. Once an Oracle VM server pool is created, the physical and virtual resourcesare managed within the boundary of the server pool. Physical resources include server hardware, networks, storage, infrastructure services (DNS, NTP, LDAP, HTTP, etc..), operating systeminstallation media and administrative accounts. The virtual resources include virtual disks, virtual network interfaces, and virtual machine conguration les. For example, an Oracle VMenvironment with multiple server pools located in one or more sites could be managed from a single Oracle VM Manager instance with each server pool's resources isolated to their respectedserver pool. An Oracle VM server pool's resources from one site can be replicated and restored to another site for disaster recovery.Restoration of the primary site's services at a disaster recovery site requires a replica of the primary site's physical and virtual resources at the disaster recovery site. A disaster recovery sitehosts a replica of the primary site's Oracle VM physical and virtual resources, i.e. server hardware, networks, storage, infrastructure services, virtual disks, and virtual machine conguration les.The failover process involves restoring the primary sites virtual machines at the disaster recovery site, then systematically starting the virtual machines and services.Note: Oracle VM Servers are not backed up and restored at the DR site. The time required to backup and restore an Oracle VM Server is signicantly greater then a PXE boot kickstartinstallation.

A disaster recovery site can be a warm failover site waiting idle to respond to a disastrous occurrence, or part of a multi-site high availability design. A multi-site design uses excess capacity withapplication high availability to mirror services across sites to handle the lose of one or more sites.



The next Figure shows a warm Oracle VM failover site waiting idle to respond to a disastrous occurrence.

The next Figure shows a warm Oracle VM failover site responding to a disastrous occurrence and running the primary sites services.

The next Figure shows a multi-site Oracle VM design with application high availability solutions to mirror services across sites as well as excess capacity to handle the lose of one or more sites.

Virtual machines that are restored at a disaster recovery site expect the same networks, storage, and infrastructure services as in the primary site. In the event that the disaster recovery site hasdierent networks, storage, and infrastructure services, the properties of each virtual machines would need to be edited to use the new networks, storage and infrastructure services beforeservices can be restored.The virtual machine operating systems are typically installed in virtual disks that are actually at les hosted on shared OCFS2 or NFS repositories. RAW disks such as ASM Disks, Log andArchive Files, etc.. are presented to the virtual machines from the Oracle VM Servers as local devices. Each virtual machine's virtual network interface card(s) (vNIC) are connected to one ormore discrete networks using Xen bridges that are managed and presented to the virtual machines by the Oracle VM pool members. Virtual disks and virtual network interface card(s) allocationsare managed using Oracle VM Manager and/or Oracle Enterprise Manager with the congurations saved in each virtual machines vm.cfg le.The virtual machine vm.cfg les, virtual disk images and RAW disks (ASM disks) can be replicated between sites using storage array replication and/or mirroring solutions. Rsync is an option if anarray does not have replication and/or mirroring functionality.As soon as the replicated storage repositories are available, the failover process for a warm recovery site starts with the installation of Oracle VM Manager with the runInstall.sh --uuid optionusing the primary sites Oracle VM Manager UUID. An Oracle VM Manager --uuid installation allows Oracle VM Manager to use the primary site' replicated repositories with the virtual machines.Tip: The Oracle VM Manager UUID is listed in the .cong le on the Oracle VM Manager host in the /u01/app/oracle/ovm-manager-3/ directory as well as in each server pool' .ovsrepo le in the



Decision Point Decision JusticationDisaster Recovery Design Prior to implementing an Oracle VM Disaster Recovery solution, its

important that an infrastructure assessment (IA) and gap analysis (GA) beperformed. During the IA/GA, the architecture of the solution will bematched to the customers SLA, Recovery Time Objectives (RTOs) andRecovery Point Objectives (RPOs) objectives.

Implementing a Disaster Recovery is a strategic decision. Disaster recoveryrequirements are calculated using SLA, Recovery Time Objectives (RTOs)and Recovery Point Objectives (RPOs) objectives. SLA, RPO and RTOobjectives and budget inuence the disaster recovery architecture anddesign.

Oracle VM Manager Oracle VM Manager will be installed in Production mode using therunInstall.sh --uuid option with the primary site's Oracle VM Manager UUID.Oracle VM Manager will be hosted on a dedicated physical server using anexternal or local Oracle 11g Standard, Enterprise or RAC database.Once Oracle Enterprise Manager is restored, the Oracle EnterpriseManager Agent and Virtualization plug-in will be installed to enable OracleEnterprise Manager integration.

As soon as the replicated storage repositories are available, the failoverprocess for a warm recovery site starts with the installation of Oracle VMManager with the runInstall.sh --uuid option using the primary sites OracleVM Manager UUID. An Oracle VM Manager --uuid installation allows OracleVM Manager to use the primary site' replicated repositories and virtualmachines.The Oracle VM Manager UUID is listed in the .cong le on the OracleVM Manager host in the /u01/app/oracle/ovm-manager-3/ directory as wellas in each server pool' .ovsrepo le in the pool le system.

Oracle VM Server Builds Oracle VM Servers will be installed using an automated build process. Oracle VM servers are installed using an automated PXE boot congurationto ensure that each server has a consistent installation conguration.

Oracle VM Server Backups Oracle VM Servers will not backed up at the primary site and restored atthe DR site.

The time required to backup and restore an Oracle VM Server issignicantly greater then an automated PXE boot kickstart installation.Oracle VM servers are installed using an automated PXE boot congurationto ensure that each server has a consistent installation conguration.

Storage A replica of the primary site's repositories with the virtual machineresources and RAW disks will be hosted at the disaster recovery site.

As soon as the replicated storage repositories and RAW disks are available,the failover process for a warm recovery site starts with the installation ofOracle VM Manager with the runInstall.sh --uuid option using the primarysites Oracle VM Manager UUID. An Oracle VM Manager --uuid installationallows Oracle VM Manager to use the primary site' replicated repositoriesand virtual machines.Virtual machines that are restored at a disaster recovery site expect thesame storage as in the primary site. In the event that the disaster recoverysite has dierent storage each virtual machine would need to be recreatedor edited to use the new storage before services can be restored.

Networks A replica of the primary site's Oracle VM networks will be maintained at thedisaster recovery site.

Virtual machines that are restored at a disaster recovery site expect thesame networks as in the primary site. In the event that the disaster recoverysite has dierent networks each virtual machine would need to be edited touse the new networks before services can be restored.

Infrastructure Services A replica of the primary site's infrastructure services will be maintained atthe disaster recovery site.

Virtual machines that are restored at a disaster recovery site expect thesame infrastructure services as in the primary site. In the event that thedisaster recovery site has dierent infrastructure services, each virtualmachine operating system would need to be edited to use the newinfrastructure services before services can be restored.

pool le system.The next example shows the content of the .cong le with the UUID in bold.# cat /u01/app/oracle/ovm-manager-3/.cong DBHOST=localhostSID=orclLSNR=1521APEX=NoneOVSSCHEMA=ovsWLSADMIN=weblogicOVSADMIN=adminCOREPORT=54321UUID=0004fb00000100009edfaa0f93184f44BUILDID=3.0.3.126The next example shows the content of the .ovsrepo le with the UUID in bold.# cat .ovsrepoOVS_REPO_UUID=0004fb0000030000554308a6997a6b2fOVS_REPO_MGR_UUID=0004fb00000100009edfaa0f93184f44OVS_REPO_VERSION=3.0This table outlines the decision points for an Oracle VM disaster recovery solution. For decisions that rely on preexisting factors or specic organizational needs, the appropriate best practice willbe discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on organizational needs, existingarchitecture, and budget resource availability.

Oracle VM Application IntegrationThis table outlines the decision points for an Oracle VM hosted application integration. For decisions that rely on preexisting factors or specic organizational needs, the appropriate best practicewill be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based on organizational needs,existing architecture, and budget resource availability.Decision Point Decision Justication

Application Support Applications must be supported by the independent software vendor (ISV)on the latest version of Oracle VM to be included in the Oracle VMenvironment.

Only applications that are supported by Oracle on Oracle VM should behosted on Oracle VM. Only applications with ISV support for the latestversion of Oracle VM can be deployed and supported by ,the ISV, and Oracle on Oracle VM.

Application Requirements andDependencies

Applications will be analyzed for requirements and dependencies and testedin accordance with 's Software Installation Standards. Applications will be analyzed for requirements and dependencies and tested

to ensure compliance with ISV specications.

Application Installation, Packaging,and Distribution

Applications should be installed and packaged using Oracle VM TemplateBuilder or Oracle Enterprise Manager.Applications that are installed and packaged using Oracle VM TemplateBuilder will be deployed as Oracle VM templates.Applications that are installed using Oracle Enterprise Manager will beinstalled on Oracle VM templates.

Application installations that are packaged in Oracle VM templates ordeployed using Oracle Enterprise Manager have a consistent installationconguration.

Application sunsetting Applications will be sunsetted in accordance with 'sHardware and Software Sunset Policy

Applications that have reached the end of their life cycle and are no longersupported by a vendor will be given a sunset date. The sunset date is whenthe product is scheduled to be removed from production.Sunsetting applications that have reached the end of their life cycle resultsin better customer service and reduced costs.



Patch Management

All patches will be regression tested in the lab environment before they aredeployed on production systems in accordance with sChange Management Policy.Noncritical xes will be applied on a Quarterly basis in accordance withs Change Management Policy.All production systems will undergo security audits in accordance withs Change Management Policy to validate congurationand patch compliance.

A key component of patch management is acquiring and vetting patches forproduction systems. Patches must be researched to identify which patches,security xes, and application updates are applicable to your environment.Newly released patches, security updates, and application updates will betested before being deployed in to production using time stamped localcustom repositories.Pre- and post-production audits will be conducted in accordance withs Change Management Policy to validate congurationand patch compliance.

Change LogRevision Change Description Updated By Date2.0 Document Creation Roddy Rodstein 09/30/132.1 Content Refresh Roddy Rodstein 01/14/142.2 NUMA Design Details Roddy Rodstein 02/13/15



Documents

Oracle Cloud Referance Design