1

EMC IT‘S ORACLE BACKUP AND

RECOVERY—4X CHEAPER, 8X FASTER, AND 10X BETTER—

SYMMETRIX VMAX, DATA DOMAIN, AND NETWORKER DATA DOMAIN TRANSFORMS EMC IT‘S ORACLE

BACKUP AND RECOVERY INFRASTRUCTURE

ABSTRACT

Migrating from a legacy availability infrastructure for Backup and Recovery

creates challenges in terms of what are the best practices for a new Backup and

Recovery deployment with EMC‘s Oracle databases for Global Data Warehouse

and mission-critical Oracle applications. This white paper will illustrate the

transformation of EMC IT Oracle Backup and Recovery Infrastructure and

highlight how the Data Domain appliance transforms EMC IT Oracle Backup

infrastructure.

August 2012

WHITE PAPER

2

Copyright © 2012 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change

without notice.

The information in this publication is provided ―as is.‖ EMC Corporation makes no representations or warranties of any kind with

respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a

particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

EMC2, EMC, the EMC logo, and the RSA logo are registered trademarks or trademarks of EMC Corporation in the United States

and other countries. VMware is a registered trademark of VMware, Inc. in the United States and/or other jurisdictions. All other

trademarks used herein are the property of their respective owners. © Copyright 2012 EMC Corporation. All rights reserved.

Published in the USA. 08/12 White Paper H10986.1

3

TABLE OF CONTENTS

EXECUTIVE SUMMARY ........................................................................................................................... 4 Audience ............................................................................................................................................ 4

EMC IT’S BACKUP AND RECOVERY JOURNEY ......................................................................................... 5 EMC IT Overview ................................................................................................................................ 5 EMC IT‘s Backup and Recovery business drivers ...................................................................................... 5 EMC IT‘s Backup and Recovery Legacy Architecture ................................................................................. 5

EMC Backup Profile – Legacy EMC IT VTL infrastructure...................................................................... 5 Legacy Backup and Recovery Components........................................................................................ 5

Legacy Backup and Recovery Pain Points ............................................................................................... 8 EMC IT‘s New Backup and Recovery—Data Domain Infrastructure ............................................................. 8

Components ................................................................................................................................. 8 EMC Data Domain Deployment Models ............................................................................................. 9

GDW/CRM BACKUP AND RECOVERY COMPONENTS .............................................................................. 10 Backup and Recovery enablers ............................................................................................................ 10

EMC IT’S ORACLE BACKUP AND RECOVERY METHODS ......................................................................... 11 EMC IT‘s Offloaded Backup and Recovery Process for Oracle Databases .................................................... 11

Step 1- TimeFinder Clone .............................................................................................................. 11 Step 2- Clone Mounted on Proxy/Backup Host .................................................................................. 11 Step 3- RMAN/NetWorker Script to Backup up a Mounted Clone to Data Domain .................................. 11

EMC IT‘s Regular Backup Process ......................................................................................................... 12

EMC IT’S OFFLOADED BACKUP EXAMPLES ........................................................................................... 13 Global Data Warehouse (GDW) Backup Size .......................................................................................... 13 GDW Pain Points ................................................................................................................................ 13 Oracle CRM Backup and Recovery Problem Statement ............................................................................ 13 Advantages of Proxy Host Solution ....................................................................................................... 13

EMC TimeFinder Clone .................................................................................................................. 13 EMC TimeFinder Snapshot ............................................................................................................. 14

ADVANTAGES OF ORACLE BACKUP AND RECOVERY DATA DOMAIN DEPLOYMENT ............................... 15 Reuse of the legacy EMC IT Backup and Recovery process ...................................................................... 15 EMC Data Domain – Deduplication technology ....................................................................................... 15

Deduplication Benefits ................................................................................................................... 15 Full Backups versus Incremental .................................................................................................... 15

ROI/TCO ........................................................................................................................................... 15 4X Cheaper.................................................................................................................................. 15 10X Better ................................................................................................................................... 15

EMC IT LESSON LEARNED .................................................................................................................... 16 Bottlenecks ....................................................................................................................................... 16

Disks .......................................................................................................................................... 16 Proxy Server: 8X Faster ............................................................................................................... 16 EMC NetWorker Server .................................................................................................................. 16 Utilization versus Vulnerability ....................................................................................................... 16

CONCLUSION ....................................................................................................................................... 17 References ........................................................................................................................................ 18 Acknowledgments .............................................................................................................................. 18

4

EXECUTIVE SUMMARY EMC IT has seen explosive growth over the last five years accelerating the need to move from its legacy, Virtual Tape Library

(VTL) Backup infrastructure to a new EMC Data Domain Backup infrastructure.

This creates challenges in terms of what are the best practices for a new Backup and Recovery deployment with EMC‘s Oracle

databases for the Global Data Warehouse and mission-critical Oracle applications.

EMC IT implemented a phased approach because of the size of the Oracle environments:

Over 670 Oracle databases in all environments:

o Production

o Test/Development

o QA

o Performance

o Patch

Greater than 30 TB of Oracle Redo Logs generated per day

Mission-Critical environments (databases) backed up daily

Moving to the new EMC IT Data Domain infrastructure, for EMC‘s mission-critical Oracle Global Data Warehouse (GDW) and

Oracle CRM production environments has delivered the following advantages:

4X Cheaper - The Data Domain appliances are a quarter the cost of the legacy EDL/VTL

8X Faster – Move from an incumbent EDL/VTL speed of 500 MB/hour to Data Domain speed of 4 TB/hour

10X Better – Based on the following:

o Reliability - More reliable than tapes – guaranteed that backups can be restored.

o Density - Stores more backups for a longer period of time, even old backups can be quickly restored.

o Protection – Production Data Domain appliances are replicated to a remote Data Domain appliance off-site.

o Speed – Both backups and restores are significantly faster. Backups are now completed within the Service Level

Agreement (SLA) with fewer support resources.

o Complexity – Standard full backups are much easier to execute and restore than incremental backups.

o Cost – Data Domain units are one-fourth the cost with up 10 times the capacity, after deduplication.

This white paper will illustrate the journey and benefits of EMC IT‘s Oracle Backup and Recovery Infrastructure and highlight how

the Data Domain appliance transformed EMC IT Oracle Backup architecture. It will also include lessons learned from EMC IT on

this migration from a legacy VTL infrastructure to a Data Domain infrastructure.

Audience

This white paper is intended for CIOs, Oracle architects, Backup and Recovery architects, storage architects, Oracle Database

Administrators (DBAs), and server and network administrators.

5

EMC IT’S BACKUP AND RECOVERY JOURNEY This section covers the drivers and the size of the journey, with respect to EMC‘s Backup and Recovery profile. It details the

legacy Backup and Recovery infrastructure, as well as the new EMC IT Backup and Recovery Data Domain infrastructure

deployment models.

EMC IT Overview

EMC is a company with over 53,000 users of IT services. It supports over 400,000 customers and partners in 5 Data Centers

with over 16 PB of storage. EMC IT has a portfolio with over 500 business applications and tools and over 8000 OS images with

more than 90 percent of all servers virtualized in 80 countries and 20 languages.

EMC IT’s Backup and Recovery business drivers

The following are the business drivers for EMC IT‘s Backup and Recovery:

Inability to meet EMC‘s backup or restore SLAs with the legacy EMC VTL infrastructure

o Erratic backup windows, some spanning over 24 hours

o Daily ―hand-holding‖/‖fire fighting‖ of the backup process by EMC IT Backup and Oracle SMEs

o Limitations on the number of backup images stored on the legacy EMC VTL infrastructure

EMC IT’s Backup and Recovery Legacy Architecture

This section describes EMC‘s Backup and Recovery profile as well as the legacy Backup and Recovery components.

EMC Backup Profile – Legacy EMC IT VTL infrastructure

The following is the current EMC IT Oracle Backup and Recovery profile:

Over 670 Oracle databases

Over 100 Terabytes of Oracle Databases backed up per day

Mission-Critical production environments backed up daily FULL

Mission-Critical Dev and Test environments backed up every two days FULL

Non Mission Critical and dev/test environments backed up FULL 2X per week

Archive log backups are run daily and when triggered by space alerts

Legacy Backup and Recovery Components

The following are the baseline Backup and Recovery components used in the legacy (VTL) infrastructure:

EMC NetWorker

NetWorker delivers complete Oracle protection through the NetWorker Module for Oracle.

NetWorker integrates with Oracle Recovery Manager (RMAN)—Oracle‘s recommended Backup and Recovery interface for its

databases. Through this integration, NetWorker simplifies Backup and Recovery for Oracle and is able to integrate Oracle server

protection into several core features, including wizard-driven configuration and push installation.

The EMC NetWorker Module for Oracle has value-added features available to reduce backup impact and required storage. This

includes integrated deduplication and snapshot management functionality.

Additionally, an exclusive feature called event-based backup helps deliver more effective backup operations based on probing for

conditions rather than starting merely by time.

To gain the benefits of consolidation and flexible availability, the world is moving increasingly to operation in virtual server

environments. NetWorker, and its Oracle support, integrate with VMware, delivering in guest, application-consistent protection

and support for environments that leverage VMware technologies such as Distributed Resource Scheduler, vMotion, and High

Availability that move and migrate virtual machines in a VMware environment.

Another benefit of NetWorker is that it is tested, certified and supported in a broad range of operating systems and

environments.

6

Oracle Recovery Manager (RMAN)

Oracle RMAN is a built-in tool that allows the database administrator (DBA) to easily backup and recovery data in an Oracle

database. RMAN handles the coordination required to ensure that transaction integrity is preserved, and sufficient information is

maintained to recover the database to any appropriate point. RMAN can create backup sets that comprise as much or as little

recovery information as the DBA requires but usually include information from the database datafiles, control files, redo files and

archived log files.

EMC NetWorker/RMAN Integration

NetWorker delivers full integration and is fully compliant with Oracle RMAN. Through this integration, NetWorker delivers

complete Backup and Recovery for Oracle databases, including the ability to capture and protect databases, individual datafiles,

tablespaces, and logs—all of the things that comprise an Oracle installation.

NetWorker, through the RMAN integration, is able to provide backups for Oracle whether the database is online or offline. Other

highlights include the ability to deliver block-level incremental backups, which helps to ensure backups are fast and capturing

the unique changes to the data. Other RMAN features include block corruption detection and surgical recovery of any detected

corrupted blocks to help ensure data is accurate and recoverable.

The solution also compresses any unused blocks to save time, space, and bandwidth. In addition, RMAN delivers restartable

backups from the point of failure for interrupted backups—very helpful, especially for very large databases. Through Oracle‘s

Data Recovery Advisor, any issues encountered and diagnosed are automatically repaired.

All of these capabilities help ensure recovery, which is the primary goal. By providing automatic log management, NetWorker is

able to bring the database back to a point-in-time, thereby restoring all data and replaying any needed logs to facilitate a

complete restore. Finally, not only can you restore back to where the data came from, but also in cases where there is other or

new hardware in place, NetWorker will restore the data to an alternate location.

EMC Disk Library (EDL)/Virtual Tape Library (VTL)

A virtual tape library is a disk based storage array front-ended by servers that emulate tape drive systems. Disk based VTLs

have significant advantages over their tape counterparts as they dramatically improve throughput speeds. This is even more

significant when it comes time for a restore. Since the backups already exist on disk, restores do not suffer from the mechanical

limitations of either the tape drive streaming speeds or the robotic arms or even the fast forward and rewind actions associated

with finding the backup set.

The following is a high-level illustration of the legacy Backup and Recovery infrastructure:

Figure 1. Legacy Backup and Recovery infrastructure with VTL

NetWorker

Virtual Tape Library Database Server

7

Proxy Host

A Proxy Host is a server used to mount a cloned copy of a database for backup purposes. It is primarily used to offload the

backup overhead from the production database. It can be used by many databases and has the advantage that the VTL can be

directly mounted to a single Proxy Host, giving significantly more throughput with SAN protocols, vs. backups over NFS. This is

much more important when performing partial restores, as the database can be rapidly restored to the Proxy Host and a partial,

or surgical restore can be performed very rapidly. Of course, if the data required is from the most recent backup, the database

does not even need to be restored from the VTL, as it still exists on the Proxy Host.

Figure 2. Legacy Backup and Recovery infrastructure with Proxy Host

The configuration of the proxy server does not require significant CPU resources. The focus of the server is strictly for IO

throughput. For this reason, EMC IT selects a server with a large number of expansion slots. As an example, for EMC IT‘s CRM

environment the proxy server configuration is shown in Table 1.

Table 1. CRM Proxy Host configuration

TimeFinder Clone

EMC TimeFinder is an EMC replication technology that is used by EMC Symmetrix VMAX storage arrays to instantly create an

exact copy of a set of LUNS. For a database, these LUNS would represent the database files for building a standby or backup

copy and optionally include the redo files, for creating a reporting copy. This capability allows the backup or clone of a database

without putting the database in a ―HOT‖ backup mode.

Quantity Hardware

1 Cisco UCS C460 M2 - 128GB RAM - 2 x 73GB HDD

8GB DDR3-1333MHz

2 10 GB Network Card - Bonded

4 lpe1200E dual-port 8GB HBA – 4 ports zoned for VMAX, 4 ports zoned to VTL device

SOFTWARE

Oracle RDBMS – single instance

RedHat 5.6

ASM

Database Server Virtual Tape Library

NetWorker

Proxy Host

8

Legacy Backup and Recovery Pain Points

The legacy Backup and Recovery infrastructure, described above, created some ―pain points‖ for EMC IT.

The following are the two that most impacted EMC IT:

Limited retention for old backups

Like all organizations, EMC IT has a need for retaining backups for extended periods. This generally consists of writing backups

off to tape and sending them to an offsite storage facility. Given the fragility of tape backup media, EMC IT could never be 100

percent sure that it was possible to restore these tapes and the cost of limited use media was expensive to maintain. By placing

these EDLs at a data center nearby, EMC IT was solving the offsite requirement, but at a cost of backup throughput.

Storage needed (large overhead)

With the introduction of EDL devices, EMC‘s version of a VTL, EMC IT quickly adopted them and began deploying the backups to

these, disk-backed, virtual tape libraries. It did not take long to outgrow these EDLs, as the amount of disk required to retain

EMC IT‘s long-term backups was massive. This required EMC IT to continue to rely on tapes for long-term backup retention.

Fortunately, the introduction of deduplication technology and devices like Data Domain‘s VTL, Virtual Tape Library helped

address this challenge.

EMC IT’s New Backup and Recovery—Data Domain Infrastructure

This section describes the legacy Backup and Recovery tools that are re-used in the new Backup and Recovery Data Domain

infrastructure. This includes: the Data Domain appliance Backup and Recovery; the deployment model needed for the EMC IT‘s

shared GDW/CRM Backup and Recovery process; and components for their Oracle Database Proxy Host Backup and Recovery

process.

Components

The new Backup and Recovery infrastructure consists of EMC IT‘s current Backup and Recovery tools:

EMC NetWorker - 7.6.3

Oracle RMAN

EMC NetWorker Module for Oracle (NMO) – 5.0

Proxy Hosts

EMC TimeFinder Clones

The major difference, between the new infrastructure and the legacy infrastructure, is the destination of the backups. In the new

infrastructure, EMC IT writes to the Data Domain appliances. The Data Domain appliances, though the deduplication feature,

allows EMC IT to store many more backups in a fraction of the space previously required. This capability has eliminated the

need for the EDL/VTL and more importantly the need for any fragile tapes.

For the offsite requirements, EMC IT uses Data Domain replication to replicate the backups to a secondary datacenter 600 miles

away, completely eliminating the need for any tapes.

Data Domain

EMC Data Domain deduplication storage systems are designed and optimized, specifically for backup and archive of data.

Support for any conventional backup or archive application through generalized support for network-attached storage (NAS)

interfaces over Ethernet; a virtual tape library (VTL) interface option over Fibre Channel; and product-specific interfaces such

as NetBackup OpenStorage and EMC Data Domain Boost

High-speed, inline deduplication using small, variable-sized sequences to identify and eliminate redundant data segments

before storing to disk

Integrated data protection technologies such as RAID 6, post-backup data verification, and periodic validation checks of

existing data sets

Automated replication of backup data for disaster recovery (DR) using cost-effective, low-bandwidth WAN links, which

enables faster ―time-to-DR‖ readiness

9

EMC Data Domain Deployment Models

EMC Data Domain systems can be deployed as a NAS device or as a VTL:

NAS deployment

When the Data Domain system is deployed as NAS, RMAN can write backups directly to it as a disk backup. This can be

extremely useful for environments that have only a few databases to backup. As the number of databases grows, the

management all of these disparate mounts can be overwhelming. For larger organizations, integration with an enterprise

backup solution, such as EMC NetWorker, is recommended. This integration provides many benefits, such as a centralized

location for managing and cataloging backups. EMC NetWorker, along with many of the other solutions, also includes

technology that can significantly improve backup throughputs.

Figure 3. Data Domain deployed as NFS mount to Host

VTL deployment

When the Data Domain system is deployed as a VTL, RMAN must be integrated with an enterprise backup application like EMC

NetWorker. This is because, like all VTLs, Data Domain systems operate with VTL or Open Storage protocols and RMAN is not

capable of interfacing with them directly.

With a VTL deployment, Data Domain systems can be connected though a 1 GB or 10 GB network typically dedicated to backup

traffic. They can also be connected directly to database server HBA ports achieving much higher rates of throughput.

Figure 4. Data Domain deployed as a VTL device

Networker

Data Domain Database Server

Data Domain Database Server

10

GDW/CRM BACKUP AND RECOVERY COMPONENTS The following illustrate the new EMC IT Backup and Recovery infrastructure components, when offloading backups to a

Proxy Host:

Figure 5. Oracle Database Proxy Host Backup and Recovery Process

As the diagram illustrates, EMC IT deploys a shared Backup and Recovery infrastructure using the following components and

best practices:

VMAX TimeFinder Clones

Deployment of a Proxy Host or EMC NetWorker Storage node

o Oracle database – single instance (backup license)

o EMC NetWorker Module for Oracle

o RedHat AS 5.6

Data Domain appliance deployed as a VTL device – DD890

Backup and Recovery enablers

The following technologies enable EMC IT‘s Oracle Backup and Recovery today:

8 Gigabit Storage Area Network

10 Gigabit dedicated Local Area Network

EMC Data Domain 890 Appliances

EMC NetWorker – Version 7.6.3

EMC NetWorker Module for Oracle (NMO) – Version 5.0

Oracle RMAN – 11.2

EMC VMAX TimeFinder Clone

Proxy Clone Hosts

O

N

T

A

C

T

U

S

To

le

ar

n

m

or

e

ab

ou

t

ho

w

E

M

C

pr

od

uc

ts,

se

rvi

ce

s,

an

d

so

lut

io

ns

ca

n

he

lp

so

lv

e

yo

ur

bu

Proxy Backup

Server

88 GGbb\\ss SSAANN

FFaabbrriicc BB

88 GGbb\\ss SSAANN

FFaabbrriicc AA

Production Data Domain

11

EMC IT’S ORACLE BACKUP AND RECOVERY METHODS

EMC IT’s Offloaded Backup and Recovery Process for Oracle Databases

The following are the high-level steps for the offloaded Backup and Recovery Proxy Host backup process:

1. Create a TimeFinder Clone

2. Include the redo logs so that the database can be verified prior to the backup. This is not necessary, but adds an extra

level of confidence that the database backup is good. It also allows the database to be opened, read-only for

investigation, reporting or extract purposes.

3. Mount the Clone on the Proxy Host server

4. Invoke the NetWorker/RMAN scrip to backup the TimeFinder Clone to Data Domain infrastructure

Step 1- TimeFinder Clone

EMC IT uses EMC TimeFinder/Clone Technology as the first step in the backup process. The Clone is a full size point-in-time

replica of the source database. Once the Clone is originally created, subsequent ―recreate‖ operations only apply the

incremental changes made to the source since the last ―recreate‖ of the Clone. This reduces the first step of the backup

process. EMC IT‘s daily clone typically takes 40 minutes for a 20TB database.

After 40 minutes, the database has a viable Backup and Recovery solution. This backup, is now available for surgical or

complete restores.

Step 2- Clone Mounted on Proxy/Backup Host

EMC IT uses a Proxy Host, or backup host, to offload the RMAN backup to a virtual tape that is the Data Domain appliance. With

a Proxy Host, a separate host is used, configured with Oracle Binaries and ASM installed. The ASM diskgroups on the Proxy Host

are made up of the source database‘s cloned devices.

Deploying a backup host has many benefits:

Allows full backup to run on the Proxy Host producing no performance impact to the production database and production

user community

Enables surgical restores from the Proxy Host from the most recent backup or from a restored backup from any point in

time. Having a point-in-time restore on a separate host allows individual tables or tablespaces to be restored using

export/import or transportable tablespace or even database links for very specific requirements.

Enables ad-hoc querying for end-users. Queries can run from the clone for ad-hoc reporting – allowing full autonomy for

reports running from the clone. This can stop less efficient or resource intensive reports from interfering with the source

database (Production).

Reverse Clone – For databases that require a quick and efficient RTO (Recovery Time Objective) the Reverse Clone allows

super quick restores back to the source.

Offloaded Stats Generation – before the next backup run, the database can be opened and stats generated on the cloned

version of the database. This allows for complete level of generation, with no impact to the production system.

Step 3- RMAN/NetWorker Script to Backup up a Mounted Clone to Data Domain

Once the Clone operation is complete, the Clone is opened as a read only database, thus verifying a valid copy, and backed up

to Data Domain using RMAN/NetWorker. EMC IT re-used the same legacy scripts that were already in place when EMC IT

moved from the EDL to the Data Domain appliance deployed as a VTL.

The only additional consideration was to change the ―filesperset‖ setting, which was changed to 1. This contributes to the most

favorable compression ratio and efficiently stores subsequent full backs to the Data Domain with a very small storage footprint.

Using the Data Domain appliance to store backups has enabled many advantages including:

Virtually no change to existing backup scripts, RMAN catalogs or pre-determined scheduling – the only consideration or

potential change being a change to the filesperset parameter.

Integrates easily with EMC NetWorker infrastructure.

12

The deduplication benefits of Data Domain allow many more images of full backups to be stored with minimal storage

overhead. This is especially significant for databases that carry a lot of unchanged history as the bulk of the data, which is

the case in all data warehouses and most OLTP databases.

Data Domain enables EMC IT to eliminate the need for a traditional Oracle incremental backup strategy. This simplifies the

restore steps and shortens the recovery time window.

RMAN Script Example

Here is an RMAN script excerpt from EMC IT‘s backup script used to backup EMC IT‘s offloaded databases.

The script uses standard Oracle RMAN commands:

connect rcvcat DatabaseA/xxxxxx@rmancatolog

connect target xxxxxxx/xxxxxx@DatabaseA

run {

set command id to 'rman database';

allocate channel t1 type 'SBT_TAPE' format '%d_full_%U';

allocate channel t2 type 'SBT_TAPE' format '%d_full_%U';

allocate channel t3 type 'SBT_TAPE' format '%d_full_%U';

….

(EMC IT uses 24 channels)

…..

send 'NSR_ENV=(NSR_DATA_VOLUME_POOL=DBORAP)';

backup tag = ‗DatabaseAbkp_hot_0209_214224' filesperset 1 database;

backup current controlfile channel t1 format '%d_cf_%U';

release channel t1;

release channel t1;

release channel t1;

……

}

Table 2. EMC IT legacy RMAN script example

EMC IT’s Regular Backup Process

For the majority of EMC‘s databases offloading to a proxy server is not required.

For those databases, where speed is not an absolute necessity and the overhead of an RMAN backup directly on the database

server does cause performance issues, EMC IT will not offload the backup to a proxy server.

In this case, EMC IT executes the backup directly from the database server. Backups are scheduled every other day as full

database backups to the same Data Domain appliances as the offloaded backups. Since the deduplication benefits are the

same, as offloaded backups, there is no need to do incremental backups, which are impacted by the extra complexity, and

restore times associated with incremental backups.

For EMC IT‘s larger databases, using this regular backup process EMC IT installs a dedicated network card to the database

server. This additional network card in the database server reduces the overhead to the public network that a large backup

would add, and assists with overall throughput.

In the future, EMC IT is looking at Data Domain‗s Boost technology to both alleviate that overhead as well as reduce the backup

window by distributing the deduplication process between the backup server and database server.

13

EMC IT’S OFFLOADED BACKUP EXAMPLES

Global Data Warehouse (GDW) Backup Size

EMC‘s Global Data Warehouse (GDW) is a central source of relational data that provides business intelligence to many key

business groups within EMC including Sales, Marketing, Finance, Manufacturing, Customer Service, Professional Services, Human

Resources and Legal departments.

It is a critical component of intellectual property and it‘s supported and maintained accordingly. One key and highly important

support requirement is the ability to backup and provide realistic and flexible recovery options to the business groups that rely

on this data.

GDW runs on a six-node Linux-based Oracle Real Application (RAC) Cluster that went live in November of 2009, as a 10TB

database. Since 2009, GDW has significantly grown and as of May of 2012 GDW has doubled to 20 TB with archive log

generation totaling from 500 to over 1500 GB daily.

GDW Pain Points

In the summer of 2011, the time to complete an online RMAN/NetWorker backup of the GDW to an EMC Disk Library (EDL)

degraded. The throughput was erratic and at times would exceed 24 hours, ―wrapping‖ over the scheduled start time of the

daily TimeFinder Clone used as part of the backup solution.

This undermined EMC IT‘s ability to successfully provide a backup of the database.

There were periods where it took a week to successfully backup the database to EDL, as the scheduled clone would cause the

online backup to fail. This resulted in daily handholding of backups and undermined the ability of IT to protect this valuable

investment.

The backup/recovery solution was in need of remediation and some redesign as: backups ran too long and would sometimes fail

leaving data protection capability limited and vulnerable. Additional staff-hours were needed to ―fire-fight‖ daily challenges.

A strict limitation to the number of full backup images that could be stored on the Disk Library, due to the large storage footprint

each backup imposed. Archived log backups became increasingly important and the number of logs needed to satisfy recovery

points became suspect.

Oracle CRM Backup and Recovery Problem Statement

EMC IT‘s Oracle 11i eBusiness Suite Database grew from 8 TB to 10 TB over a period of two years. During this two-year period,

the backup time increased from 6 hours to over 22 hours, causing significant challenges in getting a full, successful, daily

backup. This also meant that restores, to this environment, would take longer than 22 hours. These times were well beyond

EMC IT‘s SLA for the databases Recovery Time Objective (RTO).

This, over 3X increase in the backup window, was due to more than just growth in database size. It also was due to the

increase in the number of databases that needed to backup to this shared infrastructure, and was guaranteed not to be solved

on its own. As this was EMC IT‘s mission-critical database, with a zero data loss policy, 11i Database had to be backed up daily.

The increase in backup time, violated EMC IT‘s restore SLAs as well.

To resolve this challenge, EMC IT embarked on a new solution with EMC VMAX Clone and Data Domain.

Advantages of Proxy Host Solution

EMC IT uses a Proxy Host to run backups for its Oracle 11i eBusiness Suite database. The copy that is mounted to the Proxy

Host is built using EMC TimeFinder clone technology.

EMC TimeFinder Clone

TimeFinder Clone provides single or multiple point-in-time copies of the full database within the same storage array.

The cloned database is mounted on the Proxy Host for backup. A database clone requires the same disk space as the source

database. Clones are extremely fast, as the data never leaves the storage array.

The initial clone for a 20 TB database takes about 4 hours, with a low quality of service, QOS. EMC IT uses the low QOS to

eliminate the impact to production. After the initial clone, all subsequent clones use a differential clone, only copying the

changed tracks to the cloned devices. For example, EMC‘s GDW 20 TB database clone takes 40 minutes to complete. From the

Proxy Host, backup of this 20 TB database to Data Domain appliance takes 4 hours.

14

EMC TimeFinder Snapshot

Another option is to use EMC TimeFinder snapshots.

A snapshot is a point-in time "virtual" copy that occupies very little disk space. A snap requires only 10 to 20 percent of source

database space for storing only changed tracks.

In this case, backup will actually read most of the data from production disks. This is a very efficient method of offloading the

backups, but does incur a small overhead to the production database, since the database and backup are competing for the

same physical disks. A good way to minimize this is to use virtual pools in the storage array. This provides many more disks to

spread out the I/O load.

The full clone solution offloads the entire backup load from the production database server and doesn‘t impact performance for

EMC IT‘s OLTP online users. This is especially useful when backup runs for a long time.

Proxy Host copy also provides much better Recovery Time Objectives (RTO). In the event you need to restore a production

database, restore from backup media will take hours. Since proxy copy is already on disk, it takes significantly less time to

restore. You can do Reverse Clone to restore a production database and then apply missing archive logs.

This method also provides some protection against accidental data corruption/loss due to human errors. For example, a few

critical tables in the production database were dropped one hour ago due to human error. Application will work fine if you can

restore only these tables. You cannot afford to restore complete production database and suffer 1-hour data loss. Instead, you

can restore proxy database point in time just before table drop, and then copy tables back to production manually. Proxy copy is

extremely useful in such surgical repair scenarios.

15

ADVANTAGES OF ORACLE BACKUP AND RECOVERY DATA DOMAIN

DEPLOYMENT The following are the advantages the EMC IT gained when deploying the new Backup and Recovery Data Domain infrastructure

Reuse of the legacy EMC IT Backup and Recovery process

The first gain was virtually no change to existing EMC IT Backup and Recovery processes such as backup scripts and RMAN

catalogs. There was one consideration potential change to ―filesperset‖ as defined in the EMC white paper, Oracle RMAN Design

Best Practices with EMC Data Domain.

Another advantage was that the Data Domain infrastructure integrated seamless with the legacy EMC IT backup software, EMC

NetWorker.

EMC Data Domain – Deduplication technology

Deduplication technology allowed EMC IT to move its long-term backups to the Data Domain infrastructure with complete

confidence that EMC IT would be able recover those backups, if the need ever arose.

It also solved the issue with off-site storage as these Data Domain devices have the ability to replicate all backups to a Data

Domain unit at a remote site. For EMC IT, the replicated Data Domain unit is located at EMC IT‘s DR site, giving us additional

protection.

Deduplication Benefits

Data Domains deduplication technology has enabled EMC IT is to retain many more backup images with minimal storage

overhead. EMC IT can now restore backup images from several months past—without having to go through the difficult process

of locating off-site tapes and hoping that they are still valid.

Full Backups versus Incremental

Due to Data Domain‘s ability to deduplicate the backups, EMC IT was able to use daily full backups versus having to do

incremental backups. This allows a much quicker recovery process since incremental backups require restoring the full backup

then restoring the incrementals on top of that. This not only reduces the complexity of the restore process, but it also reduces

EMC IT‘s recovery window.

ROI/TCO

4X Cheaper

Typically, it is difficult to calculate actual dollars saved when utilizing shared infrastructure and the costs to administer an

environment that is not meeting SLA‘s. In EMC IT‘s case, the actual cost of the Data Domain units EMC IT replacing its EDL

with are one fourth the cost and between double and quadruple the capacity—depending on the actual de-dupe rates.

In addition, to these savings, Data Domain technology is also increasing EMC IT‘s ability to retain backups longer, and

completely eliminate the need for tapes. When factoring in all of these additional cost reductions, it is clear that overall costs

greatly exceed the 4x cheaper claim.

10X Better

The following is a mix of quantitative/qualitative measurements of ―Better‖ that include people, process, and technology:

1) Faster—8X faster with Proxy server deployment with 8GB FC deployment

2) Reliability—More reliable than tapes – guaranteed backup can be restored

3) Cost—Costs less for hardware and environments (power and cooling)

4) Density—Stores more backups creating a greater retention period

5) Protection—Replicated remote Data Domain appliance can be deployed off-site

16

EMC IT LESSON LEARNED As with all functions in IT, following is a set of best practices that will yield better performance, scalability and stability.

Bottlenecks

Disks

When configuring for high speed backups, there are many potential locations for bottlenecks. The most obvious of these is the

disk layout, in the storage array. When it comes to I/O, the capability of the disks is most often the bottleneck.

To give some idea, a 15K fibre channel drive is capable of driving between 47 and 53 GB/hour. This means that to support an

8TB/hour backup, you would need 155 of these disk drives.

For the Data Domain Backup and Recovery environments, EMC IT does not need 8 TB/hour. This means EMC IT configured its

disk pool with only 76 disks and is achieving 4 TB/hour.

Proxy Server: 8X Faster

Next up, in the stack, is the connectivity between the storage array and the proxy server. This was initially overlooked when

EMC IT migrated from the EDL to the Data Domain. The Backup and Recovery infrastructure had 8 GB SAN switches, HBAs and

ports on the SAN; however, the Backup and Recovery infrastructure initially connected it to the 2 GB SAN switch.

Even after re-cabling to the 8 GB SAN, EMC IT still found the ports on the storage array had been configured for 4 GB. After

fixing these issues, the new Backup and Recovery infrastructure effectively doubled the initial throughput of 2 TB/hour to 4

TB/hour. When compared to EMC IT‘s legacy EDL backup speeds, EMC IT improved its backup speeds by a factor of 8 from

about 500 MB/hour on the legacy EMC DL4206 EDLs to about 4 TB/hour on the new Data Domain VTL

EMC NetWorker Server

After relieving that bottleneck, the new Backup and Recovery infrastructure began to have a problem with the NetWorker server.

The NetWorker server that was deployed was already overloaded with backup requests and was not able to keep up the large

number of backup sets that were now being requested to be processed. Keep in mind that EMC IT set files per set to 1, for best

results in deduplication. Since this node was part of the original shared infrastructure and was clearly overloaded, configuring

and deploying another NetWorker server relieved the bottleneck and improved overall backup performance throughout the data

center.

Utilization versus Vulnerability

One of the constant challenges EMC IT faces with their backup infrastructure is how to maximize the utilization without creating

vulnerability.

In order to maximize the investment in the new backup infrastructure EMC IT moved from a dedicated model to shared model.

This has allowed EMC IT‘s Backup and Recovery team to become much more efficient. However, there was no mechanism to

measure the effect of consolidation on individual backup and restore times. This is what had lead EMC IT into the problem of

backups not completing. It is extremely important to monitor backup times and investigate individual backups that start taking

longer than was originally configured. An obvious example of this would be to use the proxy backup server for more than just

one environment—but without coordinated backup windows, both backups could suddenly fall out of the configured times.

17

CONCLUSION EMC IT implemented a Backup and Recovery solution to deal with the growing size of its Oracle environments:

Over 670 Oracle databases in all environments (Test/Development/QA/Performance/Patch/Production)

Over 30 TB of Oracle Redo Logs per day

Mission-critical environments (databases) backed up daily

Migrating from a legacy EMC EDL/VTL solution to Data Domain brought the following advantages to EMC IT‘s Backup and

Recovery deployment:

Moving away from limited Backup and Recovery backup retention to enabling of simple full backup to reduce the complexity

of incremental restore process

Accommodates months of full Backup and Recovery backups enabled by the Data Domain appliance

Deployment of a Proxy host architect in the Backup and Recovery architect delivers the following:

o Off load backup process off of production enabling EMC business users experience no disruption or performance issues

on the CRM production server

o Ability to reuse legacy Backup and Recovery processes/scripts

Implementation of a shared infrastructure for EMC IT‘s mission critical CRM and GDW environments

EMC TimeFinder technology, which was able to create database clones in minutes (18 TB in 40 minutes) off the production

server and mount on the proxy server and backup in less than four hours

EMC IT also learned valuable lessons from the EMC EDL/VTL to Data Domain Backup and Recovery journey:

Disks—To meet the throughput in a Data Domain the source database Array needs the correct number of drives. To drive to

8 TB/hour the array would need 155 drives

Proxy Server—Confirm the infrastructure cabled correctly and enough HBAs

NetWorker Server—Identify a near capacity NetWorker server and plan for enough nodes to meet your total Backup and

Recovery community needs

Utilization versus Vulnerability—Monitor the backup and restore times in a shared deployment to increase the Backup

and Recovery infrastructure utilization while diminishing vulnerability of long running backups , failed backed or long running

restores or failed restores

Moving to the new EMC IT Data Domain infrastructure for EMC‘s mission-critical Oracle Global Data Warehouse (GDW) and

Oracle CRM production environments has enabled the following:

4X Cheaper—The Data Domain appliances are a quarter the cost

8X Faster—Move from a legacy EDL speed of 500 MB/hour to an EMC Data Domain speed of 4 TB/hour

10X Better—Based on the following:

o Reliability—More reliable than tapes; guaranteed backup restoration

o Density—Stores more backups for a longer period of time; even restores for old backups can restored quickly

o Protection—Production Data Domain appliances are replicated to a remote Data Domain appliance offsite

o Speed—Both backups and restores are significantly faster; backups are completed within the SLA with fewer support

resources

o Complexity—Standard full backups are much easier to execute and restore than incrementals

o Cost—EMC Data Domain appliances are one-fourth of the cost with up 10 times the capacity, after deduplication

18

References

EMC Symmetrix VMAX

http://www.emc.com/storage/symmetrix-vmax/symmetrix-vmax.htm

EMC TimeFinder

http://www.emc.com/storage/symmetrix-vmax/timefinder.htm

EMC Data Domain

http://www.emc.com/backup-and-recovery/data-domain/data-domain.htm

EMC NetWorker

http://www.emc.com/backup-and-recovery/networker/networker.htm

IDC Study – Worldwide Purpose Built Backup Appliances

http://www.emc.com/collateral/analyst-reports/11530-idc-ww-pbba-2011-2015-forecast.pdf

EMC IT

http://www.emc.com/microsites/emc-it-proven/index.htm

Acknowledgments

The author would like to thank the EMC IT teams for assistance in the creation of this white paper.

EMC Corporation Hopkinton, Massachusetts 01748-9103 1-508-435-1000 In North America 1-866-464-7381 www.EMC.com

EMC2, EMC, the EMC logo are registered trademarks or trademarks of EMC Corporation in the

United States and other countries. VMware [add additional per above, if required] are registered

trademarks or trademarks of VMware, Inc., in the United States and other jurisdictions.

© Copyright 2012 EMC Corporation. All rights reserved. Published in the USA. 08/12

EMC White Paper H10986

EMC believes the information in this document is accurate as of its publication date.

The information is subject to change without notice.

CONTACT US

To learn more about how

EMC products, services, and

solutions can help solve your

business and IT challenges,

contact your local

representative or authorized

reseller—or visit us at

www.EMC.com.