H11402: EMC Transforms IT for SAP: Fully Automated … · Basics of RecoverPoint replication ..... 13 Replication modes ... EMC Symmetrix VMAX 10K, EMC VNX5300, EMC RecoverPoint,

White Paper

EMC Solutions Group

Abstract

This white paper demonstrates how EMC® RecoverPoint can be integrated seamlessly with VMware® vCenter™ Site Recovery Manager™ to perform seamless recovery and storage migration between heterogeneous storage systems such as EMC VNX® and EMC Symmetrix® VMAX® in SAP environments. This solution increases Return On Investment (ROI) by using existing investments in implementing a disaster recovery solution.

January 2013

EMC TRANSFORMS IT FOR SAP: FULLY AUTOMATED DISASTER RECOVERY EMC Symmetrix VMAX 10K, EMC VNX5300, EMC RecoverPoint, VMware vCenter Site Recovery Manager

Minimize SAP business processes downtime with a quick and automated recovery

Simplify and accelerate nondisruptive disaster recovery testing Increase ROI by leveraging existing landscape investments in DR

EMC Transforms IT for SAP: Fully Automated Disaster Recovery EMC Symmetrix VMAX 10K, EMC VNX5300, EMC RecoverPoint, VMware vCenter Site Recovery

Manager

2

Copyright © 2013 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

All trademarks used herein are the property of their respective owners.

Part Number H11402

3 EMC Transforms IT for SAP: Fully Automated Disaster Recovery EMC Symmetrix VMAX 10K, EMC VNX5300, EMC RecoverPoint, VMware vCenter Site Recovery

Manager

Table of contents

Executive summary ............................................................................................................................... 5

Business case .................................................................................................................................. 5

Solution overview ............................................................................................................................ 6

Key results ....................................................................................................................................... 7

Introduction.......................................................................................................................................... 8

Purpose ........................................................................................................................................... 8

Scope .............................................................................................................................................. 8

Audience ......................................................................................................................................... 8

Terminology ..................................................................................................................................... 8

Technology introduction ..................................................................................................................... 10

Overview ........................................................................................................................................ 10

EMC RecoverPoint .......................................................................................................................... 10

EMC Symmetrix VMAX 10K ............................................................................................................. 10

Virtual Provisioning ................................................................................................................... 10

EMC VNX5300 ................................................................................................................................ 11

EMC Unisphere .............................................................................................................................. 11

VMware vCenter Site Recovery Manager ......................................................................................... 11

SAP ERP 6.0 ................................................................................................................................... 12

Basics of RecoverPoint replication ..................................................................................................... 13

Replication modes ......................................................................................................................... 13

Main components .......................................................................................................................... 13

EMC RecoverPoint appliance ..................................................................................................... 14

Write splitters ............................................................................................................................ 14

Replication sets and consistency groups ................................................................................... 14

Storage Replication Adapter ...................................................................................................... 14

Journal volumes ........................................................................................................................ 14

Initial replication ............................................................................................................................ 15

Solution architecture .......................................................................................................................... 16

Overview ........................................................................................................................................ 16

Physical design .............................................................................................................................. 16

Hardware resources ....................................................................................................................... 17

Software resources ........................................................................................................................ 18

Virtual machine resources .............................................................................................................. 19

SAP application environment ......................................................................................................... 20


Manager

4

Solution configuration ........................................................................................................................ 21

Overview ........................................................................................................................................ 21

Configuring replication ................................................................................................................... 21

Enabling bi-directional replication ............................................................................................. 21

Customization specific to SAP ........................................................................................................ 22

Heterogeneous test environment ............................................................................................... 22

Replication mode ...................................................................................................................... 23

Startup priorities ....................................................................................................................... 24

Network settings ....................................................................................................................... 24

/etc/hosts file ........................................................................................................................... 24

SAP shared folders .................................................................................................................... 25

SAP startup sequence ............................................................................................................... 25

SAP user logon .......................................................................................................................... 26

Load-generation environment .................................................................................................... 27

Bi-directional recovery testing ................................................................................................... 27

System integrity and quality control .......................................................................................... 27

Test and validation ............................................................................................................................. 29

Overview ........................................................................................................................................ 29

Test scenario 1: Nondisruptive failover testing ............................................................................... 29

Test scenario 2: Disaster recovery .................................................................................................. 32

Test scenario 3: Failback ................................................................................................................ 34

Test scenario 4: Point-in-time recovery ........................................................................................... 37

Conclusion ......................................................................................................................................... 42

Summary ....................................................................................................................................... 42

Findings ......................................................................................................................................... 42

References.......................................................................................................................................... 44

White papers ................................................................................................................................. 44

Product documentation .................................................................................................................. 44

SAP notes ...................................................................................................................................... 44

Appendix A: RecoverPoint replication modes ..................................................................................... 45

Continuous data protection ............................................................................................................ 45

Continuous remote replication ....................................................................................................... 46

Concurrent local and remote replication ......................................................................................... 47

Appendix B: Setting up replication ..................................................................................................... 49

Preparing VMAX 10K and VNX5300 for RecoverPoint connectivity .................................................. 49

Configuring RecoverPoint ............................................................................................................... 52

Configuring SRM ............................................................................................................................ 54


Manager

Executive summary

For most businesses, the growth in eCommerce and focus on consumers’ experience has led to more opportunity and competition. Simply stated, businesses have expanded beyond traditional geographic boundaries and markets, interacting with consumers directly. Capitalizing on this opportunity requires:

Responsiveness: Consumers have choices. If they are not satisfied with the service level they receive, they go elsewhere to find what they need, leading to severe productivity loss.

Market and customer intelligence: Availability of data and information is critical to any business for understanding customer behaviors and preferences to maintain competitive advantages.

Supply, demand, and order fulfillment management: What may seem to be a simple transaction to an eCommerce consumer in reality necessitates the seamless availability of a “federated” landscape of dependent business applications.

Ultimately, these business requirements translate into the challenges for SAP IT including:

Scalability: Providing increased performance to handle increasing transactional processing volumes while maintaining access to increasing amounts of data.

For example, customers who have started out with smaller legacy systems for SAP environments can opt to scale outward to more powerful storage solutions like EMC® Symmetrix® VMAX® .

Disaster Recovery: How to respond to increasing pressures to mitigate planned and unplanned downtimes as the growth in production operations gives rise to the need for new levels of protection against disasters or logical corruption.

Additionally, from the standpoint of business continuity, Information Security Management Systems (ISMS) standards require regular testing of backup and recovery solutions to ensure effectiveness in the event of a disaster. Unfortunately, regular failover testing is rather difficult for businesses with very low downtime tolerance.

Traditional disaster recovery (DR) solutions, such as an offsite tape restore, may no longer be adequate to address increasing recovery time objectives (RTO) and recovery point objectives (RPO). Storage replication solutions can satisfy these requirements. However, maintaining an idle data center for disaster recovery may not be suitable for some organizations either. These challenges demand a solution that offers effective, affordable, and efficient disaster recovery protection for these critical SAP business functions, without wasting any previous investments.

Business case


Manager

6

This solution provides customers with a strategy that enables DR, using VMAX and VNX® series interchangeably as production and DR storage for SAP applications.

The solution team designed the architecture for central SAP installations, distributed implementations (database, central services, and application are on separate virtual machines), and heterogeneous EMC storage solutions such as VNX and VMAX families.

We implemented this using a Symmetrix VMAX 10K storage array together with an existing VNX5300 storage array to form a bi-directional disaster recovery solution. We achieved this by replicating both storage devices to each other with EMC RecoverPoint and designing automated recovery plans using VMware® vCenter™ Site Recovery Manager™ (SRM).

This solution demonstrates the following capabilities:

Fast and seamless failover

This solution allows quick and fully automated failover on multiple systems—in the correct order, in a matter of minutes, and with a few clicks of a button. This allows enterprises to be resilient against disasters and to regain productivity sooner.

Automated failback

This solution facilitates automated reversal of the data centers’ roles after failover. After the original production site is operational, failback is just a matter of performing another failover.

Point-in-time (PIT) recovery without database restore

The solution can perform PIT recovery without the need to restore and roll forward the entire databases just before a logical corruption occurs.

Nondisruptive failover testing

This solution allows online testing by restoring a snapshot and encapsulating it in a private network during the test. This procedure effectively reduces the need for disruptive testing. Information security audit requirements are now easier to manage.

Solution overview


Manager

This solution significantly improves business agility and scales out for many common needs for SAP production and nonproduction systems:

The bi-directional replication strategy enabled by EMC RecoverPoint allows two sites to mutually replicate and serve as production sites and recovery sites of each other. This strategy reuses the existing storage systems—VNX in this case, to effectively increase ROI.

The integration of EMC RecoverPoint and VMware vCenter SRM provides a seamless end-to-end recovery strategy, which requires minimal downtime for failover and failback, and point-in-time recovery operations.

This solution provides an environment for performing regular recovery testing without any downtime, to be compliant with the security audit requirements.

Customers can leverage this solution to migrate their SAP environments between heterogeneous system environments.

Key results


Manager

8

Introduction

This white paper demonstrates how EMC RecoverPoint can be integrated seamlessly with VMware vCenter SRM to perform seamless recovery and storage migration between EMC VNX and EMC Symmetrix VMAX systems.

It describes the benefits in the following scenarios:

After migrating business data from VNX to Symmetrix VMAX 10K, relocating the existing VNX5300 from the production site to the recovery site

Configuring two active production data centers to mutually protect each other

To achieve the stated purpose, the document covers the following:

Components used and the key concepts involved in implementation

The reference architecture of the bi-directional disaster recovery system using EMC Symmetrix VMAX 10K and VNX5300

Installation and configuration procedures for EMC RecoverPoint and VMware Site Recovery Manager

Use-case functionality test results

This white paper is intended for SAP administrators, storage administrators, IT architects, and disaster recovery managers who:

Have virtual SAP environments on existing storage and plans to migrate to a VMAX storage family, while keeping good use of the original storage after migration.

Have existing virtual SAP environments on Symmetrix VMAX storage solutions and aim to find more economic strategies in implementing disaster recovery solutions.

Table 1 includes the terminology used in this white paper.

Table 1. Terminology

Term Definition

Asynchronous replication Process of committing data writes when the data has been sent out for replication without waiting for acknowledgement from the receiving side.

Asynchronous replication provides crash-consistent protection and recovery to specific points-in-time, with a small RPO.

CDP Continuous data protection.

CLR Concurrent local and remote replication.

Purpose

Scope

Audience

Terminology


Manager

Term Definition

Consistency group RecoverPoint configuration that logically groups replication sets that must be consistent with one another. The consistency groups ensure that updates to the production volumes are written to the replicas in a consistent write order and that the replicas can always be used to continue working from or to restore the production source.

CRR Continuous remote replication.

Deduplication A specialized data compression technique for eliminating duplicate copies of repeating data.

Gatekeepers Small, three-cylinder devices (approximately 3 MB) used by the solutions enabler to send control commands to the Symmetrix VMAX array.

Journal volume Timestamped application writes stored in RecoverPoint journals for later recovery to selected points-in-time.

KPI Key performance indicator.

PIT Point-in-time; usually refers to a system state that occurred at a specific date and time in the past.

Replica volume Volume that contains the replicated image from a production site.

Replication set RecoverPoint configuration that defines an association between a production volume and the local and/or remote volumes to which it is replicating.

Repository volume Volume that stores configuration information about RecoverPoint appliances (RPAs) and consistency groups. This volume enables a surviving RPA to take over the replication activities of a failed RPA from the same cluster.

RPA RecoverPoint appliance.

RPO Recovery point objective.

RTO Recovery time objective.

Synchronous replication Process of committing data writes only after the receiving side acknowledges the data. This ensures that writes between two sites are always consistent.


Manager

10

Technology introduction

This solution uses the key components described in Table 2.

Table 2. Key components

Component Role

EMC RecoverPoint Storage replication device

EMC Symmetrix VMAX 10K Production site storage

EMC VNX5300 Recovery site storage

EMC Unisphere® Storage management application

VMware vCenter Site Recovery Manager DR failover coordination

SAP ERP 6.0 Enterprise application software

Note While Symmetrix VMAX and VNX models were used in the solution testing,

this solution is generally applicable to other models as well.

EMC RecoverPoint is an advanced enterprise-class disaster recovery solution designed with the performance, reliability, and flexibility required for enterprise applications in heterogeneous storage and server environments. It provides bi-directional local and remote data replication, without distance limits.

The Symmetrix VMAX 10K system is built on the highly scalable EMC Virtual Matrix Architecture, which enables the storage to grow seamlessly and cost-effectively from an entry-level, single-bay, single-engine configuration to a six-bay, four-engine system with 512 GB cache memory, 1,080 drives, and up to 1.5 PB usable capacity.

Built for simplicity and ease-of-use, Symmetrix VMAX 10K is 100 percent virtually provisioned, with storage tiering automatically managed by EMC Fully Automated Storage Tiering for Virtual Pools (FAST VP). Symmetrix VMAX 10K supports Flash, Serial Attached SCSI (SAS), and Near-Line SAS (NL-SAS) drives with RAID 1, 5, and 6 protection options.

Symmetrix VMAX 10K systems are preconfigured and ready for same-day installation and startup. They support all EMC Symmetrix monitoring and management tools, including the latest enhanced Symmetrix Management Console, which provides simpler installation and management with smart wizards.

The integrated RecoverPoint splitter for Symmetrix VMAX 10K enables local and remote replication for flexible and efficient RPOs and RTOs.

Virtual Provisioning

EMC Virtual Provisioning™ simplifies storage configuration and management, improves capacity utilization, and enhances performance by enabling the creation of thin devices. Thin devices present more capacity to applications than what is physically allocated in the storage array.

Overview

EMC RecoverPoint

EMC Symmetrix VMAX 10K


Manager

Physical storage that supplies disk space to the thin devices comes from a thin pool. The thin pool is composed of data devices that provide the actual physical storage. You can add or remove data devices to grow or shrink thin pools nondisruptively. Virtual Provisioning supports local and remote replication with RecoverPoint on VMAX 10K.

Virtual Provisioning is described in detail in EMC Solutions Enabler Symmetrix Array Controls CLI Version 7.3 Product Guide.

EMC VNX5300 storage array is a member of the VNX series storage platforms, which delivers innovation and enterprise capabilities for file, block, and object storage in a scalable, affordable, and easy-to-use solution. Designed to meet the high-performance, high-scalability requirements of midsize and large enterprises, the VNX series enables enterprises to grow, share, and cost-effectively manage multiprotocol environments.

The VNX series is powered by quad-core processors, making it two to three times faster than its predecessor. The VNX processors support the demands of advanced storage capabilities such as Virtual Provisioning, compression, and deduplication. VNX arrays incorporate the RecoverPoint splitter, which supports unified file and block replication for local data protection and disaster recovery.

EMC Unisphere is the central management platform for EMC Symmetrix VMAX and VNX series, providing a single combined view of file and block systems, with all features and functions available through a common interface. Unisphere is optimized for virtual applications and provides industry-leading VMware integration. It automatically discovers virtual machines and VMware ESX® servers and provides end-to-end, virtual-to-physical mapping.

VMware vCenter SRM is a disaster recovery framework that integrates with EMC RecoverPoint to automate recovery of VMware datastores so that it becomes as simple as pressing a button.

SRM is an extension to VMware vCenter that enables integration with array-based replication, discovery, and management of replicated datastores, and automated migration of inventory from one vCenter to another. SRM does not replicate any data, but, instead, leverages an external replication solution such as RecoverPoint. SRM servers coordinate the operations of the replicated storage arrays and vCenter servers on the production and recovery sites so that, as virtual machines on the production site are shut down, virtual machines on the recovery site start up and resume the responsibility of providing the same services, using the data replicated from the production site.

Migration of protected inventory and services from one site to the other is controlled by a recovery plan that specifies the order in which virtual machines are shut down and started up, the computing resources that are allocated, and the networks they can access. Integrated with RecoverPoint, SRM also enables testing of recovery plans, using a temporary copy of the replicated data, in a way that does not disrupt ongoing operation on either site.

EMC VNX5300

EMC Unisphere

VMware vCenter Site Recovery Manager


Manager

12

SAP ERP 6.0, powered by the SAP NetWeaver technology platform, is a world-class, fully integrated enterprise resource planning (ERP) application that fulfills the core business needs of midsize and large enterprises across all industries and market sectors. SAP ERP 6.0 delivers a comprehensive set of integrated, cross-functional business processes and can serve as a solid business-process platform that supports continued growth, innovation, and operational excellence.

SAP ERP 6.0


Manager

Basics of RecoverPoint replication

RecoverPoint offers different modes of replication, as shown in Figure 1.

Figure 1. RecoverPoint data replication options

Types of data replication in RecoverPoint include:

Continuous data protection (CDP) for local replication

Continuous remote replication (CRR) for remote protection

Concurrent local and remote replication (CLR) for combined local and remote protection

Appendix A: RecoverPoint replication modes provides more details.

The following components enable this technology:

RecoverPoint appliance (RPA)

Write splitters

Replication sets and consistency groups

RecoverPoint Storage Replication Adapter (SRA)

Journals

Replication modes

Main components

Continuous Data Protection (CDP)

Continuous Remote Replication (CRR)

Concurrent Local and Remote

Replication (CLR)

PRD

PRD

PRD

Local replica

Local replica

Remote replica

Remote replica

*PRD – Production volume


Manager

14

EMC RecoverPoint appliance

RecoverPoint is available as a physical appliance. This enables support of large amounts of information stored across heterogeneous environments. This out-of-band approach enables RecoverPoint to deliver continuous replication. RPAs run the RecoverPoint software and manage all aspects of data replication. For local replication, a cluster configuration of two or more active RPAs is deployed. This supports immediate switchover to another appliance if one of the RPA nodes in a cluster goes down. For remote replication, a RecoverPoint cluster is deployed at both sites.

RPAs use powerful deduplication, compression, and bandwidth reduction technologies to minimize the use of bandwidth and dramatically reduce the time lag between writing data to storage at the source and target sites.

Write splitters

RecoverPoint monitors writes using a technology called write splitters, which ensure that all host writes to a protected volumes are tracked and a copy is sent to local RecoverPoint appliances. RecoverPoint supports three types of write splitters in general: array write splitters, intelligent fabric-based write splitters, and host-based write splitters.

In this solution, RecoverPoint uses an array-based write splitter that operates in each front-end adapter of the Symmetrix VMAX 10K array, as well as inside the storage processors of an EMC VNX storage array.

Replication sets and consistency groups

Regardless of the mode chosen, RecoverPoint allows you to define an association between a production volume and the local and/or remote volumes to which it is replicating, called a replication set.

These replication sets, which must be consistent with one another, are logically grouped into consistency groups. The consistency groups ensure that updates to the production volumes are written to the replicas in a consistent write order and that the replicas can always be used to continue working from or to restore the production source.

Storage Replication Adapter

The EMC RecoverPoint SRA for VMware vCenter Site Recovery Manager provides integration of SRM with RecoverPoint. The RecoverPoint SRA supports discovery of arrays attached to RecoverPoint and of consistency groups that are enabled for management by SRM. It also supports SRM functions such as failover and failover testing by mapping SRM recovery plans to the appropriate RecoverPoint actions.

Journal volumes

Journal volumes hold data that is waiting to be distributed to target replica volumes and retain copies of the data previously distributed to the target volumes. Journal volumes facilitate operational recovery to any point-in-time, as long as the relevant journal history is retained.


Manager

Each consistency group has its own set of journal volumes, which allows for differing retention periods across the consistency groups. Each consistency group has two or three journal volumes assigned to it:

Local journal volumes, if configured

Remote journal volumes, if configured

Production journal volume, which stores information about the replication process (called marking information) that is used to synchronize the replication volumes at the two sites much more efficiently, when required

Regardless of the mode chosen, RecoverPoint allows you to define an association between a production volume and the local and/or remote volumes to which it is replicating to, called a replication set.

These replication sets can be logically grouped into consistency groups. The consistency groups ensure that updates to the production volumes are written to the replicas in a consistent write order and that the replicas can always be used to continue working from or to restore the production source.

Each consistency group is in one of the following states:

Replicating: When the consistency group is enabled, the splitter is replicating to the RPAs, and the RPAs are transferring to the replica journal/journals.

Marking: If the RPAs are unable to transfer to the replica journal, the location of the changes is stored in the RPAs as well as on the production journal volume. When contact with the remote site is reestablished, the remote replica is synchronized, but only at those locations that were marked as having changed.

No marking/no replication: The splitter does not write to the RPAs. This can be caused by a manually disabled consistency group or by a disaster on the production site (no RPAs are available).

Initial replication


Manager

16

Solution architecture

This solution used a Symmetrix VMAX 10K storage array with an existing VNX5300 storage array to form a bi-directional disaster recovery environment. In this solution, we replicated both storage devices to each other with EMC RecoverPoint and designed automated recovery plans using SRM.

The technical design adopted in this solution demonstrates an effective disaster recovery technique using CLR technology. In our test scenario, the production systems VM2 and VM3 were initially in a VNX storage solution on site C. Later, VM2 was migrated to VMAX storage on site A.

Figure 2 depicts the final architecture of the solution environment after migration.

Figure 2. Solution architecture

Overview

Physical design


Manager

We used two data centers to host SAP systems VM2 and VM3:

1. We used two data centers:

Site A: Contained VMAX 10K; production site of VM2 and recovery site of VM3.

Site C: Contained VNX5300; production site of VM3 and recovery site of VM2.

2. We configured the two pairs of RPAs contained in each data center (RPA1 and RPA2 on site A, RPA3 and RPA4 on site C) as a cluster.

3. We partnered each RPA cluster with an SRM server.

4. We configured consistency groups on the RecoverPoint Management application so that the datastores of each SAP system can be replicated locally and remotely.

5. We installed two SAP systems:

VM2 – SAP IDES ERP, Oracle, Suse Linux, distributed SAP installation

VM2 was initially installed on the VNX5300 array and then migrated to the VMAX 10K array. This DR solution is also used in migrating SAP systems in VNX to VMAX storage.

VM3 – SAP IDES ERP, SQL Server, Windows, central SAP system installation

VM3 was independently installed on VMAX, and then migrated to the VNX array. This use case demonstrates bi-directional recovery.

6. We integrated SRM with RPA and configured it to allow mutual failovers between the two sites.

Table 3 details the hardware resources for the solution.

Table 3. Solution hardware environment

Hardware Quantity Configuration

Storage (Production site A)

1 EMC Symmetrix VMAX 10K with:

Single engine

42 GB cache memory

66 x 400 GB, 15K FC drives, plus spares

13 x 200 GB Flash drives, plus spares

34 x 2TB SATA drives, plus spares

8 Gb FC connectivity

1 Gb network connectivity

Enginuity 5876.159.102

Hardware resources


Manager

18

Hardware Quantity Configuration

Storage (site C) 1 EMC VNX5300 with:

58 x 600 GB, SAS drives, plus spares

9 x 200 GB Flash drives, plus spares

30 x 2 TB SATA drives, plus spares

8 Gb FC connectivity

1 Gb network connectivity

VNX Operating Environment

Block software version: 05.31

VMware ESXi™ server

2 2 x quad-core CPUs, 2.93 GHz,

64 GB RAM

4 x 8 Gb FC HBA

2 x 1 Gb Ethernet

1 4 x 10 cores CPUs, 2.00 GHz

400 GB RAM

4 x 8 Gb FC HBA

2 x 1 Gb Ethernet

1 2 x quad-core CPUs, 1.86 GHz

16 GB RAM

2 x 4 Gb FC HBA

2 x 1 Gb Ethernet

Network switch 2 1 Gb Ethernet switches

FC switch 4 8 Gb/s FC switches, 2 per site

RecoverPoint appliance

4 Gen 4, 2 RPAs per site

Table 4 details the software resources for the solution.

Table 4. Solution software environment

Software Version Purpose

EMC Unisphere for VMAX 1.0.0.1018 Symmetrix VMAX configuration and management tool

EMC Unisphere for VNX 1.1 VNX management software

EMC Solutions Enabler 7.4 Symmetrix VMAX management software

EMC RecoverPoint 3.5 EMC replication software, installed on each of the 4 RPAs

VMware vSphere 5.0.0 B623860 Hypervisor hosting all virtual machines

Software resources


Manager

Software Version Purpose

VMware vCenter 5.0.0 B755629 Management of VMware vSphere

VMware vCenter Site Recovery Manager

5.0.1 B633117 Managing failover and failback of virtual machines

EMC RecoverPoint adapter for VMware vCenter Site Recovery Manager

2.0 EMC software for integrating RecoverPoint and SRM

SUSE Linux Enterprise Server 11

2.6.32.24 Server operating system for SAP VM2

Microsoft Windows Server 2008 R2 *64 Server operating system for SAP VM3

Microsoft SQL Server SQL Server 2008 R2

Database for SRM

Table 5 details the virtual machine resources for the solution.

Table 5. Virtual machine environment

Role Purpose Quantity Configuration

SAP VM2 ABAP SAP Central Services server (ASCS)

Critical production 1 2 vCPUs, 8 GB RAM, SLES 11, 2 vNICs

SAP VM2 database server Critical production 1 8 vCPUs, 16 GB RAM, SLES 11, 1 vNIC

SAP VM2 Application servers Critical production 3 8 vCPUs, 16 GB RAM, SLES 11, 2 vNICs

SAP VM3 Central System Non-critical production 1 16 vCPUs, 24 GB RAM, Windows 2008 R2, 2 vNICs

SRM site A application SRM management system 1 2 vCPUs, 4 GB RAM, Windows 2008 R2, 1 vNIC

SRM site A database SRM management system 1 2 vCPUs, 8 GB RAM, Windows 2008 R2, 1 vNIC

SRM site C application SRM management system 1 4 vCPUs, 8 GB RAM, Windows 2008 R2, 1 vNIC

SRM site C database SRM management system 1 2 vCPUs, 8 GB RAM, Windows 2008 R2, 1 vNIC

Load runner controller Simulate user activity 1 2 vCPUs, 16 GB RAM, Windows 2003 R2, 1 vNIC

Load runner generator Simulate user activity 1 8 vCPUs, 8 GB RAM, Windows 2003 R2, 1 vNIC

Virtual machine resources


Manager

20

Table 6 details the SAP application environment for the solution.

Table 6. SAP application environment

Specifications VM2 VM3

Number of datastores, type

2, VMFS 1, VMFS

Operating system Suse Linux Enterprise Server (SLES) 11

Windows 2008 R2

Database application

Oracle 11g R2 database server

SQL Server 2008 R2

Size 1 TB SAP data, 1.4 TB allocated in 2 datastores

630 GB SAP data, 1 TB allocated in 1 datastore

SAP application SAP ERP 6.0 EhP 4 ABAP IDES

SAP ERP 6.0 EhP 5 ABAP IDES

Installation type Distributed Central

SAP application environment


Manager

Solution configuration

Building this solution involves two major steps:

Replication configuration

SAP-specific customization

The storage configuration procedure is as follows:

1. Prepare VMAX 10K and VNX5300 for RecoverPoint connectivity.

a. Create LUNs and replica volumes.

b. Provision RPA repository and gatekeepers.

c. Tag all VMAX 10K devices to be protected by RecoverPoint.

2. Configure RecoverPoint.

a. Install RecoverPoint.

b. Configure RecoverPoint access to the splitters on the VMAX 10K and VNX arrays.

c. Create consistency groups.

d. Enable bi-directional replication.

3. Configure SRM.

a. Installing SRM virtual machines and SRA for integration to RPA.

b. Establish connections between protected and recovery sites.

c. Configure RecoverPoint array managers.

d. Configure protection groups.

e. Configure inventory mappings.

f. Customize virtual machine recovery options.

g. Customize recovery site IP addresses.

h. Create recovery plan.

Appendix B: Setting up replication provides more detailed explanations of these steps. For a detailed discussion of all the steps, refer to Site Recovery Manager Administration – vCenter Site Recovery Manager 5.1.

Enabling bi-directional replication

Adding a second consistency group with the roles of the data centers reversed (that is, production volumes on site C are replicated to replica volumes on site A) effectively enables bi-directional replication.

You can use single set of paired SRM (and RecoverPoint) sites to protect virtual machines in both directions. Each site can serve as a production and recovery site simultaneously, but for a different set of virtual machines. Bi-directional operation is

Overview

Configuring replication


Manager

22

not limited to array-based replication. However, whenever you use an array based-replication as in this solution, each of the array’s LUNs replicates in one direction only. Two LUNs in the same array can replicate in different directions from each other.

Some SRM customizations are valuable in automating recovery in an SAP environment. A distributed SAP system requires the following steps during the recovery startup:

1. Change IP addresses of the SAP instances.

2. Update hosts files to reflect internal IP settings.

3. Mount the /sapmnt/<SID>/SYS/exe, /sapmnt/<SID>/SYS/global, and /sapmnt/<SID>/SYS/profile folders of the ABAP Central Services instance to the SAP application and database servers.

4. Start the applications in this order:

a. SAP ABAP Central Services instance

b. Oracle database instance

c. All SAP ABAP Application instances

This solution used the flexibility of the SRM recovery plan by inserting operating system commands that perform the previous tasks during a failover. This fully automates the startup process until all SAP systems are ready for login.

To achieve the desired level of automation, the test environment was prepared as described in the following sections.

Heterogeneous test environment

We considered diversity when we built the test environment for a more extensive scope and more credible results. This includes SAP releases, types of databases and operating systems. We also tested the solution across components with non-uniform time zones.

Customization specific to SAP


Manager

Replication mode

We set up CLR in both VM2 in the consistency group SiteA_PRD and VM3 in SiteC_TST, as shown in Figure 3 and Figure 4.

Figure 3. RecoverPoint topology showing VM2 in CLR mode

Figure 4. RecoverPoint topology showing VM3 in CLR mode


Manager

24

Startup priorities

SRM allows virtual machines to be started up in a particular order by assigning a priority. Priority 1 virtual machines are started up first, followed by priority 2 virtual machines, and so on. We changed the priorities of the virtual machines as follows:

Priority 1—ASCS; to make the SAP shared folders available for mounting

Priority 2—Database instance; to ensure that the database is started before the SAP application instances are started

Priority 3—All application instances

Figure 5 shows the priority assignment.

Figure 5. Virtual machine startup priorities

Network settings

We configured two static IP addresses for each virtual machine:

An internal 192.168.x.x IP address

A service IP for log in (SAP virtual machines only)

You can achieve this by configuring two network interface cards (NICs) for each virtual machine or by creating a script that binds the internal IP addresses as virtual. We used the former in this solution.

/etc/hosts file

SRM updates only the entry of the virtual machine's own hostname. Ensure that the internal IP addresses are used before and after recovery. Steps are as follows:

1. Create a hosts.i file as a template that maps the hostnames to the internal IP addresses.

2. Insert a script as a post-startup recovery step to reset the hosts files with contents from the hosts.i file to keep consistency.


Manager

SAP shared folders

You can insert a mount command as a post-startup recovery step on every application instance or add the mount point entries in the /etc/fstab file. This solution used the latter technique to mount the /sapmnt/<SID>/SYS/exe, /sapmnt/<SID>/SYS/global, and /sapmnt/<SID>/SYS/profile folders of the ASCS instance to the SAP application servers.

SAP startup sequence

We inserted the following scripts as a post-startup recovery step on the recovered virtual machine, with a 5-minute timeout. These scripts were assigned as follows:

On all virtual machines: change the host script:

cp /etc/hosts.i /etc/hosts

On SAPVM2DB (Priority 2): assign the startdb script:

mount -a

su - oravm2 -c "lsnrctl start"

su - vm2adm -c "startdb"

ON SAP virtual machines: assign the startsap script:

su - vm2adm -c "startsap"

Figure 6 shows the scripts that were applied to the automated application startup after powering on.

Figure 6. Scripts to automate application startup in post-power on steps


Manager

26

SAP user logon

All SAP instances communicate with internal IPs. Thus, the hosts file only included internal IP addresses, as shown in Figure 7.

Figure 7. Hosts file

For SAP logon, we created two entries; one is for logging in to the production site A, the other for the recovery site C, as shown in Figure 8. Note that full IP addresses will not be shown in Figure 8 and Figure 9.

Figure 8. SAP logon

We created two logon groups using transaction code SMLG, which corresponds to the two sets of IP addresses. Users can just switch the logon group used in the SAP GUI after the failover, as shown in Figure 9.


Manager

Figure 9. Logon groups created in VM2

Load-generation environment

We used LoadRunner for every test in this solution to run successive updates in SAP tables and to simulate a real-world environment. The timestamps on the table entries inserted by LoadRunner served as counterchecks of the timestamps recorded in SRM or RecoverPoint, but at the SAP level. Timestamps from LoadRunner updates provide more accurate application-level RTO and RPO key performance indicators (KPIs) for testing. The tests were considered successful when the results from SRM/RecoverPoint and SAP were consistent.

Bi-directional recovery testing

We performed the recovery tests on both VM2 and VM3 systems to validate bi-directional recovery. However, because the procedure was the same, for brevity this white paper shows only the screenshots of the more complex VM2 system.

System integrity and quality control

We took timestamps carefully to ensure consistency among all components. We performed SAP checks before and after testing for each scenario according to the best practices to ensure that the failed-over system was healthy technically and functionally.


Manager

28

We performed checks on the following transactions/commands:

SICK – System installation check

SM51 – Instance overview

SM50 – Work process overview, per instance

SM21 – System logs

ST22 – ABAP dumps

DB02 – Database management

SM12 – Display and delete locks

SM13 – Administrate (sic) Update Records

df–Ph1 – For Linux file systems

Note These controls were observed, but the raw data and screenshots are not shown in this paper.

1 df –Ph displays file systems, but in a POSIX human-readable format.


Manager

Test and validation

In the context of operational efficiency and effectiveness, this section aims to test and demonstrate the following capabilities:

Nondisruptive failover testing

This solution allows online testing by restoring a snapshot and encapsulating it in a private network during the test. This procedure effectively reduces the need for disruptive testing.

Disaster recovery

This solution allows the client to quickly automate the entire failover operation with a few clicks of a button.

Failback testing

Because site A may be geographically closer to users, or applications running previously on it are presumably more resource intensive, failback to the original site is recommended. This operation reinstates failed-over systems on VNX to their original servers as soon as possible. This design allows for rapid failback to reduce downtime for the operation.

PIT recovery

The solution aims to show how easy it is to perform PIT recovery, eradicating the need to restore and roll forward entire databases just before the logical corruption occurs.

Test scenario

An organization originally has a central site C that contains SAP production systems VM2 and VM3 on VNX5300. A second auxiliary site A that contains VMAX 10K was built. VM2 will be failed over/migrated from site C (VNX5300) to site A (VMAX 10K) using this solution.

To prepare the migration, we simulated disaster recovery failover on VM2 and VM3 on site C while both were in production.

Test objectives

The test objectives were:

Performing failover testing on the recovery site without shutting down any virtual SAP system

Testing bi-directional recovery

Achieving an RTO of 30 minutes

Overview

Test scenario 1: Nondisruptive failover testing


Manager

30

Test procedure

The test procedure was as follows:

1. Ran load on VM2.

2. Took timestamps and performed test recovery on VM2 and VM3. During testing, we did a health check on both systems to verify integrity.

3. Logged in to the SAP GUI, took the timestamp, and performed an SAP health check.

4. Checked the availability logs of VM2 and VM3 on site C to see if the production systems had gone down during testing.

Findings

Table 7 shows the findings of this test.

Table 7. Nondisruptive failover test findings

Success criteria Findings Result

VM2 and VM3 stay online Availability logs showed that both systems were available during failover

Passed

Test failover is 100% successful Logs completed without errors Passed

Test VM2 is operational System health status validated availability and integrity after completion

Passed

Duration of test recovery falls within RTO

Timestamp in SRM showed recovery time of 00:13:47

Passed

To summarize this test:

Initiation of the testing required only three steps. Additional manual steps were not needed until completion, as shown in Figure 10.


Manager

Figure 10. Test recovery execution

The recovery was successful in SRM, as shown in Figure 11.

Figure 11. Final status of recovery plan in SRM after testing

Conclusion

A fully consistent bi-directional SAP DR was easily made available for DR testing in less than 30 minutes while primary sites were available for production users with continued DR replication.

1 1

1 2

1 3


Manager

32

Test scenario

After a successful simulation in scenario 1, the same production SAP system VM2 on site C was failed over to site A for planned migration/disaster recovery. To simulate the disaster, we terminated the database processes and triggered disaster recovery from site C to site A.

Test objectives


Performing failover on the recovery site


Test procedure


1. Ran load on VM2.

2. Took the timestamp and performed planned migration/disaster recovery on VM2.

3. Logged in to the SAP GUI, took the timestamp, and performed an SAP health check.

4. Checked the availability log of VM2 on the protected site.

Findings


Table 8. Disruptive failover test findings


Test failover is 100% successful Logs completed without errors Passed

Recovered VM2 is operational System health status validated availability and integrity after completion

Passed

Duration of failover falls within RTO

Timestamp in SRM showed recovery time of 00:15:06

Passed


Initiation of the testing required only four steps. Additional manual steps were not needed until completion, as shown in Figure 12.

Test scenario 2: Disaster recovery


Manager

Figure 12. Recovery execution using VMware SRM

1 1

2

1 3

1 4


Manager

34

The recovery was successful in SRM and validated in SAP, as shown in Figure 13.

Figure 13. Final status of recovery plan executed in VMware SRM

Conclusion

The actual failover required only four steps. A fully consistent SAP DR was available in less than 30 minutes. Manual intervention was not needed after the recovery was initiated. This technique can be used to migrate systems from VNX to VMAX or vice versa.

Test scenario

VM2 was on site A after recovery in scenario 2. Assume that site C is now fully operational after it has “crashed.” We then re-established protection between the two sites, but with the roles reversed for VM2: Site A was the protected site, replicating to recovery site C. VM3 remained unchanged.

Test objectives


Performing failover on the recovery site


Test procedure


1. Issued reprotection on VM2 to reverse the direction of protection.

2. Took the timestamp and recovered VM2 to the original site.

3. Logged on to the SAP GUI, took the timestamp, and performed an SAP health check.

Test scenario 3: Failback


Manager

4. Checked the VM2 availability log in the protected site.

5. Failed over to site C again to verify if the reprotection was effective.

Findings


Table 9. Failback test findings


Reprotection is 100% successful

Logs completed without errors Passed

Failover is 100% successful Logs completed without errors Passed

Recovered VM2 is operational System health status validated availability and integrity after completion

Passed

Duration of failback falls within RTO

The timestamp in SRM showed:

Reprotection time of 00:01:48

Recovery time of 00:16:48

Passed


Initiation of reprotection required just four steps. Additional manual steps were not needed until completion, as shown in Figure 14. Then another four steps were needed for the failover.

Figure 14. Reprotection procedure

1 1

1 2

1 3

1 4


Manager

36

Figure 15 shows how reprotection reversed the direction of replication so that further post-processing was no longer necessary.

Figure 15. SRM reprotection log showing successful reprotection in a reverse direction

Figure 16 compares the replication status in the RecoverPoint Manager application after reprotection was triggered. The replication source and target volumes were automatically changed by SRM as well. Arrows were added for reference.


Manager

Figure 16. RecoverPoint VM2 topologies after failover and failback

Conclusion

Reprotection needed only four steps and the actual failback needed another four steps. All DR changes to databases and file systems were reinstated to the original production site in less than 30 minutes.

Test scenario

The system was corrupted because of an application logic error after a transport. We performed a point-in-time (PIT) recovery to restore the system before the transport.

Test objectives


Performing PIT recovery without using a database restore


Test scenario 4: Point-in-time recovery

After failover reprotection

After failback reprotection


Manager

38

Test procedure


1. Populated the table VBAK and captured a timestamped screenshot of the final entries.

2. Created a RecoverPoint bookmark. The screenshot taken previously represented the system state of the bookmark, as shown in Figure 25.

3. Populated the table VBAK further and captured a timestamped screenshot of the final entries. These entries represented the erroneous data that caused the PIT restore.

4. Used RecoverPoint to restore VM2 to a bookmark:

a. Powered off VM2 in vCenter.

b. Recovered production.

c. Enabled image access of production.

d. Resumed production.

e. Powered on VM2 in vCenter.

5. Logged on to the SAP GUI, displayed the table VBAK, and captured a timestamped screenshot of the final entries.


Manager

Findings


Table 10. Point-in-time recovery test findings


VM2 is operational after the PIT restore

SAP log in and health check verified application integrity

Passed

PIT restore is successful Timestamps were consistent through the screenshots but the “erroneous” data was no longer found

Passed

Duration of restore falls within RTO

The entire process completed in 00:25:00 Passed

Figure 17 shows the bookmark S1_T1 created in RecoverPoint.

Figure 17. RecoverPoint journal bookmark

Figure 18 shows that the final bookmarked entry was Sales Doc 3451802 inserted at 23:12:15. Figure 18 shows the System Status window on the right-hand side.

Figure 18. Table VBAK status during RecoverPoint bookmarking


Manager

40

Figure 19 shows that user activity resumed after bookmarking and the table was populated further. We captured this image approximately 6 minutes after capturing Figure 18.

Figure 19. VBAK table after RecoverPoint bookmarking

Figure 20 shows the RecoverPoint status in the middle of the restore process. We captured this image 10 minutes after capturing the previous image.

Figure 20. RecoverPoint in the middle of the restore process


Manager

Figure 21 shows the status after the PIT recovery. Note that the result of the PIT recovery is correct and consistent, even if the time zone of the Management GUI (Figure 20, Windows time) is 8 hours ahead of the RecoverPoint server time and 12 hours ahead of the SAP GUI time. The latest entry was the same sales document, that is, Sales Doc 3451802.

Figure 21. VBAK status reverted to when the bookmark was created

Conclusion

Data recovery from a corruption can occur in minutes without the need of a full database restore or file system recovery.


Manager

42

Conclusion

This solution offers effective, affordable, and efficient disaster recovery protection for critical business functions without wasting previous investments. This strategy significantly improves business agility and scales out for common SAP production system and test system needs, providing:

Bi-directional replication of a central SAP environment and distributed SAP environments between heterogeneous arrays by using the integrated RecoverPoint splitters for Symmetrix VMAX 10K and VNX across two sites.

Rapid and automated failover for planned and unplanned maintenance, enabled by integration of RecoverPoint and SRM.

Nondisruptive disaster recovery testing using RecoverPoint and SRM.

Point-in-time database recovery in a virtualized environment from bookmarked RecoverPoint images.

Redeemed storage investment by reusing VNX as DR storage after migration to Symmetrix VMAX.

The key findings of this proven solution are as follows:

This solution is able to manage the varying DR complexities of SAP environments.

Initiating the failover testing requires only three steps. A fully consistent bi-directional SAP DR is easily made available for DR testing in less than 30 minutes, while the production systems remain completely undisrupted. Additional manual steps are not required until completion.

The actual failover requires only four steps to initiate. A fully consistent SAP DR is available in less than 30 minutes. Manual intervention is not required after the recovery is initiated. This technique can be used for VNX/VMAX migrations.

Initiation of reprotection requires only four steps. The actual failback requires another four steps. All DR changes to databases and file systems are reinstated to the original production site in less than 30 minutes.

Data recovery from a corruption can occur in minutes without the need of a full database or file system recovery using RecoverPoint point-in-time recovery.

Summary

Findings


Manager

The solution implements control measures to protect against human errors and technical limitations:

We performed the tests using super user accounts for convenience, but the solution is capable of role-based access control (RBAC) to manage configuration and administrative activities in a more granular sense, if needed. For more information, refer to Site Recovery Manager Administration–vCenter Site Recovery Manager 5.1.

While the entire failover process is automated, the trigger itself is manual. This ensures that a failover can be initiated only after proper processes are followed, executive approvals have been granted, and a state of disaster (or resumption of normal operations) has been declared.

The failover operation can be triggered with one or two steps. However, we put confirmation prompts in place to prevent inadvertent executions.


Manager

44

References

For additional information, see the following white papers:

Business Continuity and Disaster Recovery for Oracle 11g Enabled by EMC Symmetrix VMAX 10K, EMC RecoverPoint, and VMware vCenter Site Recovery Manager

EMC Symmetrix VMAX RecoverPoint Splitter and Enhancements in Enginuity 5876

For additional information, see the following product documents:

EMC Solutions Enabler Symmetrix Array Controls CLI Version 7.3 Product Guide

Site Recovery Manager Installation and Configuration–vCenter Site Recovery Manager 5.1

Site Recovery Manager Administration–vCenter Site Recovery Manager 5.1

For additional information, see the following SAP notes:

Note 923610: Memory parameter recommendations for banking systems

Note 1171650: Automated Oracle DB parameter check

Note 1431798: Oracle 11.2.0: Database Parameter Settings

White papers

Product documentation

SAP notes


Manager

Appendix A: RecoverPoint replication modes CDP continuously captures and stores data modifications locally, enabling local recovery from any PIT with no data loss. CDP supports both synchronous and asynchronous replications. Figure 22 illustrates the CDP workflow.

Figure 22. CDP workflow

The CDP workflow is follows:

1. The application server issues a write operation to a LUN protected by RecoverPoint. The write is intercepted by the RecoverPoint array splitter.

2. The splitter “splits” (duplicates and distributes the write request) to both the protected volume and RPA.

3. (3a) RPA acknowledges back to the splitter when receiving the write. (3b) At the same time, RPA moves the data into the local journal volume, along with the timestamp and any bookmarks for the write operation.

4. After the write request has been secured in the journal, it is then applied to the target replica volumes. The write order is preserved during distribution.

Continuous data protection


Manager

46

Continuous remote replication (CRR) supports synchronous and asynchronous replication between remote sites and a wide area network (WAN). CRR supports synchronous replication when the remote sites are connected through Fibre Channel (FC) and provides zero RPO. Asynchronous replication provides crash-consistent protection and recovery to specific points-in-time, with minimal RPO.

Figure 23 illustrates the CRR workflow.

Figure 23. CRR workflow

The CRR workflow is follows:


2. The splitter duplicates the write and sends it simultaneously to the protection volume and to its local RPA.

3. The local RPA receives the write operation:

3a. In asynchronous mode, acknowledgment is sent back from the local RPA to the splitter.

3b. In synchronous mode, acknowledgement is instead delayed until the write is received by the remote RPA. This request is bundled with other writes, deduplicated (that is, redundant blocks are removed), sequenced,

Continuous remote replication


Manager

and timestamped. The bundle is then compressed and transmitted with a checksum for delivering to the remote RPA cluster.

3c. If the receiving RecoverPoint is not reachable, the write request is temporarily stored on the production journal volume until the remote RecoverPoint is accessible once again. The write entries are moved from the production journal to the target journal and normal replication is resumed.

4. When the package is received on a remote site, the receiving RPA verifies the checksum first to ensure integrity and then decompresses the data.

5. The remote RPA writes the data to the journal volume on the remote site.

6. The write is then distributed to the target replica volumes through an RPA FC connection after the data has been stored safely in the journal. The write order is preserved during distribution.

CLR is a combination of CRR and CDP and provides concurrent local and remote data replication. Figure 24 illustrates the CLR workflow.

Figure 24. CLR workflow

Concurrent local and remote replication


Manager

48

The CLR workflow steps are as follows:


2. The splitter duplicates the write and sends it simultaneously to the protection volume, and to its local RPA.

3. The local RPA receives the write operation:

3a. In asynchronous mode, acknowledgment is sent back from the local RPA to the splitter.

3b. The write request is then sent to the local journal and applied to the local replica volume. This completes the CDP sequence.

3c. In synchronous mode, acknowledgement is instead delayed until the write is received by the remote RPA. This request is bundled with other writes, deduplicated (that is, redundant blocks are removed), sequenced, and timestamped. The bundle is then compressed and transmitted with a checksum for delivery to the remote RPA cluster.

3d. If the receiving RecoverPoint is unreachable, the write request is temporarily stored on the production journal volume until the remote RecoverPoint is accessible once again. The write entries are moved from the production journal to the target journal and normal replication is resumed.

4. When the package is received on a remote site, the receiving RPA verifies the checksum first to ensure integrity, and then decompresses the data.

5. The remote RPA writes the data to the journal volume on the remote site.

6. The write is distributed to the target replica volumes through an RPA FC connection after the data has been stored safely in the journal. The write order is preserved during distribution. This completes the CRR sequence.

CDP is normally used for PIT recovery where business data or application logic has been severely compromised. On the other hand, CRR is normally used for disaster recovery. CLR thus implies that two replicas are made for every volume protected (local and remote) to protect from both scenarios.


Manager

Appendix B: Setting up replication

Configuring the RecoverPoint splitter on the storage array requires the following volumes to be provisioned beforehand.

Step 1: Creating LUNs and replica volumes

Create the following LUNs for RecoverPoint usage:

One repository volume (at least 3 GB) for each RPA cluster.

One replica volume each in the protected and recovery site for every consistency group.

Three journal volumes (production journal, local journal, and remote journal volumes) for every site.

Eight unique gatekeeper volumes each for RPA1 and RPA2.

Figure 25 shows how we created the gatekeeper volumes using Unisphere for the VMAX GUI.

Figure 25. Provisioning gatekeeper volumes using Unisphere

Consider the following during capacity planning and risk assessment:

The size of a replica volume must be equal to or greater than the size of the protected volume. For example, if the protected volume is 500 GB, CDP or CRR consumes an additional 500 GB and CLR consumes at least 1,000 GB, excluding the journal volumes. This means a CLR consumes twice as much as a CDP or CRR alone.

The journal size depends on the RPO. Longer RPO requires larger journal sizes.

Different consistency groups can be created in the same configuration. Production systems can be assigned to a consistency group in CLR mode while others can be assigned to a group in either CDP or CRR mode.

Preparing VMAX 10K and VNX5300 for RecoverPoint connectivity


Manager

50

SAP systems falling under the same consistency group will failover as one bundle. Assigning a separate consistency group to each SAP system is highly recommended.

The size of the journal volumes should reflect the RPO. You need to determine the expected peak change in the environment to calculate the minimum size of the journal volume.

The formula is:

( ) ( )

( )

Note Target side log size is 20 percent (0.2) according to the best practice and an additional 5 percent is allocated (1.05) for internal use.

For example, to calculate the minimum journal size to satisfy a 48-hour rollback requirement (172,800 seconds) with an estimated 10 Mb new data writes per second:

( ) ( )

( )

Step 2: Provisioning RPA repository and gatekeepers

To provision RPA repository and gatekeepers, create masking views in VMAX and storage groups in VNX to present these volumes to the RPA cluster.

We created the following masking views for this purpose:

Recoverpoint_view: This presents the repository, journal, and replica volumes to all nodes in the RPA cluster.

RPA1_GK_view: This presents the eight gatekeepers to RPA1.

RPA2_GK_view: This presents the eight gatekeepers to RPA2.

Step 3: Tagging all VMAX 10K devices to be protected by RecoverPoint

As of RecoverPoint 3.4 (SP1 and later), production and replica LUNs must be tagged for RecoverPoint use in VMAX 10K splitter configurations. Otherwise, the devices will not be accessible to the splitter. Note that this step is not required for VNX storage.


Manager

Figure 26 shows how we tagged devices through Unisphere for VMAX.

Figure 26. Tagging devices using RecoverPoint for VMAX

Alternatively, the following command can be executed from the Solution Enabler for every device to be tagged:

symconfigure –sid <Symmetrix ID> -cmd “set dev <device list or

device range> attribute=RCVRPNT_TAG;” commit

Figure 27 shows the output of this command.

Figure 27. Tagging a RecoverPoint device from Solution Enabler

To verify the device tag status, input the following command:

symdev –sid <Symmetrix ID> show <device list or device range>


Manager

52

Figure 28 shows the status with the device tagged as RecoverPoint.

Figure 28. Verifying the RecoverPoint tag status

It is a best practice to tag also the RecoverPoint gatekeepers, journal volumes, and repository volumes.

Step 1: Installing RecoverPoint

As shown in Figure 29, the RecoverPoint Installer wizard guides you through the entire setup procedure including setting up the RPA cluster. Perform the steps described in the section Preparing VMAX 10K and VNX5300 for RecoverPoint connectivity before running the wizard.

Figure 29. RecoverPoint Installer wizard

Step 2: Configuring RecoverPoint access to the splitters on the VMAX 10K and VNX arrays

You need to configure the VMAX 10K and VNX splitters in RecoverPoint to “bridge” the replication. We did this by using the New Splitter wizard in RecoverPoint through Splitter > Add New Splitter, as shown in Figure 30.

Configuring RecoverPoint


Manager

Figure 30. Adding a new splitter in RecoverPoint

Step 3: Creating a consistency group

After successfully installing RecoverPoint, you can configure the consistency groups for the replication setup.

The New Consistency Group wizard guides you during the entire setup. You can find the wizard in the RecoverPoint interface by right-clicking Consistency Groups and selecting Add Group, as shown in Figure 31.

Figure 31. Adding a consistency group in RecoverPoint


Manager

54

Consider the following before launching the wizard:

Replication policy settings for the consistency group

Which site should be considered as the production site and which should be considered as the recovery site?

Which type of replication should be used (CDP/CRR/CLR)?

A one-to-one mapping of the production volumes to corresponding remote replica volumes

Which volumes are allocated for local and remote journal

Installation prerequisites

Before installing SRM, ensure that the following prerequisites are met:

Each site must contain a vCenter server with at least one vSphere data center being managed.

The recovery site must support array-based replication with the protected site.

At least one virtual machine must be located on a datastore that is replicated by RecoverPoint on the protected site.

A reliable IP network is required to connect the two sites.

The recovery site must have access to the same public and private networks as the protected site, though not necessarily having access to the same range of network addresses.

VMware tools must be installed on all virtual machines.

Step 1: Installing SRM virtual machines and the SRA for integration to RPA

Perform the following steps:

1. Install a database server on each site for SRM use. The databases will store the recovery plans and inventory information.

2. Install an SRM server on both sites.

Configuring SRM


Manager

3. Install a RecoverPoint SRA on the SRM server on both sites to enable integration.

4. Install the SRM plug-in on at least one vSphere client. SRM configurations are transparent to both sites, but it is highly recommended to install the SRM plug-in on both sites.

For detailed steps, refer to VMware vCenter Site Recovery Manager Installation and Configuration—Center Site Recovery Manager 5.1.

Step 2: Establishing connections between protected and recovery sites

Before configuring SRM, change the consistency group’s management mode to “Group is managed by SRM, RecoverPoint can only monitor. You can change this on the Policy tab after selecting the consistency group, as shown in Figure 32.

Figure 32. Policy tab in a RecoverPoint consistency group

When this is completed, the two SRM servers must be connected in SRM to establish reciprocity. You can do this through the Connect to Remote Site wizard, by specifying the IP/port details and authenticating credentials of the other (remote) SRM server, as shown in Figure 33.


Manager

56

Figure 33. Connecting the remote site in SRM

The new SRM interface contains an easy-to-follow guide tab that explains the steps throughout the configuration process.

Step 3: Configuring RecoverPoint array managers

After the protected and recovery sites have been paired, configure the RecoverPoint array managers (through the SRAs installed earlier) for SRM to discover replicated devices, compute datastore groups, and initiate storage operations. To do this, use the Add Array Manager wizard, as shown in Figure 34.


Manager

Figure 34. Adding storage arrays in SRM

Typically, you need to perform this step only once after the sites are connected. Additional reconfiguration is not needed unless there is a change in connection information or credentials.

After this step, all configurations can be done on just one SRM and will automatically be synchronized to the other SRM server.

Step 4: Configuring protection groups

You can create a protection group after configuring the array managers.

A protection group is used to organize virtual machines that should be failed over together, such as in this solution where the SAP database, SAP Central Services instance, and three application instances are failed over as one entire set.

You can create many protection groups, but all virtual machines assigned to a protection group must belong to the same datastore group (consistency group).

The Create Protection Group wizard automatically detects the datastores protected by RecoverPoint and the virtual machines inside it. You need to specify which virtual machines among the list must be added to a protection group, as shown in Figure 35.


Manager

58

Figure 35. Creating a protection group in SRM

SRM then creates placeholder virtual machines automatically. Placeholders reserve the resources on the recovery site and act as a “shadow standby” before a recovery operation. The placeholders cannot be started independently of SRM and are removed altogether if the protection group is deleted.

Figure 36 shows the actual virtual machines and the placeholders on the remote sites.

Figure 36. Placeholders at each site

In this solution, we created a separate protection group for VM2 and VM3 respectively, because both systems belong to different datastores and are cross-replicating toward each other’s site.

Step 5: Configuring inventory mappings and placeholder virtual machines

As shown in Figure 37, the Resource mappings tab allows you to define which resources on the recovery site will serve as the counterparts to the ones on the protected site, such as network, resource pool, and virtual machines. SRM cannot

Site A Site C


Manager

protect a virtual machine unless it has valid inventory mappings for key virtual machine resources.

SRM derives the resource assignments of a newly created placeholder from the inventory mappings established on the protected site.

EMC recommends that you group virtual machines into folders for easier management.

Note A counterpart folder must exist in the recovery vCenter server as well.

Figure 37. Configuring resource mapping and placeholder in SRM

Step 6: Customizing virtual machine recovery operations

Especially in an SAP landscape, it is customary to start up applications according to the business priority (for example, SAP Production > Third Party Production > SAP Development and ERP > HR > BI), and in a particular order (such as DB server > SAP Central Services > App instances).

These specifics can be customized to suit the recovery requirements of an organization to ensure that even complex recovery plans can be fully automated.

This solution exploits this feature so that the SAP virtual machines are started up in the correct order, and a script is executed within the recovery sequence to start up the database, SCS, and all application instances, ready for login on handover.

1

2

3

4


Manager

60

Step 7: Customizing recovery site IP addresses

One of the complicating factors in a typical failover is the readjustment of IP settings to adapt to the infrastructure differences after recovery. You may need to readjust IP settings on multiple virtual machines.

SRM can also manage this tedious task. You can either do this individually by using the SRM plug-in, or by using a bulk IP customization utility (dr-ip-customizer.exe), which is especially useful for first-time configurations.

This utility generates a comma-separated value (CSV) file containing the IP settings of all virtual machines configured for SRM failover. You can generate and edit the file as follows:

1. Log in to the command prompt or console of the vCenter server on the recovery site.

2. By using the dr-ip-customizer utility in the bin folder of the SRM installation path, generate the existing IP table and specify the file name and location where it will be saved.

C:\Program Files (x86)\VMware\VMware vCenter Site Recovery

Manager\bin>dr-ip-customizer.exe –cfg ..\config\vmware-

dr.xml –cmd generate –o <csv filename to generate> –vc

<VMware vCenter server hostname>

3. Edit the CSV file and customize the existing IP settings according to the IP settings on the recovery site. Figure 38 shows the CSV file opens in a typical spreadsheet application. Note that full IP addresses are not shown in this figure.

Figure 38. CSV file where IP addresses can be changed

4. Run the dr-ip-customizer utility again to upload the edited CSV.

C:\Program Files (x86)\VMware\VMware vCenter Site Recovery

Manager\bin>dr-ip-customizer.exe –cfg ..\config\vmware-

dr.xml –cmd apply –csv <path to the csv file> –vc <VMware

vCenter server hostname>


Manager

Step 8: Creating a recovery plan

A recovery plan is like an automated runbook. It controls every step of the recovery process, including the order in which SRM powers on and powers off virtual machines, the network addresses that recovered virtual machines use, and so on.

When a recovery plan is created, it prescribes a series of recovery steps that SRM automates depending on the function executed. Figure 39 shows these functions.

Figure 39. Functions in SRM GUI

The following list explains each of the functions:

Test: Online simulated recovery; SRM performs all recovery steps inside a test network so that the test can run without disrupting the virtual machines in the production environment.

Cleanup: A function to undo the result of test recovery (such as unmounting recovered snapshots and returning placeholders) and to reset the recovery plan to a ready state.

Recovery: The actual migration of virtual machines in the recovery plan to the recovery site. There are two types that differ only in error tolerance:

Planned migration: If errors occur during migration, the execution is stopped and the errors must be resolved before resuming.

Disaster recovery: If errors occur, SRM attempts to continue execution.

Reprotect: An automated function that completely reverses the direction of protection after a failover. The original protected site becomes the new recovery site and the original recovery site becomes the new protected site, provided that the original site is operational.

Cancel: Function to terminate an ongoing test at any time or an actual recovery during certain steps.


Manager

62

Figure 40 shows an SRM recovery plan. These steps change depending on the function.

Figure 40. SRM recovery plan in progress

Recovery plans are flexible and customizable. You can make readjustments to the workflow as needed, such as:

Specifying steps that are run or skipped during test recoveries

Customizing timeouts to allow some time for a step to complete before the next step

Creating message prompts to force the plan to pause until the user acknowledges

Specifying the recovery priority of a virtual machine

Adding custom recovery steps for the SRM server or the recovered virtual machine using scripts

Documents

H11402: EMC Transforms IT for SAP: Fully Automated … · Basics of RecoverPoint replication ..... 13 Replication modes ... EMC Symmetrix VMAX 10K, EMC VNX5300, EMC RecoverPoint,