Welcome to Symmetrix VMAX SRDF/Star and Cascaded Star ... · 7/1/2016 · Use, copying, and distribution of any EMC software described in this publication requires an applicable

Copyright © 2014 EMC Corporation. Do not copy - All Rights Reserved.

Welcome to Symmetrix VMAX SRDF/Star and Cascaded Star Solutions. Copyright © 1996, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013 , 2014 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

EMC2, EMC, Data Domain, RSA, EMC Centera, EMC ControlCenter, EMC LifeLine, EMC OnCourse, EMC Proven, EMC Snap, EMC SourceOne, EMC Storage Administrator, Acartus, Access Logix, AdvantEdge, AlphaStor, ApplicationXtender, ArchiveXtender, Atmos, Authentica, Authentic Problems, Automated Resource Manager, AutoStart, AutoSwap, AVALONidm, Avamar, Captiva, Catalog Solution, C-Clip, Celerra, Celerra Replicator, Centera, CenterStage, CentraStar, ClaimPack, ClaimsEditor, CLARiiON, ClientPak, Codebook Correlation Technology, Common Information Model, Configuration Intelligence, Configuresoft, Connectrix, CopyCross, CopyPoint, Dantz, DatabaseXtender, Direct Matrix Architecture, DiskXtender, DiskXtender 2000, Document Sciences, Documentum, elnput, E-Lab, EmailXaminer, EmailXtender, Enginuity, eRoom, Event Explorer, FarPoint, FirstPass, FLARE, FormWare, Geosynchrony, Global File Virtualization, Graphic Visualization, Greenplum, HighRoad, HomeBase, InfoMover, Infoscape, Infra, InputAccel, InputAccel Express, Invista, Ionix, ISIS, Max Retriever, MediaStor, MirrorView, Navisphere, NetWorker, nLayers, OnAlert, OpenScale, PixTools, Powerlink, PowerPath, PowerSnap, QuickScan, Rainfinity, RepliCare, RepliStor, ResourcePak, Retrospect, RSA, the RSA logo, SafeLine, SAN Advisor, SAN Copy, SAN Manager, Smarts, SnapImage, SnapSure, SnapView, SRDF, StorageScope, SupportMate, SymmAPI, SymmEnabler, Symmetrix, Symmetrix DMX, Symmetrix VMAX, TimeFinder, UltraFlex, UltraPoint, UltraScale, Unisphere, VMAX, Vblock, Viewlets, Virtual Matrix, Virtual Matrix Architecture, Virtual Provisioning, VisualSAN, VisualSRM, Voyence, VPLEX, VSAM-Assist, WebXtender, xPression, xPresso, YottaYotta, the EMC logo, and where information lives, are registered trademarks or trademarks of EMC Corporation in the United States and other countries.

All other trademarks used herein are the property of their respective owners.

© Copyright 2014 EMC Corporation. All rights reserved. Published in the USA.

Revision Date: February 2014 Revision Number: MR-1WP-SRDFSTAR.5876.3.0

Symmetrix VMAX SRDF/Star and Cascaded SRDF Solutions 1


This course covers 3-data-center SRDF solutions, comprising Cascaded SRDF, SRDF/EDP, and SRDF/Star in Open Systems environments. Underlying SRDF technologies used to implement Star are discussed in detail. SRDF/Star using both concurrent and cascaded SRDF are covered. Positioning, architecture, and operations of SRDF/Star, as well as the procedures for moving a production workload to another site in the event of a planned or unplanned switch are described.



This module focuses on three SRDF data center solutions and cascaded SRDF.



The right remote replication solution can limit your exposure to planned and unplanned downtime, enabling fast application recovery and fast failback. In the event of a disaster or unplanned outage, data protection and faster business restart are critical across the organization.

EMC offers the largest number of choices for insuring data availability with its portfolio of three site replication solutions. Customers deploying EMC’s three site solutions can enable fast application restart in the event of local or regional disasters, along with having fast site failback using SRDF based data resynchronization.



SRDF/AR is an automated remote replication solution that uses both SRDF and TimeFinder to provide a periodic asynchronous remote replication of a restartable data image for UNIX and Windows environments. It is the least expensive of the listed solutions, because it can be configured to run on lower bandwidth networks than the other solutions. It offers a typical RPO of hours.

Concurrent SRDF permits the maintenance of two data copies. Usually, one copy running SRDF/S is maintained at a nearby location and offers zero data loss if the primary site fails. The second copy operating in SRDF/A mode offers an out of region recovery site with an RPO of seconds to minutes.

Cascaded SRDF is a 3 site disaster recovery configuration in which data from a primary site is synchronously replicated to a secondary site, and then asynchronously replicated to a tertiary site. The major benefit provided with a “cascading” configuration is its inherent capability to continue replicating from the secondary site to the tertiary sites in the event that the primary site goes down.

Available since Enginuity 5874, SRDF/EDP is a lower cost solution to achieve no data loss at an out-of-region site. Using cascaded SRDF, combined with diskless R21 devices in the intermediate (Pass-Thru) site, data passes through the intermediate to the out-of-region site. Symmetrix cache at the intermediate site buffers the synchronous I/O and converts it to asynchronous SRDF/A traffic.

SRDF/Star is a three-site disaster-restart solution that can enable resumption of SRDF/A with no data loss between two remaining sites, providing continued remote-data replication and preserving disaster-restart capabilities. It offers a combination of continuous protection, changed-data resynchronization, and enterprise consistency between two remaining sites in the event of the Workload Site going offline due to a site failure.



Cascaded SRDF uses a dual role SRDF R1/R2 device (referred to as SRDF R21 device) on the secondary site, which acts as both an R2 to the primary site and an R1 to the tertiary site.

The major benefit provided with a cascading configuration is its inherent capability to continue replicating from the secondary site to the tertiary sites in the event that the primary site goes down.

Since the copy at the secondary site is up to date, if the primary site fails, production can continue at either site with no data loss. If both the primary and secondary sites fail, the tertiary site can effect a disaster restart with data that is at most two SRDF/A cycles behind.

Since at least two copies of production data are always accessible, there is no need to provision BCVs at the tertiary site, as would be the case with SRDF/AR.



Cascaded SRDF introduces the concept of the dual role R1/R2 device, referred to as an R21 device. The R21 device is both an R1 mirror and an R2 mirror.

The underlying technology of cascaded SRDF devices is the same as concurrent RDF. The only difference is that one mirror is an R2 and the other an R1. Like a concurrent RDF device, each mirror of a cascaded RDF device must belong to a different RDF (RA) group.



The command symrdf list -sid 81 –cascade lists the cascaded SRDF devices configured on Symmetrix 81. The output lists two devices, BD and BE.

Note that these two devices belong to groups 19 and 79 simultaneously. In a particular relationship, the way to identify the personality of the devices is to read the flags under the MODES “T” column.

The entry S..2. in the line reporting on RDF group 19 indicates that the RDF devices in Symmetrix 81 in group 19 are running in synchronous mode and are operating as R2 devices. The entry A..1. indicates that the devices are running in asynchronous mode as R1 devices in group 79.



These are the SRDF modes permitted in cascaded SRDF when the R21 is a disk based device. While the R1 to R21 leg can run any mode of SRDF, the R21 to R2 leg is limited to asynchronous or adaptive copy disk mode. If the R1 to R21 leg is running in asynchronous mode, the R21 to R2 leg cannot be running in asynchronous mode.



These are the constraints for disk based R21 devices.

The restrictions shown on this slide pertain to disk based R21 devices. Additional restrictions apply to diskless R21 devices, and are covered in the SRDF/EDP section of this course.



Available with Enginuity 5874, SRDF/EDP is a lower cost solution to achieve no data loss at an out-of-region site. For this solution, cascaded SRDF mode of operation is used as the building block. By using a diskless R21 device in the intermediate site, the intermediate site provides data pass through to the out-of-region site.

SRDF/EDP supports SRDF/Star for continuous remote data replication protection. Due to a disaster, if that intermediate site goes offline, SRDF/Star permits the production and out-of-region sites to establish an asynchronous link with minimal resynchronization between sites A and C.



A diskless R21 device can offer a cache buffer external to the source Symmetrix. This buffer makes it possible to create a zero data loss SRDF/A solution, where traditional SRDF/A offers a RPO of seconds to minutes.

A diskless device cannot be mapped to the host. Therefore, no host will be able to directly access a diskless device for read or write. Diskless RDF devices are only supported on GIGE and Fiber RAs. All Symmetrix replication technologies, other than RDF (TimeFinder/Snap, TimeFinder/Clone, and Open Replicator), will not work with diskless devices as either the source or the target of the operation.

The R1 and R2 volumes must both be thin or, both must be standard.

(For example:

• Thin R1 diskless R21 Thin R2

• Standard R1 diskless R21 Standard R2)

Creation of the diskless devices can be done by the Customer Engineer via the bin file, or by using Config Manager. A sample of the syntax is shown on this slide.



These are the SRDF modes permitted in cascaded SRDF when the R21 is a diskless device. The R1 to R21 leg can run synchronous, adaptive copy disk, or adaptive copy write pending mode. The R21 to R2 leg is limited to asynchronous or adaptive copy write pending mode.



The diagram shows a configuration where the R1 devices in Symmetrix 80 are paired with diskless R21s in Symmetrix 81. The RDF group connecting the devices is number 20. The same devices are acting as R1 devices for R2 devices in Symmetrix 82. The group to which these devices belong is number 80. The pair state for both sets of relationships is Suspended.

The order in which the links are brought up is significant when using SRDF/EDP. Since the diskless devices cannot store data, the link between the R21 and the R2 has to be brought up first. Here, an attempt to synchronize the R1 to R21 device pair before synchronizing the R21 to R2 device pair fails. Performing the synchronization by synchronizing the R21 to R2 pair first and the R1 to R21 pair second succeeds.



First released with Enginuity 5874, R22 devices are a form of concurrent RDF device configured with two R2 SRDF mirrors that receive data from two different R1 devices. R22 devices are also supported at certain levels of Enginuity 5773. Only one of the R2 SRDF mirrors that belong to the R22 device can accept reads or writes from its R1 device at a time. The other SRDF mirror is marked as blocked to its R1 device.

R22 devices are designed to benefit SRDF/Star solutions that operate by using only the active SRDF links, although a recovery path is in place to ensure that you can fail over to another site in case of a disaster at the production site.

The figure on the slide illustrates an R22 device used in a Concurrent SRDF/Star solution. Note that all Star configurations (Concurrent, Cascaded, and EDP) support the use of R22 devices. In all cases, the R22 device is configured with two SRDF R2 mirrors, but only the SRDF R2 mirror that accepts I/Os from active SRDF links is enabled at a time.

The main advantage of R22 devices is to make Star more resilient and easier to reconfigure after a failure. We will cover SRDF/Star in more detail later on in this course. All Star examples later in this course use R22 devices.



This is a setup that is intended to support SRDF/Star. The first symrdf list command shows that devices BD and BE belong to groups 19 and 79 on Symmetrix 81. The second symrdf list command shows that devices BD and BE belong to groups 49 and 79 on Symmetrix 82. One can also see that devices BD and BE are R22 devices on Symmetrix 82, as indicated by the “2” in the MODES column in the output of the second symrdf list command.



These are the key points covered in this module. Please take a moment to review them.



This module focuses on SRDF technologies that support SRDF/Star.



This lesson covers SRDF/S and SRDF/A basics. The concept of dependent write consistency will also be discussed.



Synchronous SRDF mode is primarily used in campus environments. In this mode, Symmetrix maintains a real-time mirror image of the data from remotely mirrored volumes.

Data on the source (R1) volumes and target (R2) volumes are always fully synchronized. Data movement is at the block level.

The sequence of operations starts when an I/O write is received from the host/server into the source cache. Next, the I/O is transmitted to the target cache, then a receipt acknowledgment is provided by the target back to the cache of the source. Lastly, an ending status is presented to the host/server.

Synchronous mode is one of three modes in which SRDF can operate. The other modes are Asynchronous and Adaptive copy. Unlike competitive products, SRDF can be dynamically switched to operate in another mode without interrupting host I/O.

Like all synchronous replication solutions, synchronous SRDF has architectural limitations that must be understood:

• The maximum distance over which Synchronous SRDF can be used is limited by application timeouts and speed-of-light issues.

• Link bandwidth must be sized for peak workload at all times.



SRDF/A’s architecture delivers replication over extended distances with no performance impact.

SRDF/A uses Delta Sets to maintain a group of writes over a short period of time. Delta Sets are discrete buckets of data that reside in different sections of the Symmetrix cache. Starting at 1, each Delta Set is assigned a numerical value that is one more than the preceding one.

There are four types of Delta Sets to manage the data flow process.

The Capture Delta Set in the source Symmetrix (numbered N in this example) captures (in cache) all incoming writes to the source volumes in the SRDF/A group.

The Transmit Delta Set in the source Symmetrix (numbered N-1 in this example) contains data from the immediately preceding Delta Set. This data is being transferred to the remote Symmetrix.

The Receive Delta Set in the target system is in the process of receiving data from the transmit Delta Set N-1.

The target Symmetrix contains an older Delta Set, numbered N-2, called the Apply Delta Set. Data from the Apply Delta Set is being assigned to the appropriate cache slots ready for de-staging to disk. The data in the Apply Delta Set is guaranteed to be consistent and restartable should there be a failure of the source Symmetrix.



The Symmetrix performs a cycle switch once data in the N-1 set is completely received, data in the N-2 set is completely applied, and the 15 second minimum cycle time elapsed. During the cycle switch, a new delta set (N+1) becomes the capture set, N is promoted to the transmit/receive set, and N-1 becomes the apply Delta Set.

The minimum cycle timer can be set on a per RDF group basis by the user through Symmetrix Configuration component. Shorter minimum cycle timer settings increase bandwidth requirements.

The time that the transmit data needs to go across depends on the volume of writes and the link bandwidth. Overloading of the link can cause SRDF/A cycles to be extended beyond the minimum cycle time.

The speed of the Apply cycle depends on the volume of data that needs to be applied. Slower disks and busy volumes can slow down the Apply cycle and cause a delay in cycle switching. This is why, from a performance viewpoint, it is always a good idea to use balanced Symmetrix configurations for SRDF/A. This means that the infrastructure of the target Symmetrix should be at least as fast as the infrastructure of the source Symmetrix, so as not to create a bottleneck during the Apply cycle.



Database application consistency forms the backbone of SRDF/A design. Inherently, all database applications are consistent, which means that a database application does not issue a dependent write unless a predecessor write is completed.

For example, a DBMS does not issue a dependent data write unless a predecessor write to the log was successfully completed. EMC’s consistency technology honors this dependent write logic. By honoring write ordering at the time of the Delta Set switch, SRDF/A guarantees dependent write consistency.



This lesson describes the role of the SRDF daemon in managing synchronous SRDF consistency.



Storrdfd (pronounced “store” R-D-F-D) is a process that runs as a daemon on Unix systems and as a service in Windows. It is referred to as the SRDF daemon and uses the base daemon for all its communications with the Symmetrix, such as the issuing of syscalls.

In an SRDF/S environment, the RDF daemon cooperates with RDF-ECA to maintain consistency for composite groups.

If GNS is enabled on the host, the SRDF daemon interacts with the GNS daemon to acquire composite group definitions. Otherwise, it gets definitions from the SYMAPI database.

In an SRDF/A environment, the SRDF daemon is responsible for cycle switching when Multi-Session Consistency is enabled.

The RDF daemon is designed for full cooperation with other RDF daemons. Any task for which the daemon is responsible, such as an MSC cycle switch, can be initiated by one RDF daemon and completed by another RDF daemon. At no time is there a single point of failure if there are two or more RDF daemons monitoring the same processes.

It is therefore advisable to have more than one host running the SRDF daemon in an environment where the daemon’s services are necessary. Such a configuration provides redundancy in case one of the daemons stops unexpectedly.



Enginuity Consistency Assist is a feature that stalls write I/O to a user-provided list of devices prior to a consistent TimeFinder split or a consistent activation of Open Replicator, TimeFinder Clone, or TimeFinder Snap. Reads are allowed to continue during this time. Once the split or activation is complete, I/O is allowed to flow again. The stalling of write I/Os guarantee that the copy of data being split or activated is dependent write consistent.



ECA was enhanced to support SRDF and is referred to as RDF-ECA. It interacts with the RDF daemon to manage consistency of a user-defined RDF consistency group. RDF ECA can manage consistency for CKD and FBA devices.



In the context of TimeFinder, ECA Window is the name given to a 30 second timer that starts when a consistent split or consistent activation is initiated.

In the context of RDF-ECA, the 30 second timer is started by the Symmetrix after it determines that a write I/O to a device in a consistency group cannot complete on the remote array.

Once the ECA timer starts, Enginuity does not accept write I/Os to the affected devices. Instead, it asks the host HBAs to retry the I/O. When the required action completes, the ECA window is closed and I/O is permitted to flow again.

If, for some unexpected reason, the required action does not complete before 30 seconds are up, Enginuity closes the ECA window. It allows I/O to flow again while recording an error message in the host-based log file.



When synchronous RDF consistency is enabled for a consistency group, the RDF daemon polls the Symmetrix every second to monitor the health of the con-group.

Assume that the links connecting one of the Symmetrix pairs fail. When the source Symmetrix fails to complete writes to the remote devices, it starts the ECA timer window. All subsequent writes to the devices belonging to the composite group in that Symmetrix are turned back with retry requests issued to the host HBA. During this time, no dependent writes are issued by the application, because the host database application has not been notified of the completion of the predecessor write.



When the RDF daemon recognizes that one of the Symmetrix pairs has lost connectivity, it requests the remaining Symmetrix arrays to open an ECA window, which will hold incoming writes as well.

Once all ECA windows are open and write I/O is stopped to the entire consistency group, the daemon logically suspends the remaining communication links.



Once all the links have been suspended, the ECA windows are closed and the writes to the local arrays are allowed to complete. Note that writes to the first group of devices were held as soon as the links failed and the remote writes did not complete. Thus, if the host was running a database application, no dependent write could have been issued by the host application between the time that links on the first Symmetrix failed and I/O flow was restored by the RDF daemon to all devices. This makes the target site data consistent.



This lesson covers SRDF/A MSC (multi-session consistency) cycle switching and MSC cleanup.



A second function of the RDF Daemon is to maintain Multi-Session Consistency (MSC) in an SRDF/A environment. MSC is important when consistency must be maintained between multiple production applications running on multiple SRDF groups. The example on this slide illustrates how the RDF daemon maintains consistency while cycle switching during normal MSC operations.

The RDF Daemon (or daemons) monitors all groups and manages cycle switching for all R1 Symmetrix arrays whose sessions are managed by MSC.

When the minimum cycle time, which by default is set to 15 seconds, has elapsed:

• The RDF Daemon verifies that each R1 Symmetrix array has completed transferring the Transmit Delta Set to the R2s, and

• Each R2 Symmetrix has completed applying the apply delta sets.

Until these conditions are satisfied for each RDF group in each Symmetrix array, the cycle switching is not initiated and the present cycle gets elongated.

Once all RDF groups indicate their readiness to switch, the RDF daemon briefly holds writes to the source arrays and switches the cycles, first on the source, and then on the target arrays. The cycle switching is an asynchronous process, so all the source and target boxes do not switch in the same instant, they switch one after the other. Host writes are allowed to flow into the source array as soon as the source array has switched, whereas transmit data is allowed to flow into the target array as soon as the target array has switched.



The third and final function of the RDF daemon is to manage SRDF/A multi-session consistency in the event of a failure of communication between the source and target.

When there are multiple Symmetrix arrays or SRDF groups participating in a multi-session consistency group, the Symmetrix sets the “MSC cleanup required” flag if the receive Delta Set was completely received at the time the failure occurred.

A single Symmetrix, with the SRDF/A MSC flag set, cannot determine the correct action to take for a completely received Delta Set without information from other Symmetrix arrays in the SRDF/A MSC protected consistency group.

MSC Cleanup can be invoked by any of the following methods:

• The RDF daemon performs MSC cleanup automatically if it can communicate with the target arrays.

• The API/CLI automatically performs MSC cleanup during the processing of any RDF control command.

• User can manually execute MSC cleanup through CLI.

The MSC Cleanup Needed status is exported to user-visible displays, such as query output. MSC Cleanup commits receive cycle data in case of failure during cycle switch instead of discarding it unnecessarily.



Assume that the links between the source and target arrays have tripped. The Receive Delta set in the top array is incomplete, while the Receive delta set in the bottom array is complete.

In the top case, because the Receive Delta set is incomplete, the only valid choice for the Symmetrix is to discard it, because the dependent write principle only works for complete Delta sets.

For the bottom case, the Receive Delta set is complete. Since this is an MSC protected group, the Symmetrix cannot decide what to do on its own.

• If all Receive Delta sets were complete, it would be correct to Apply the data.

• However, if any of the Receive Delta sets are incomplete, then the data must be discarded.

• The Symmetrix sets the MSC Cleanup Needed flag.

In the example displayed on this slide, MSC cleanup is undertaken by one of the three methods mentioned earlier:

1. The RDF daemon

2. Any RDF control command issued by the API/CLI

3. The user issues an explicit “symstar cleanup” command



Once the RDF daemon on the source side notices a trip event, it runs the MSC cleanup logic on the target arrays if it can communicate with them. The legend A=N means the Apply Delta set is numbered N. Similarly, R=N+1 means that the number of the Receive Delta set is N+1. Though the table shown here uses two Symmetrix units, the logic works for larger numbers of arrays.

1. In this case, none of the SRDF/A sessions have the “MSC Cleanup Needed” flag set. This occurs when all the Receive Delta sets were incomplete, and all were automatically discarded. There is no Cleanup action to take and it is not invoked automatically.

2. Only some Symmetrix arrays have the “MSC Cleanup Needed” flag raised. Also, ALL Apply delta set numbers are the same. This means that some Symmetrixes had to discard their incomplete Receive Delta sets. Consequently, all the Symmetrixes needing MSC Cleanup must discard their completely received Delta Sets.

3. All Symmetrixes have the “MSC Cleanup Needed” flag raised. In this case, ALL Apply Delta set numbers must be the same. This indicates that all Receive Delta sets are complete and all the Receive Delta sets can be applied.

4. Only some Symmetrix units have their flag raised. Also, one or more Symmetrixes with the flag raised have a Receive Delta set number that matches the Apply cycle number for a Symmetrix that discarded its incomplete Receive cycle. This indicates a failure in the middle of a cycle switch. So, all the completely received Receive Delta sets in the Symmetrix arrays with the flag raised are applied.



This lesson covers the SDDF and SRDF features that were developed to support SRDF/Star.



Each Symmetrix logical volume is allotted a quota of 16 SDDF sessions. These sessions allow the Symmetrix to track changes using bitmaps, which flip from a zero to a one whenever a monitored track changes.

SDDF sessions are used to monitor changes in BCVs, Clones, Snaps, Change Tracker, and Open Replicator.

SDDF functionality was enhanced for SRDF/Star to enable differential resynchronization between two target sites. Once Star is enabled, two sessions are created and activated at the Synchronous target site, and one SDDF session is created at the Asynchronous target site.



When Star Protection is enabled, two SDDF sessions are created at site B, and one SDDF bitmap is created at site C. The bitmaps at site B are always active during normal Star operation. They are alternately marked and cleared after every two or more SRDF/A MSC cycles elapse between sites A and C.

The bitmap at site C stays inactive during normal Star operation.



If the primary site fails, data transmission to both sites stop simultaneously.

Under these circumstances, the data at Synchronous Target B is more recent than the data at Asynchronous Target C.

In the course of recovery, an inclusive OR of the two bitmaps is performed at site B. This operation marks all tracks updated in the current bitmap and all tracks updated in the previous bitmap, as owed by site C. Since the bitmap initialization at site B occurs every two plus cycles, it is possible that the inclusive OR will result in more than the minimum required tracks being marked as invalid. This is not a problem since by copying a few more tracks than needed, we err on the side of caution.

MSC cleanup should be run at site C, if needed.

If a business decision is made to run production at site B, the RDF devices at site B are turned into R1 volumes and paired with corresponding R2 volumes at site C. An RDF establish now copies the invalid tracks from site B to site C.

If the decision is to run at site C, the devices at B are turned into R2s, and those at site C into R1s. An RDF restore updates the C site with tracks owed by B to C.



The failure described here is often referred to as a rolling disaster, where the first failure is succeeded by a second one. Here, the first fault disrupts the links between A and B. This causes the synchronous consistency group to trip, leaving the data at site B consistent. The SDDF sessions at site B are frozen for later conversion to invalid tracks. Data processing continues at site A, and Site C continues to get updated.

When the synchronous link fails, the SDDF session at site C is activated on a cycle boundary just prior to the next cycle switch. This SDDF session records new writes coming into site C. Additionally, a token counter is started at C. It starts counting the number of cycle switches after activation.

Shortly after the first failure, the primary site fails, causing data transmission to stop at site C. If the second failure occurs more than two SRDF/A cycle switches after the first failure (as recorded by the token counter), site C will be more current than site B.

A Star query after the final primary site failure indicates which side is more current.

An inclusive OR between the two SDDF bitmaps at site B and an inclusive OR between the resulting bitmap and the bitmap at site C, creates the invalid track table that must be resolved when the two sides are synchronized.

If data at site C is more current, the synchronization should cause tracks to flow from C to B. If the token counter indicates that B is more current than C, new data flows from B to C.



When Star Protection is enabled, two SDDF sessions are created at site A and one SDDF bitmap is created at site C. The bitmaps at site A are always active during normal Star operation. They are alternately marked and cleared after every two or more SRDF/A MSC cycles elapse between sites A and C.

The bitmap at site C stays inactive during normal Star operation.



The failure of site B in Cascaded Star is a major failure, since reconfiguration from cascaded to concurrent Star must be undertaken in order to provide remote data protection. When the link between A and C is activated, the SDDF bitmaps at site A are used to determine the invalid tracks that must be moved from A to C.



If the workload site fails in a cascaded star environment and the decision is made to switch production to either target site, the SDDF sessions are not needed because the differences between the B and C sites are recorded in the track tables.



A half delete operation can be executed on a dynamic RDF pair using SYMCLI commands. After the half delete command is executed, the device in the Symmetrix on the left turns into a regular device, and the one on the right retains its identity. A SYMCLI query shows it as a half pair. The SRDF pair state must be suspended, failed over, split, or partitioned before a half delete can be performed.

The half delete of SRDF pairs is used by SRDF/Star in a disaster situation.

The command is also available for general use, but only in special cases. If an existing RDF relationship is rendered null and void by the physical removal of one of the Symmetrix arrays, without the termination of the SRDF relationships, the half delete command can be used to dissolve remaining RDF volumes.

Do not use the half delete command when both arrays in an RDF relationship have visibility to each other.



The half swap operation changes the personality of one SRDF volume, regardless of whether the other RDF volume is visible or not.

There are two uses for the half swap command while reconfiguring devices during a Star action:

1. Sometimes during a site reconfiguration, an R2 device is half swapped so it becomes an R1 device. This makes the pair relationship “duplicate” since there are now two R1 devices in the pair pointing at each other.

2. At other times in the course of a site reconfiguration, a half swap converts a duplicate device pair into a normal device pair by turning one member of the duplicate pair from an R1 to an R2.



The two functions described on this slide were created for the purpose of Star and are not available to users.

Creation of a dynamic RDF pair without copying data is an action that risks data corruption if it was not 100% certain that the devices in the pair did, in fact, contain identical data. This function is used in the case of a planned workload site switch when applications are halted and all three sites are made equal prior to a switch.

Creation of a dynamic RDF pair with an incremental refresh is only possible based on the SDDF bitmaps at the synchronous and asynchronous target sites. This is the key behind SRDF/Star’s ability to switch workload sites without a full refresh.






This module describes Star configurations and their operations using concurrent and cascaded SRDF.



This lesson gives an overview of concurrent and cascaded SRDF/Star configurations. It also describes the impact of different failures on each kind of Star configuration.



EMC SRDF/Star is a three-site disaster-restart solution that can enable resumption of SRDF/A with no data loss between two remaining sites, providing continued remote-data replication and preserving disaster-restart capabilities. It offers a combination of continuous protection, changed-data resynchronization, and enterprise consistency between two remaining sites in the event of the Workload Site going offline due to a site failure.

As more businesses require solutions to provide the highest levels of disaster restart capabilities, SRDF/Star is the industry’s first solution to enable organizations to satisfy those requirements.



In the year 2001, EMC built a special version of a multi-hop Mainframe SRDF/AR for a New York based financial services company. This version of Mainframe SRDF/AR maintained a differential relationship between the source (site A) and remote site (site C). If the bunker site (site B) failed, a full resynchronization between A and C was no longer necessary. By virtue of using DeltaMark (SDDF) sessions, an incremental relationship was maintained between sites A and C.

In 2003, SRDF/A was released and concurrent SRDF/A and SRDF/S became possible. So, when a large European company (in the same line of business as the American company), paid a visit to their friends in New York, they got the idea for a product with the functionality of Star. Late in 2003, the Europeans came to Hopkinton and had a meeting with Engineering where they outlined their requirements.

EMC decided to implement a product, as desired by the European customers, and call it STAR, which was supposed to be an acronym for Symmetrix Triangulated Automated Replication. It took 2 years from the first conversation with the customer, and 18 months of development to produce a version of Star on the Mainframe in 2004. The Open Systems version was released in 2005. To conform to EMC’s naming architecture for the SRDF products, the name SRDF/Star was chosen.

In 2008 when Cascaded SRDF was released, Star functionality was enhanced to support this feature. In 2009 when SRDF/Extended Distance Protection was released, Star was enhanced to support SRDF/EDP.



Concurrent SRDF/Star enables concurrent SRDF/S and SRDF/A operations from the same source volumes.

The primary business benefit of Star is that in the event of a workload site outage, it is possible to undertake a differential resynchronization between the two remaining sites, followed by the resumption of production at either site.



Cascaded SRDF/Star was introduced in 2008 with the release of Enginuity 5773. Cascaded RDF allows a synchronous R2 target to also act as a source for SRDF/A. The long distance site in cascaded RDF uses this source to receive its data feed. In the event of a failure of the workload site, the synchronous target has up to date data. The asynchronous target data is not more than two SRDF/A cycles behind the source site data.



The events of September 11, 2001 made businesses more aware of the critical need to recover their data after a disaster. A few years ago, Share, a major bank from New York, did a presentation entitled “The Effects of 9/11.” After the attacks on September 11, this bank failed their data processing over to their New Jersey site. Later that week, they were asked by federal regulators, “How are you protected now?”

As the importance of information continues to increase, companies are increasingly interested in protecting their data and minimizing their down time after a failure. SRDF/Star offers customers the business benefits that are of high priority for institutions with mission critical data.

Both cascaded and concurrent Star have their uses, depending on the application environment. If the loss of the primary data center is the principal concern, cascaded Star is a good choice.

If there is a risk of losing the synchronous target as well as the workload site, concurrent Star would be a better choice.



Primarily, EDP is available for cascaded Star mode. Concurrent Star with diskless R21s has limited functionality. Star, with Extended Distance Protection, can be built either with R2 devices or R22 devices at the Asynchronous target site.



It is possible to configure more than one Star triangle per Symmetrix. It is also possible to include more than one Symmetrix in a Star triangle.

Each application, or group of applications, that would fail over together should be in one Star triangle. Applications or application groups designed to fail over separately, should be in their own Star triangles.

A maximum of 250 SRDF groups are allowed in VMAX 40K and VMAX 20K/VMAX systems, compared to 32 SRDF groups on VMAX 10K/VMAXe systems.

A maximum of 64 SRDF groups per director are allowed in VMAX 40 and VMAX 2K/VMAX systems, compared to 32 SRDF groups per director on VMAX 10K/VMAXe systems. Though the theoretical maximum number of SRDF groups that an RDF director can support is a high number, an RDF director should only be assigned a maximum of 4 to 6 SRDF groups if the workload is heavy.



Prior to the release of Solutions Enabler 7.0, there was no support for R22 devices. All Star configurations were set up with an R2 device at the asynchronous target. The source site had a concurrent source, or R11 device if Star was running in concurrent mode. Otherwise, the synchronous target had a cascaded R21 device if Star was running in cascaded mode.



R22 devices were introduced in Enginuity 5874 and Solutions Enabler v7.0. Certain versions of Enginuity 5773 also support R22 devices. These devices can maintain a target relationship with two R1 devices, though not at the same time. The big advantage of using R22 devices is during Star reconfigurations after a Workload site failure. R22 devices make reconfigurations more resilient in case there is a link failure while the reconfiguration is in progress.

Along with R22 devices, newer Star configurations employ R11s at the source site and R21s at the synchronous target. In a concurrent Star setup, the R21 to R22 link between B and C is passive. In a cascaded Star configuration, the R11 to R22 link between A and C is passive.



There are 6 possible fault conditions that can arise in a Star setup. These comprise the failure of the three sites and failure of the three links. Depending on whether Star was operating in Concurrent or Cascaded mode, the response to the failures will be different.



This is a list of possible actions after a failure occurs in a concurrent Star setup: 1. If the link between A and B fails, it is still possible to run production at A with remote

protection available at site C. 2. The same holds true if site B fails. 3. If the link between A and C fails, there are two possibilities. The first is to continue running

production at A with remote protection at B. The second is to reconfigure concurrent Star to cascaded Star and run in Star protected mode.

4. If site C fails, the only option is to continue running at site A with remote protection at B. 5. If the link between B and C fails, there is no effect on Star operations because the standby

links between B and C are not used unless there is a failure of site A. 6. If site A fails, production has to be switched to site B or site C. This necessitates a

reconfiguration of the RDF devices. The devices at the site to which production was switched become R1 devices, and the remaining site is reconfigured to become R2 targets to the new production site.

The choice of which location to fail over to depends on customer needs and the location of customer resources.



This is a list of possible actions after a failure occurs in a cascaded Star setup: 1. If the link between A and B fails, it is still possible to run production at A with remote

protection available at site C, but only after a reconfiguration. 2. The same holds true if site B fails. 3. If the link between A and C fails, there is no effect on Star operations because the standby

links between A and C are not used unless there is a failure of site A. 4. If site C fails, the only option is to continue running at site A with remote protection at B. 5. If the link between B and C fails, there are two possibilities. The first is to continue running

production at A with remote protection at B. The second is to reconfigure cascaded Star to concurrent Star and run in Star protected mode.

6. If site A fails, production has to be switched to site B or site C. This necessitates a reconfiguration of the RDF devices. The devices at the site to which production was switched become R1 devices, and the remaining site is reconfigured to become R2 targets to the new production site.

The choice of which location to fail over to depends on customer needs and the location of customer resources. As is obvious from the digram shown on this slide, if you ignore case 5 where reconfiguration is optional, 3 of the 6 failure scenarios require an RDF reconfiguration in Cascaded Star if you want to retain remote data protection after the failure.



This lesson describes SRDF/Star operations using concurrent SRDF. It covers failure conditions and recovery from the failure conditions.



The symstar command in Solutions Enabler is responsible for managing Star. It issues the SRDF commands needed to perform all Star actions.

The symstar setup action marks the RDF groups to which the devices in the CG belong as groups intended for use by Star commands. Similarly, the devices in the CG are identified as devices participating in a Star configuration.

A Star configuration can be in several states, as shown in this slide. Each state is clearly defined, and transitioning from one to the other is achieved by the use of the actions shown.

The Disconnected, Connected, and Protected states refer to the relationship between the Workload site, connecting path, and either target site. This is why there are two rectangles in the Disconnected, Connected, and Protected states. The Star Protected state refers to all three sites as a complete entity.

A Disconnected state between the Workload site and either remote site, indicates there is no data flow between the sites. The RDF relationship may not be properly defined, e.g. after a Workload site failure and a workload switch. The remote site does not contain a copy of the production data, hence, there is no remote data protection. The connect action has to be issued to transition from the Disconnected to the Connected state.

A Connected state between the Workload site and either target, indicates there is data flow between the sites. The RDF relationship is properly defined. The target site in question is not necessarily synchronized with the Workload site. There is no consistency protection for the remote data. The protect action transitions the Workload site and remote site in question from the Connected to the Protected state.

In the Protected state there is data flow between the Workload site and remote site. The RDF relationship is properly defined. Dependent write consistency of the data at the synchronous target site is assured through RDF-ECA. Dependent write consistency at the asynchronous target is assured through MSC. The enable action transitions all three sites into the Star Protected state.

In the Star Protected state, there is data flow and consistency protection at each individual target site. Additionally, the differential relationship between the Synchronous and Asynchronous target sites is defined. If the Workload site were to fail, a differential resynchronization between the two surviving sites would be possible.



The Star options file is created by the user with a text editor. It specifies parameters shown on the next page. The setup command translates the contents of the options file and writes them into the Star internal definitions file. This file is used by the symstar command for all its actions.

The internal definitions file is created as a result of executing the symstar setup action. It should not be modified by users. Any changes should be instituted through the options file.



A detailed description of all the parameters in the options file is out of scope for this course and can be found in the Solutions Enabler Symmetrix SRDF/Star CLI manual. The important ones are the three site names, where site specific names such as Singapore or London can be chosen. Here, the names sun1starA, sun1starB, and sun1starC have been chosen because they made sense in the lab where the screen captures were taken.

The adaptive copy tracks value is an invalid track threshold value. If a symstar protect command is issued for the synchronous target, there will be a wait until the number of invalid tracks are below this number before the SRDF mode of the target being protected is switched from adaptive copy to synchronous. If a symstar protect command is issued for the asynchronous target, there will be a wait until the number of invalid tracks are below this number before the SRDF mode of the target being protected is switched from adaptive copy to asynchronous.

The SYMCLI_STAR_ACTION_TIMEOUT value specifies, in seconds, the length of time that a symstar protect or a symstar enable command will wait to complete before timing out.

The SYMCLI_STAR_COMPATIBILITY_MODE = V70 specifies that features introduced in Version 7 of Solutions Enabler, such as state tracking and Star with EDP, are permitted. Note that with Solutions Enabler V7.6 the SYMCLI_STAR_COMPATIBILITY_MODE setting is no longer supported.

The SYMCLI_STAR_AUTO_DISTRIBUTE_INTERNAL_FILE setting has a default value of Yes, and will automatically distribute the internal definition file to the remote arrays.

The setup command, shown here, sets up a concurrent Star configuration using the composite group sun1star.



After symstar setup is run, the symstar query output identifies the site from which the query was executed. It identifies the Workload site as sun1starA. The first target, or the Synchronous site is sun1starB. The second target, or the asynchronous site is sun1starC. Both targets are in the Disconnected state, indicating that the SRDF links are in the suspended state. The Mode of operation is concurrent because the –opmode concurrent option was indicated during the setup step.



The connect action starts the data flow between the source and the two target sites in adaptive copy mode. Each target site must be connected individually. The protect action switches the mode to Synchronous or Asynchronous depending on the target site after the invalid tracks are below the threshold value specified in SYMCLI_STAR_ADAPTIVE_COPY_TRACKS. Finally, the enable action enables Star protection.



A transient target site fault in a Star configuration is defined by the loss of a site other than the Workload site. The target site could become unavailable because of a loss of network communications, or a problem at the site itself, such as a power failure. It does not affect production at the Workload site.

After a transient fault, the site state changes from Synchronized or Consistent to Pathfail or PathFail;CleanReq. At this point there is no data flow between the Workload and the failed target site. The data at the target site is consistent, since consistency protection was in force.



After the connection between the Workload and target sites are reinstated, the reset action performs the necessary cleanup at the target site. Since the failure occurred on the Synchronous Target, MSC cleanup was not necessary. reset disables consistency group protection and sets the RDF mode to adaptive copy, but does not resume the RDF links. It also transitions the target site in question to the Disconnected state.

If the target site has BCVs, this is the time to take a gold copy of the data at the target site before resynchronization begins and data consistency at the target is destroyed.

Finally, the connect – protect – enable sequence reinstates Star protection.



When the Workload site disaster warrants a move of the production workload to the Synchronous or Asynchronous target site, an Unplanned Switch Operation becomes necessary. After the disaster, the system transitions from the Star Protected to Star Tripped state. The Synchronous target transitions to the PathFail state, and the Asynchronous Target to the PathFail or PathFail;CleanReq state. Recovery operations must be undertaken to start production at one of the target sites.

The distinction between PathFail and PathFail;CleanReq states is the need for MSC cleanup at the Asynchronous Target site.

If it is decided to switch to a remote site and preserve data at that site, the switch command transitions the sites to the Disconnected state. From that state it is necessary to issue a connect command to arrive at the Connected state.

If the decision is made to switch to a remote site and preserve the data of the other remote site, then the switch command transitions the remaining sites to the Connected state.



The decision about which site to switch to depends on the customer’s infrastructure capabilities and the nature of the disaster that may have effected the campus Synchronous Target Site.

In the case of a rolling disaster, the Asynchronous Target site can be more up-to-date than the Synchronous target. This can happen if the links to the Synchronous Target Site fail first, and the Asynchronous target continues to receive data for awhile, then, the Workload site fails completely. The “symstar query” command can assist in making the decision about which data is the most recent and must be preserved.

In the example that follows, Workload is switched to the Synchronous Target Site while keeping the data of the Synchronous Target Site.



In the example shown here, the Workload site has failed. Assume that the query reveals that the data at the Synchronous target is more recent.

The system state is StarTripped.

The Synchronous Target Site transitions to the PathFail state.

The Asynchronous Target Site transitions to the PathFail; CleanReq state if the failure occurred in the middle of a Delta Set switch. Otherwise, it transitions to the PathFail state.



The cleanup step is necessary only if the state of the Asynchronous Target Site was PathFail; CleanReq after the failure of the Workload site.

The cleanup command can be issued from either remaining site. This performs MSC Cleanup at the Asynchronous Target Site.

After cleanup is performed, a BCV copy of the data should be taken to preserve a consistent data copy from the point of time of the failure.



In the example shown here, the Workload site is being moved from sun1starA to sun1starB, while retaining data in sun1starB. Note that this represents the “Keep Local Data” option on the Unplanned Switch state flow diagram shown earlier.

The switch command reconfigures the RDF devices at sun1starB and turns them into R11 devices. The RDF pair relationship between B and A is now duplicate.



Before issuing the connect command, BCV golden copies should be taken at site C, whose data consistency will be destroyed during the differential synchronization between sites B and C. The connect action activates the link between sites B and C. Since we are using R22s, the work to connect Site C is significantly less than in 5773 and earlier. At the end of the connect step, sites B and C have an active relationship such that data can flow in adaptive copy mode.

The protect action enables MSC protection between sun1starB and sun1starC. Star protection is not possible because three sites are not available.

An excerpt from the symstar query output shows that the second target site is in the Protected state.



Continuing with the example shown earlier, let us assume that the problem that caused the Workload site at sun1starA to be shut down has been resolved. When the connect command is used at sun1starB, the RDF volumes in sun1starA are reconfigured so that they become R21devices. An adaptive copy synchronization is initiated between sun1starB and sun1starA.



A Star query shows that the Workload is running at sun1starB. The async target site sun1starC is protected while the new sync target site, sun1starA, is connected.



This example shows the steps to switch the Workload site back to the original site sun1starA in a planned fashion. The key command here is halt. A planned workload switch is typically used either to move back home after the resolution of a Workload site failure, or in the course of a disaster drill. All site moves are allowed as long as the sites are functional and the RDF connectivity is present.



The “halt” command ensures that all three sites are identical and write disables the R1 devices if they are mapped to an FA. The query shows that the halt was successful. The Workload site still remains at sun1starB.



The switch command now resets the RDF relationships so that sun1StarA devices have the RDF11 attribute and has a concurrent relationship with sun1starB and sun1starC. Both targets transition to the Disconnected state. Now, the connect – protect - enable action sequence transitions the system to the Star protected state.



The query command shows the result of the switch action. The workload is back to sun1starA and the two target sites are in their original state. The sites are in the disconnected state.



The switch command leaves the synchronous and asynchronous target in a disconnected state, as seen in the previous slide. The connect, protect, and enable actions can be used to reinstate Star protection again.



This lesson discusses operational considerations for Star in Cascaded and EDP configurations. SRDF and Star documentation is also discussed.



There are a few important differences between the normal operating conditions of Concurrent Star and Cascaded Star:

1. Since the consistency of the asynchronous site data is dependent on the consistency of the synchronous site data, the asynchronous target can only be protected if the synchronous target is protected as well. Consequently, after the two sites have been connected, the synchronous target must be protected first.

2. While both the synchronous and asynchronous targets are in the protected state, an unprotect action on the synchronous site will not be permitted.

3. If the synchronous target is disconnected while the asynchronous target is protected (as can happen after a failure of the links between the workload site and the synchronous target), a connect action will not be permitted on the synchronous target. The asynchronous target must be tripped or unprotected before the connect with the synchronous target is allowed.

4. Since only the asynchronous site can be taken out of service without disrupting remote data protection, it is only permissible to isolate the asynchronous target from the Protected, Protected state.



The SRDF/Star manual has a detailed table that describes the actions that can be performed on a EDP Star configuration. Some of the significant restrictions are provided here.



The recommended documentation for SRDF and SRDF/Star is listed. All documentation can be downloaded from the EMC Support site (support.emc.com).






These are the key points covered in this course. Please take a moment to review them.


Documents

Welcome to Symmetrix VMAX SRDF/Star and Cascaded Star ... · 7/1/2016 · Use, copying, and distribution of any EMC software described in this publication requires an applicable