Docu36066 Documentum XPlore 1.2 High Availability and Disaster Recovery Guide

EMC DocumentumxPloreVersion 1.2

High Availability and Disaster Recovery Guide

EMC Corporation

Corporate Headquarters

Hopkinton, MA 01748-9103

1-508-435-1000

www.EMC.com

Legal Notice

Copyright 2011-2012 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to changewithout notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATIONMAKES NO REPRESENTATIONSOR WARRANTIES OF ANY KINDWITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLYDISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. Adobe and Adobe PDFLibrary are trademarks or registered trademarks of Adobe Systems Inc. in the U.S. and other countries. All other trademarksused herein are the property of their respective owners.

Documentation Feedback

Your opinion matters. We want to hear from you regarding our product documentation. If you have feedback about how we canmake our documentation better or easier to use, please send us your feedback directly at [email protected]

Table of Contents

Preface ................................................................................................................................. 7

Chapter 1 Overview ...................................................................................................... 9Introduction ...................................................................................................... 9Backup and restore process ............................................................................... 10

Backup levels and types ................................................................................ 10Backup process............................................................................................. 11

Recovery Point Objective (RPO)......................................................................... 11Restore Time Objective (RTO) ........................................................................... 12Balancing RPO and RTO ................................................................................... 12Backup risk ...................................................................................................... 13Multi-site disaster recovery ............................................................................... 13High availability and disaster recovery cost........................................................ 14

Chapter 2 Backup and Disaster Recovery ................................................................... 15Introduction ..................................................................................................... 15Content Server recovery overview ..................................................................... 15xPlore backup configurations ............................................................................ 19Restore Content Server...................................................................................... 20

Synchronize the restored data ....................................................................... 21Restore if content is newer......................................................................... 21Restoring if metadata is newer................................................................... 22

Restore an index at a particular point in time...................................................... 23Restore data after an instance fails ..................................................................... 25

Chapter 3 xPlore High Availability Deployments ......................................................... 27Overview ......................................................................................................... 27Content Server full-text objects and initialization files ......................................... 28Set up Content Server and xPlore HA ................................................................ 30

Spare instance HA (standby) ......................................................................... 30Dual primary instance HA (active-active) ....................................................... 32

Procedure ................................................................................................ 34Enable full-text queries to be serviced by the secondary xPloredeployment.............................................................................................. 36

xPlore dual mode with an existing FAST HA deployment ............................... 37Setting up xPlore for HA after dual mode has been configured .................... 38

xPlore HA dual mode with an existing FAST HA deployment ......................... 40

3

Table of Contents

List of Figures

Figure 1. Backup for Object Consistency .............................................................................. 16Figure 2. RPO for disaster recovery site (including data replication from the

primary site) .................................................................................................... 18Figure 3. Storage Area Backup is Newer .............................................................................. 22Figure 4. Database Backup is Newer .................................................................................... 22Figure 5. Restoring xPlore Instances Decision Tree................................................................ 24Figure 6. Full-text Index Objects .......................................................................................... 29Figure 7. xPlore Spare Instance HA...................................................................................... 32Figure 8. Two xPlore Primary Instances HA ......................................................................... 33

4

Table of Contents

List of Tables

Table 1. Backup scenarios .................................................................................................. 20Table 2. HA Strategy Comparison ...................................................................................... 28

5

Table of Contents

6

Preface

This guide describes xPlore high availability and disaster recovery concepts in the context of businesscontinuity and provides concrete examples.

Intended Audience

This guide is intended for administrators who are responsible for planning xPlore high availabilitydeployments.

Revision History

The following changes have been made to this document.

Revision Date Description

November 2011 Publication for version 1.2.

December 2012 Added section Restore data after an instance fails, page 25

7

Preface

8

Chapter 1

Overview

These topics are described:

Introduction, page 9

Backup and restore process, page 10

Recovery Point Objective (RPO), page 11

Restore Time Objective (RTO), page 12

Balancing RPO and RTO, page 12

Backup risk, page 13

Multi-site disaster recovery, page 13

High availability and disaster recovery cost, page 14

Introduction

Business continuity is the umbrella term that covers all efforts to keep critical data and applicationsrunning despite any type of interruption. Interruptions can be both planned and unplanned. Plannedinterruptions include regular maintenance or upgrades. Unplanned interruptions could includehardware or software failures, data corruption, natural or man-made disasters, viruses, or humanerror.

Enterprises can use the following solutions to meet business continuity requirements:

Backup and restore process, page 10

Backup and recovery is an essential part of both basic data disaster recovery and operationalrecovery strategies. Operational recovery is the ability to recover from errors that can occur on aregular basis but are not catastrophic. Some examples of these kinds of errors are data corruptionor accidentally deleted files. In data disaster recovery, only data backups are replicated but notarget hardware/software installation exists. The source installations hardware/software must bereinstalled and the data backups are reapplied.

Chapter 3, xPlore High Availability Deployments

9

Overview

High availability provides a high level of business continuity as well as high performance throughload balancing for mission critical systems. In a high availability configuration, a secondarysystem is maintained as an exact copy to which the primary system can fail over.

Multi-site disaster recovery, page 13

Disaster recovery responds to catastrophic failures. When your installation fails, you recover it byrestoring it to a previously consistent state (that is, a particular point in time) from your backups.Restoring an installation to a particular point in time is also known as a point-in-time recovery.To restore your installation to a consistent state, use backup coordinator software (such as EMCNetworker). Using backup coordinator software ensures that all of your installations componentsand data source backups are synchronized. For more information about backup and recovery withContent Server, see the Documentum Content Server 6.5 Backup and Recovery White Paper on EDN.

Your business users and information technology department together can provide a direction for yourhigh availability and disaster recovery planning. For instance, information technology departmentcan have standardized on Microsoft Failover Clustering.

Backup and restore process


Backup levels and types, page 10

Backup process, page 11

Backup levels and types

Back up of data is basically the full-text indexes, whereas a backup of configuration information isnecessary to rebuild xPlore into the same configuration you had. You need only back up the xPlorexDB files to back up both the data and configuration.

Backup levels You can back up an xPlore federation, domain, or collection with a full,incremental, or cumulative backup:

Full: A complete copy of all of the required files.

Incremental: Only the changes after the most recent backupeither a full or incremental backup.Faster backup than cumulative; could be slower to restore than cumulative because there couldbe more files to restore. To recover your system completely, first apply the last full backup andall incremental backups in sequence up to the most recent incremental backup. Therefore, therecovery is successful only up to the last good incremental backup (because each incrementalbackup depends on the previous ones).

Cumulative: (also known as a differential) All of the changes after the most recent full backup;that is, it includes all of the incremental backups after the last full backup. Slower backup thanincremental; could be faster to restore than incremental because there could be fewer files torestore. The amount backed up is larger per differential than incremental.

Note: xPlore does not support native differential backups.

10

Overview

Backup types Backups can be cold, warm, or hot:

ColdA full backup of the entire system while the entire system is shut down.

Although the backup and restore times can be long, you do not need to perform a cold backupoften. In addition, the restore time can be accurately estimated.

WarmA backup that is performed while some functionality of the system is made unavailable.

Although the backup and restore times can be medium to long, the system can still providesome services.

HotA backup performed while the system continues to run normally and all functionality isavailable.

Backup process

Backing up a system replicates copies of the required files at a specific point in time to anotherlocation (for example, another server, datastore, or tape). Replication can be asynchronous orsynchronous. In synchronous replication, the primary system must receive a confirmation from theremote system that the data was successfully received. So, the data is always in sync, but performancecan be slow because of large data volume or the long distance between systems. In asynchronousreplication, the primary system does not wait for the remote system to confirm that the data wasreceived. So, performance is faster than synchronous replication, but the data cannot always be insync. There is also the potential for data loss, so consistency procedures must be run upon recovery.However, if the asynchronous replication technology supports consistency groups, then the data canbe assured of being in sync. Consistency groups enable you to define a group of storage areas forinterdependent applications, such as the Content Server, RDBMS, full-text indexes, to be coordinated.Furthermore, that consistency group monitors all disks that are assigned to it (as well as the I/Owrites to these disks) to ensure write-order fidelity across these disks. EMC Symmetrix Data RemoteFacility/Asynchronous (SDRF/A) supports consistency groups.

A snapshot , which is a type of point-in-time backup, can be a backup of the changes since the lastsnapshot. You can create copy-on-write or split-mirror snapshots. A copy-on-write snapshot is adifferential copy of the changes that are made every time new data is written or existing data isupdated. A copy-on-write snapshot uses less disk space than a split-mirror snapshot but requiresmore processing overhead. In a split-mirror snapshot, the mirroring process is stopped and a fullcopy of the entire volume is created.

Recovery Point Objective (RPO)

When you plan for backup and recovery, decide how much data loss you are willing to incur. Usethis decision to calculate how often you need to perform backups. Backups should be performed atfixed intervals. The length of time between backups is called the Recovery Point Objective (RPO);that is, the maximum amount of data that you are willing to lose.

After determining your recovery time and recovery point objectives, then you can determine howmuch time you actually have to perform your backups; the amount of time you have to perform yourbackup is typically called your backup window. The backup window determines the type andlevel of your backups. For example, if you have a system that requires 24-hour, 7-days-a-week,

11

Overview

365-days-a-year availability, then there is no offline backup window. So, you would have to performan online backup (also known as a hot backup) in which the system is not taken offline.

As the number of backups increase, the space required to store them will also increase. Therefore,you should consider how long you are required to retain your backups (which is also referred to as adata retention period). Plan for the appropriate amount of storage space.

Restore Time Objective (RTO)

You should decide how long youre willing to wait until the data is completely restored and businessapplications become available. The time it takes to completely restore data and for businessapplications to become available is called the Recovery Time Objective (RTO). Your RTO can bedifferent from your RPO. For example, you might need an extremely short RPO to minimize potentialdata loss, but you can have a long RTO in which you can tolerate 12 to 24 hours to recover dataand get your business applications back online.

Balancing RPO and RTO

To determine your backup and recovery, high availability, and multi-site disaster recovery plan,you first determine the degree to which your xPlore installation is mission criticalthat is, if yourorganization considers your xPlore installation to be down for a short period of time (1 hour forexample) as being a severe outage. However, your Documentum repository could be mission criticalbut xPlore full-text search functionality might not be. In general, the shorter your RTO and smalleryour RPO, the more mission critical your xPlore installation is.

In order to meet a short RTO and small RPO, a mission critical installation must have a negligiblerisk to recover from any kind of planned or unplanned interruption within the specified RTO andRPO. In general, a mission critical configuration will cost more (in terms of both cost and staffing)than a non-mission configuration because it will require more hardware, planning, and testing.Mission critical systems should not rely solely on backups: You should also consider investing inhigh availability solutions.

Sometimes a partial recovery that makes your most critical indexes available for querying or indexingnew content may be suitable for your xPlore installation. Some examples of an acceptable partialrecovery are:

To make the previous 2 days of indexed data available for querying within 1 hour but makeall of your indexes available within 24 hours.

To make querying of all indexes available within 1 hour but resume indexing within 24 hours.

Note: Although you may not meet your RTO in a mission critical system, you can still refeed yourcontent for indexing if the restore from your backups fails.

When your RPO is in units of minutes or hours, then you will probably need to make frequentincremental backups or have a high availability configuration. Because the amount of data lost onthe Content Server is reflected in the corresponding xPlore indexes, the xPlore RPO is always thesame as the Content Server RPO. Because any index data loss (because of a failure or data corruption)can always be refed to xPlore from the Content Server, any backup of or high availability for xPloreis a way of shortening the RTO (not the RPO).

12

Overview

Note: Partial refeeds and indexing are normally required to restore xPlore to a point-in-time that issynchronized with Documentum Content Server. (There is usually a lag time between new contentthat was added to a Documentum repository and that content being crawled and indexed by xPlore.)

Backup risk

In any backup strategy, there is a certain amount of inherent risk. For example, backup files canbe corrupted or lost.

To mitigate risk:

Completely test your procedures to make sure you can recover and make your xPlore installationoperational within the required RTO and RPO. (An operational deployment is able to indexand return query results.)

A best practice is to periodically test your backups (for example, by performing a restore in a testenvironment) to confirm that the backup process is working properly.

Multi-site disaster recovery

In a multi-site disaster recovery (DR) configuration, the primary installationthat is, the set ofmachines, installed software, and data that comprise the systemis replicated to a remote site.The remote site must be located far enough away from the primary installation so that the samecatastrophe will not damage it as well.

Furthermore, when the remote site is brought online, it must be configured like the primary siteTheremote instance must mirror the primary instance with the same instance name, port number,directory structure, and username/password combination.

The remote site does not have to recover to the exact same state as the primary site. For example,when a series of disk blocks are written on the primary site, they are sent to the remote site, but someof the blocks might be received out of order, some intervening blocks might be not be received, orthe disk could be corrupt at the remote site. In order to bring the remote system online, all thereceived blocks must be contiguous and in the same order as they were written on the primary site.That is, bringing the remote system online is equivalent to auto-recovering xPlore at the primary site.Although the remote site can run on reduced capacity compared to the primary site, EMC stronglyrecommends, as a best practice, that the remote site run on hardware and software that has the samecapacity as the primary one.

For example, EMC Recoverpoint is a multi-site disaster recovery product. For information aboutEMC disaster recovery technologies, see http://www.emc.com/solutions/business-need/business-continuity-availability/affordable-disaster-recovery.htm.

In disaster recovery, EMC strongly recommends performing regular backups because bringing theremote system online is not guaranteed to succeed all the time (for example, the remote disk mighthave failed or data replication might have been unknowingly misconfigured). However, if no backupexists, the content can be re-fed from the original source system (that is, the Documentum repository).

13

Overview

High availability and disaster recovery cost

The cost of high availability and disaster recovery is determined as follows:

More servers

More disk storage: Backups (for example, you could have 3 days of backups that require 6 times the amounts of

database storage); however, cost can be lowered if backups are not needed for the indexes(instead you could refeed, which would have, however, a longer RTO)

Active-active high availability requires duplicate disk storage and I/O whereas active-passiveusually uses duplicate servers but shared disk storage

Automation cost is higher with an automatic failover system such as Microsoft FailoverClustering than with manual failover such as xPlore spare instance.

14

Chapter 2

Backup and Disaster Recovery


Introduction, page 15

Content Server recovery overview, page 15

xPlore backup configurations, page 19

Restore Content Server, page 20

Restore an index at a particular point in time, page 23

Restore data after an instance fails, page 25

Introduction

When your full-text index installation fails, you recover it by restoring it to a previously consistentstate (that is, a particular point in time) from your backups. Restoring to a particular point in time isalso known as a point-in-time recovery.

If the associated Content Server has also failed, then you will need to coordinate your full-text indexrecovery with the Content Server recovery.

Content Server recovery overview

Content Server is the software that manages the repository and implements the core contentmanagement capabilities available to clients and applications. The repository is an abstractionrepresenting all the data managed by Content Server. The repository consists of content andmetadata. Examples of content are documents, emails, and attachments, and examples of metadataare content owners, modification dates, and retention periods. Content is stored in storage areas onthe server hosts file system or an external storage device such as a NAS or SAN device. Metadata isstored in a relational database, and that database also stores its data on the database hosts file systemor external storage devices. As Content Server adds content, it adds the metadata for the content intothe database. For example, when an email with an attachment is stored, the email and the attachmentare stored as content, and information about the email sender and receivers, and the fact that theattachment is associated with a particular email are stored as metadata.

15


Since Content Server modifies both the storage area and the database for every operation on content,any backup and recovery strategy must take into account the interdependency of the data in thestorage areas and in the database. The idea is to capture the storage areas and the database as close toa single point in time as possible. When these elements are recovered to that same point, ContentServer will be consistent and will function properly. In practice, the backups of the two elementsmay not be precisely to the same point in time. In those cases, there are tools to inspect and repairthese consistency issues.

Full-text indexing also has data dependencies to consider when planning backup and restore.

Figure 1. Backup for Object Consistency

For smaller installations, suspending the system and performing a cold backup is a good strategyfor insuring storage area and database consistency, and is the simplest use case. (This case iscovered in .) However, Content Server is commonly used in enterprise-wide applications requiringhigh-availability and short RPOs, so we will also discuss some backup and restore strategies thatdo not require shutting down Content Server.

You may need to recover your entire Content Server deployment or only part of it. A recovery of yourentire deployment is described here because it is the most complex kind of recovery.

Although there are many different potential storage configurations for Content Server, they all have anumber of commonalities:

1. Content Server is usually deployed in a disaster recovery configuration: All data stores for the database, content, and full-text indexes are replicated from the primary

site to a disaster recovery site.

The disaster recovery site is backed up using a BCV-like (that is, snapshot) technology.

2. Data replication from the primary site to the disaster recovery site completes within the RPO.

Because the amount of data being replicated differs between the database, content, and full-textindexes, the overall amount of time it takes for each one to be completely replicated also differs.In general, content takes the longest to replicate; whereas the database takes the shortest to

16


replicate. Consequently, the replication time for content is the maximum replication time for theentire deployment (as illustrated in Figure 2, page 18). The data replication rate depends on thenetwork bandwidth and the oncoming load as well as a host of other environment factors. Inorder to set the RPO correctly, you must determine the maximum amount of time it takes for data(new and modified) to be completely replicated from the primary site to the disaster recovery site.

3. Backups are performed as follows: Content is backed up incrementally or, if Centera is the storage area, not required to be backed

up (as illustrated in Figure 2, page 18).

The database is hot backed up incrementally at regular intervalsas determined by therecovery point objective and the maximum replication time.

The amount of time that elapses between database backups (referred to as the databasebackup interval) should be at least three times as long as the entire deployments maximumreplication time (in Figure 2, page 18, the database backup interval is four times the maximumreplication time).

Full-text indexes are hot, cold, or warm backed up.

The full-text index backup interval is at least two times that of the database backup interval(as illustrated in Figure 2, page 18). In addition, you must wait for at least two times the entiredeployments maximum replication time before backing up the full-text index.

Tip: In practice, since the full-text indexes can be restored point-in-time from the database andcontent, the full-text indexes could be backed up as infrequently as once a week whereas thedatabase and content would be backed up many times a day (as illustrated in Figure 2, page18). Full-text index backups must be performed frequently enough to meet the RTO. Thatis, the full-text index point-in-time recovery is determined by the amount of data receivedduring the full-text index backup interval.

17


Figure 2. RPO for disaster recovery site (including data replication from the primary site)

Note:

Italicized text indicates that the operation is optional (as is the case for Content Backup).

The time scale and line lengths of the operations indicate only a rough order of magnitudethatis, they are not typical nor do they represent a recommendation.

Figure 2, page 18 illustrates the most complex recovery case in which object metadata (in the database)is replicated faster than both the content storage area and the full-text indexes. Consequently, whenthe primary site completely fails (the Point of Failure in Figure 2, page 18), the database is morecurrent than the content (relative to consistency) and there may be objects that point to content thatdoes not exist in the storage areasthe Content Server is considered to be in an inconsistent state.

Note: If the content is newer than the metadata, there will be content in the storage area that is notassociated with any objects in the database. This is called orphaned content. Orphaned contentindicates that some transactions occurred in the database that were not included in the databasebackup. However, orphaned content will not adversely affect Content Server.

To recover from an inconsistent Content Server state:

1. Restore the database to the point in time when it was consistent with the content.

See Restore Content Server, page 20.

For example, assuming that the replication data latency is at least three times as great as the timeit takes to back up the database, you should restore the database to the last incremental backupor to the second-to-the-last backup.

Do not roll back the database to a point before the full-text backup, because a re-feed of thefull-text index will leave objects on the full-text index that may never get cleaned up.

2. After synchronizing the database and content, re-feed some content through Content Server.

18


Tip: You can optionally re-feed the content after synchronizing the full-text indexes with thedatabase and content.

3. Roll back the full-text index to its last backup before the database backup.

See Restore an index at a particular point in time, page 23.

The ftintegrity tool can be used to identify objects that have not been indexed (and objects that areindexed but not in the repository) over a certain period of time and use that information to makethe full-text indexes consistent with the Content Servers database and content.

xPlore backup configurations

xPlore features different levels of backup, which can be combined with other products to implementyour desired level of backup and recovery (including multi-site disaster recovery). xPlore mustbe offline to perform any restore operations. These are the levels within xPlore at which you canperform backup and restore tasks:

Note: Each level supports a backup scope and restore technology.

Collection

In large environments, such as a very large, single Documentum repository, you can selectivelyback up and restore collections, so that collections with older indexes can be backed up andrestored separately from ones with newer indexes.

Domain

You can back up and restore all the collections in a domain (across all xPlore instances) together ina coordinated fashion. A domain-level backup would be very efficient for a single xPlore instancethat was servicing many Documentum repositories (that is, multi-tenant and shared resources).

Federation

You can back up and restore your complete xPlore installation.

In xPlore, you can perform these types of backup:

Cold

A cold backup is the simplest kind of backup but it disrupts the service the most because xPloremust be shutdown for the duration of the backup.

Warm

A warm backup allows xPlore to return queries (possibly with a higher latency) but indexing isstopped (the index agent does not send any documents to xPlore for indexing) for the durationof the backup.

Hot

A hot backup allows xPlore to both return queries and continue indexing.

Backup technologies xPlore supports these backup technologies:

Native xDB backups

In this kind of backup, xDB creates its own backup file.

File-based backups

19


In this kind of backup, individually listed files or entire directories (and their contents) are copiedfrom their current location to a backup staging area. The specific directories and files that arebacked up are the xPlore federation directory dsearch_home/data, dsearch_home/config and /dblogfiles.

Note: Cumulative file-based backups are generally not recommended, since most files are touchedwhen they are opened. In addition, Windows file-based backup software requires exclusive accessto a file during backup, requiring a cold backup.

Volume-based (snapshot) backups

This kind of backup backs up disk blocks on the filesystem. A third-party product, such as EMCTimefinder, is typically required.

File- and volume-based backup and restore operations typically require two times the disk space(which includes the index and temporary space) as the size of the current index. Third-party backupproducts such as EMCNetworker are also supported. All backup and restore commands are availableas command-line interfaces (CLI) for scripting. See the xPlore Administration Guide for details aboutthe xPlore backup and restore scripts.

Note:

See Powerlink for white paper examples of using specific backup manager products with xPlore.

See the EMC Documentum xPlore Disaster Recovery Using EMC Networker whitepaper,https://community.emc.com/docs/DOC-9548.

Table 1, page 20 describes the differences between supported backup combinations. Periodic fullbackups are recommended in addition to differential backups.

Table 1. Backup scenarios

Level Backup type DR technology Backup scope

collection warm xPlore full only

domain warm xPlore full only

warm or hot xPlore full, incremental, orcumulative

cold or warm volume* full or cumulative

xPlore federation

cold or warm file* full only

* For file-based and volume-based backups, back up the following files on each instance:indexserverconfig.xml file, the xDB transaction log files in dsearch_home/dblog, and the databaseand index files in dsearch_home/data. Each xPlore instance has one or more domains and asingle xDB transaction log for instance data recovery. Back up each instance in a multi-instanceenvironment to a single file, then restore the instance from this file.

Restore Content ServerTo restore Content Server:

1. Shut down the server, if it is running.

20


Use the procedure described earlier.

2. Restore the RDBMS (metadata) and the storage areas (content).

Restore your backup data using the tools you used to do the backups.

3. Restart the server.

Use the procedure described earlier.

4. Synchronize the restored data.

The last step is described in the next section.

Synchronize the restored data

After the content and metadata are restored, there are only three possibilities: the metadata andcontent are restored to the exact same time (typical only with cold backups), the content is newer, orthe metadata is newer. In this section, newer refers to the time of creation of the data at the productionsite, not the time of the backup. Since there can be different time lags for content and metadata, abackup of content that happened later in time can have older data than that of the metadata backup,even though the metadata backup occurred earlier in time.

For example, assume that the content data that is replicated to the DR site lags by two hours, and thatthe metadata lags by one hour. If your metadata backup is taken thirty minutes after the contentbackup, then the metadata backup is actually thirty minutes newer than the content backup.

If the restore point is the same for both (the exact same time), then the data is synchronized, and nofurther action is required. For small repositories, the administrative job, dm_ConsistencyChecker,will perform a number of consistency checks and help verify that the restore was successful. See EMCDocumentum Content Server Administration Guide for details about how to run the job. For largerrepositories, that job will not be able to complete in time to meet your RTO, so youll have to developanother strategy to insure the consistency of your repository.

In planning for DR and backup and restore, use the characteristics of your architecture to plan for thescenario that is likely to occur.

Restore if content is newer

If the content is newer than the metadata, there will be content in the storage area that is notassociated with any objects in the database. This is called orphaned content. Orphaned contentindicates that some transactions occurred in the database that were not included in the databasebackup. However, orphaned content will not adversely affect Content Server, so you may choose toignore the orphaned content.

If the timestamps for your backups are reliable, and you account for any variance in lag times for thedata replication from the production site, you will be able to identify the case of orphaned content.You can also use the dm_ConsistencyChecker job to test for this condition. For large installations,it may not be possible to run the consistency checker job within the RTO period, so the timestampmethod may be your method of choice to identify this condition.

21


Figure 3. Storage Area Backup is Newer

Orphaned content can be removed by running the administrative job, dm_DMClean, described inEMCDocumentum Content Server Administration Guide, but typically youwould only use this on a smallrepository as there will not be enough time to run it on a large repository and meet your RTO goal.However, you may choose to run it at a later time as part of regular maintenance of Content Server.

Restoring if metadata is newer

When the metadata is newer than the content, there will be objects in the database that point tocontent that does not exist in the storage areas. This condition is likely if the lag time for hot backupof the content is longer than the lag time for the backup of the metadata. Your recovery strategy mustplan for a way to repair this condition, since Content Server will be adversely affected by this.

Figure 4. Database Backup is Newer

One possible scenario involves restoring the database from an earlier incremental backup. If youuse this method, the content will be newer than the metadata, and you can use the techniquesdescribed for this condition.

22


Another possibility is to use your database tools (transaction logs or journals) to unroll the state of thedatabase back in time to match (or to be earlier than) the state of the storage areas. Depending onhow you have set up your database, you may be able to roll back the metadata and lose very little.

To determine how far to roll back the database, you can use the timestamp method to attempt to rollback to just before the content backup. If you have considerable experience with Content Server,or if you use the services of EMC Professional Services (and qualified 3rd party Integrators), it ispossible to create a customized application, perhaps a script, for restore that will check the objects inthe database against the contents to find the oldest database objects that have content successfullyrestored to the content area.

Restore an index at a particular point in time

Figure 5, page 24 A typical workflow for restoring xPlore instances:

23


Figure 5. Restoring xPlore Instances Decision Tree

1. Retrieve a list of object IDs for which objects need to be re-fed by running ftintegrity (or a query)for this period: Start date and time the most recent xPlore backup before the Documentum system crash

occurred.

End date and time when the Documentum system crash occurred.

2. Restore xPlore from backups.

3. Re-feed objects using the results from ftintegrity (or a query).

For detailed instructions about backing up and restoring full-text indexes to a particular point in time,see the xPlore Administration Guide.

24


Restore data after an instance fails

In HA setup, you can restore data from one instance to another when one of the instances fails.

The following steps use two example instances: InstanceA and InstanceB. xPlore is set up on thesetwo instances with mirrored information: same folder structures, same username/password, sameinstance name, same ports, etc. The example assumes that data corruption occurs on InstanceB.

1. Stop xPlore and IndexAgent on InstanceA. Record the time the index agent is shut down.

2. Stop xPlore and IndexAgent on InstanceB.

3. Backup dsearch_home/config/indexserverconfig.xml and dsearch_home/config/XhiveDatabase.bootstrap on InstanceB. You will use this backup to recover configurationinformation after you restore data.

4. Copy the directories dsearch_home/data and dsearch_home/config from InstanceA to InstanceB.Do not change file permission.

5. Optionally, start xPlore and the index agent in normal mode (not migration mode) on InstanceAto put InstanceA in service.

6. Update dsearch_home/config/indexserverconfig.xml on InstanceB with settings inthe original backup copy. Change the attributes url and hostname of the XML elementindex-server-configuration.node to match those values in the backup file.

For Windows systems that use UNC paths, update the following XML elements or attributesto their original values in the backup copy:

/tracing-config/.tracing.output[@dir]

/storage-locations/storage-location[@path]

/admin-config/backup-location[@path]

7. Update dsearch_home/config/XhiveDatabase.bootstrap on InstanceB to match the originalfile. Change attribute host of the element /server/node to match the value in the backup file.

For Windows systems that use UNC paths, update the following XML elements or attributes totheir original values in the backup copy. Update the path only and do not change the file name.

/server/node/log/[@path]

/server/database/segment/file/[@path]

8. For Windows systems that use UNC paths:

a. Open the dsearch_home/config/luceneindex.bootstrap file.

b. For each element update the host and data values with the values specified in thebackup copy. The domain, collection, and the rest of the path remains the same.

For example, change \\hostA\dataA/test/ThesaurusDB/lucene-index/prefLabel to\\hostB\dataB/test/ThesaurusDB/lucene-index/prefLabel.

c. Navigate the updated path and open the folder.

For example, navigate to \\hostB\dataB/test/ThesaurusDB/lucene-index/prefLabel.

d. Open the subfolder that begins with El.

25


For example, open EI-6f0a2fd0-12ff-4df8-8e26-4ea3ec7dae28.

e. Open the index_info file.

f. Update the host and data values in the indexPath. The rest of the path remains the same.

For example, change \\hostA\dataA\{domain}{collection}\lucene-index\dmftdoc\EI-6f0a2fd0-12ff-4df8-8e26-4ea3ec7dae28\LI-7fd7ac68-a567-4d31-bf1c

-93f9aa5def02 to \\hostB\dataB\{domain}{collection}\lucene-index\dmftdoc\EI-6f0a2fd0-12ff-4df8-8e26-4ea3ec7dae28\LI-7fd7ac68-a567-4d31-bf1c

-93f9aa5def02.

9. If InstanceB failed some time ago, you must clear many duplicate indexing tasks that are queuedas dmi_queue_items. Run DQL to truncate the queued items for IndexAgentB: In the followingexample, IA_user is the user for IndexAgentB, date_value is a time just before the index agentshutdown time recorded in step1.

?,c,delete dmi_queue_item objects where name=IA_user and

date_sent

Chapter 3

xPlore High Availability Deployments


Overview, page 27

Content Server full-text objects and initialization files, page 28

Set up Content Server and xPlore HA, page 30

Overview

In a high-availability (HA) deployment, if an xPlore instance fails, then a corresponding instancecan take over; CPS, indexing, and search operations continue on the secondary instance. An HAdeployment provides the following benefits:

Increased query availability because support is provided for failover.

Redundancy if a host fails.

If you require xPlore high availability, you will typically also require high availability for yourContent Server. Therefore, you should coordinate your Content Server and xPlore high availabilitydeployments. Although many Content Server and xPlore high availability configurations arepossible, a single repository served by two Content Servers is typical and can be used as a buildingblock for more complex deployments. See the EMC Documentum Content Server Enterprise EditionInstallation Guide for more information about the different high-availability deployments andinstallation instructions for Content Server.

xPlore supports these kinds of high availability deployments:

Spare instance (standby) A spare instance is a warm instance that can be manually broughtonline when a running node fails. When the spare instance is started, it rebuilds the index fromthe previous instances transaction log, which must be accessible in a shared directory (the dataand index directories must also be accessible to the spare instance). This functionality is includedwith xPlore.

You could also deploy a hybrid model wherein you use Active-Passive or Active-Active clustersfor the indexes only.

Active-Passive Active-Passive is a failover technology that enables an application and its datato automatically start up on alternative hardware after the primary application fails. Microsoft,Redhat, and Veritas cluster servers are examples of active-passive clusters. Although failover

27


is automated, active-passive cluster requires a heavy investment in duplicate servers but notduplicate disk storage.

Active-Active Active-Active HA consists of two complete, running xPlore deployments. Twodifferent index agents send the same content to each deployment. Queries are directed to one ofthe two active deployments or load balanced across both of them. In Active-Active HA, you mustmanage two deployments and is the most expensive option because it requires duplicate serversand disk storage. You could have as much as four times the number of hosts as your initial xPloredeployment (or have high availability in multiple data centers).

Table 2, page 28 illustrates how the cost increases as recovery time increases and automation becomesavailable. The xPlore spare instance and active-passive options are not available with FAST. If youhave an active-active HA for FAST, then that strategy can be used for xPlore.

Table 2. HA Strategy Comparison

HA Strategy Cost Recovery Time FailoverAutomation

Notes

Spare instance $ Slowest None Not as automatedbut veryinexpensive.

Active-Passive $$ Medium Yes Highlyautomated;standard andwell-knowntechnology.

Active-Active $$$ Fastest Yes Has been inplace the longesttime. Requiresmore time toadminister theduplicate system.

Content Server full-text objects andinitialization files

Because HA requires configuring the full-text indexing system in a manner beyond basic installation,it is important to understand how the full-text indexing system is configured in the repository as wellas the Content Server and xPlore initialization files. Figure 6, page 29 illustrates the relationshipsbetween the objects that contain configuration information about the full-text indexes, index servers,and index agents and the Content Server and xPlore initialization files.

28


Figure 6. Full-text Index Objects

Each repositorys dm_server_config.fulltext_location specifies the dm_location object(dsearch for xPlore), in which file_system_path specifies the path to dmfulltext.ini.

The dmfulltext.ini file, which is created when the server is installed, contains informationused by Content Server to find the index agent query plug-in binary files. dmfulltext.ini islocated in: Windows: %DOCUMENTUM%\fulltext\dsearch\dmfulltext.ini

Unix: $DOCUMENTUM/fulltext/dsearch/dmfulltext.ini

where $DOCUMENTUM (Unix) and %DOCUMENTUM% (Windows) are environment variablesthat specify the Documentum Content Server root installation directory. Windows default:C:\Documentum. Unix default: none.

Each repositorys server.ini uses the ftengine_to_use parameter to reference thedm_ftengine_config.r_object_id. The server.ini file is located: Windows: %DOCUMENTUM%\dba\config\repository\server.ini

Unix: $DOCUMENTUM/dba/config/repository/server.ini

Note: ftengine_to_use is only used in dual mode. It is not a required.

For each full-text index, dm_fulltext_index.ft_engine_id specifies thedm_ftengine_config object to use.

dm_fulltext_index.is_standby specifies whether the dm_fulltext_index object is onstandby and the index to use for queries. If the is_standby value is 0, then the index specifiedby the dm_fulltext_index object is the default and is used for queries; if the value is 1, then theindex specified by the dm_fulltext_index object is in standbymode and is not used for queries.

Note: The ftengine_to_use property in server.ini overrides the dm_fulltext_index

.is_standby setting; that is, queries are sent to the index that is referenced by the

29


dm_fulltext_index object, which is, in turn, specified by the ftengine_to_use property,even if the dm_fulltext_index objects is_standby=1.

dm_fulltext_index.is_standby does not set the index to use for indexing requests; thatis, both indexes service all indexing requests.

The dm_ftindex_agent_config object represents an index agent configured for the repository.Its properties record status information and configuration information about the index agent.

Set up Content Server and xPlore HA

These instructions apply to either a completely new or existing Content Server and xPloredeployment.

Spare instance HA (standby), page 30

Dual primary instance HA (active-active), page 32

For instructions about deploying xPlore HA in Windows Server 2008 Failover Clustering(Active-Passive), see theMultinode EMC Documentum xPlore High-Availability Deployment UsingWindows Server 2008 Failover Clustering White Paper (https://community.emc.com/docs/DOC-9547).

Spare instance HA (standby)

Figure 7, page 32 illustrates a Content Server HA deployment coupled with xPlore spare instanceHA. A single spare instance can support multiple running xPlore instances. Your risk increasesin proportion to the number of running xPlore instances that you want a single spare instance tosupport. You must perform more manual tasks when replacing a primary instance than whenreplacing a secondary instance; for example:

Edit xDB.properties and indexserver-bootstrap.properties in all other xPlore instances to referencethe new primary instance.

Update all clients, such as xDB initialization, index agent, and query plugin, to point to the newprimary instance name.

For instructions on configuring xPlore spare instance, see the xPlore Administration Guide.

30


In Figure 7, page 32:

Host1: Primary Content Server host Content Server (CS1)

Connection broker (CB1)

Host2: Secondary Content Server host Content Server (CS2), which contains the stand-by full-text index


Host3: Primary xPlore host Primary instance (Pri_Inst)

Secondary instance (Sec_Inst)

Host4: Spare xPlore host Spare primary instance (Pri_Inst_Sp)

Spare secondary instance (Sec_Inst_Sp)

Host5: xPlore primary instance data, log, and configuration files (Config_Files), which mustbe writable from both the running instances as well as the spare instances.

NAS/SAN: Full-text indexes

Host4 maintains the spare instances. The Pri_Inst/Host3 configuration files (Config_fileson Host5 in the figure) must reside in a location that is writable by Pri_Inst_Sp/Host4 andSec_Inst_Sp/Host4 because when you start the spare instances on Host4, they must load thesame configuration parameters. For indexing to be routed to the correct Content Server, IA1mustconnect to CB1 and IA2 must connect to CB2.

The After Failover diagram illustrates that after Sec_Inst on Host3 fails, you manually startthe Sec_Inst_Sp on Host4, which reads the configuration data from the same location asSec_Inst_Sp and all indexing now goes through Sec_Inst_Sp on Host4. If the entire Host3machine fails, then you can switch to both Pri_Inst_Sp and Sec_Inst_Sp on Host4.

31


Figure 7. xPlore Spare Instance HA

Dual primary instance HA (active-active)

Figure 8, page 33 illustrates multinode HA coupled with two xPlore primary instances on separatehosts. In Figure 8, page 33:

32


Note: The active-active HA configuration maintains indexing in case of a failover; however, you mustmanually restart query services in the standby Content Server.

Host1: Primary Content Server host Content Server (CS1)


Host2: Secondary Content Server host Content Server (CS2), which contains the stand-by full-text index


Host3: First primary xPlore host Index agent (IA1) that connects to CS1

Primary instance (Pri_Inst1)

Host4: Second primary xPlore host Index agent (IA2) that connects to CS2

Primary instance (Pri_Inst2)

Host5: First xPlore primary instance data, log, and configuration files (Config_Files), whichmust be writable from both the running instances as well as the spare instances.

Host6: Second xPlore primary instance data, log, and configuration files (Config_Files), whichmust be writable from both the running instances as well as the spare instances.

NAS/SAN: Full-text indexes for the first primary xPlore host

NAS/SAN2: Full-text indexes for the second primary xPlore host

Figure 8. Two xPlore Primary Instances HA

33


Procedure

This procedure is for setting up a single repository with HA (that is, two or more Content Serverhosts).

If you are setting up a single repository with a single Content Server host, follow these steps butspecify the same connection broker for both index agents.

After setting up xPlore HA, initialize and verify full-text indexing (see To initialize and verifyindexing:, page 36).

To set up HA for full-text indexing:

Prerequisites:

Two Content Servers on separate hosts in a high-availability deployment.

Note:

Ensure that no users are connected to the repository.

All scripts are located in:

Windows: %DM_HOME%\install\admin

Unix: $DM_HOME/install/admin

1. (Optional) If you are upgrading an existing Content Server 5.3 SP2 or 5.3 SP3 high-availabilitydeployment, execute the create_fulltext_objects_ha.ebs script with theHACleanupBeforeUpgradeStep.

Caution: Execute this script after deleting existing index agents and index servers but beforeinstalling new index agents and index servers.

Note:

xPlore 1.1 does not support Content Server 5.3.

Executing the create_fulltext_objects_ha.ebs script with theHACleanupBeforeUpgradeStep deletes all dm_ftengine_config anddm_fulltext_index objects.

The syntax is:

dmbasic -f create_fulltext_objects_ha.ebs -e HACleanupBeforeUpgradeStep -- repositorysuperuser password

where repository is the name of the repository, superuser is the user name of a user with superuserprivileges in the repository, and password is the superusers password.

Note:

Enter a space before and after the double hyphen.

No environment variables in paths.

2. Install and configure the primary xPlore deployment.

3. Install and configure the primary xPlore index agent.

Specify the first Content Servers connection broker as the connection broker for this index agent.

34


The index agent connects to the repository and creates the full-text indexing objects that arerequired to run the scripts in the subsequent steps.

4. Shut down the primary xPlore index agent.

5. On the primary Content Server host:

Note: Because the scripts create and configure objects in the repository, you can execute thescripts on either Content Server.

a. Log in as the installation owner.

b. Execute create_fulltext_objects_ha.ebs as follows:

dmbasic -f create_fulltext_objects_ha.ebs -e HAPreInstallStep -- repositorysuperuser password

where repository is the name of the repository, superuser is the user name of a user withSuperuser privileges in the repository, and password is the Superusers password.

Note:



Creates a new dm_fulltext_engine_config object for the secondary xPlore deployment.

6. Install and configure the secondary xPlore deployment.

7. Install and configure the secondary xPlore index agent.

Specify the secondary Content Servers connection broker as the connection broker for this indexagent.

Do not start this index agent.

8. On the primary Content Server host:

Note: Because the scripts create and configure objects in the repository, you can execute thescripts on either Content Server.

a. Log in as the installation owner.


dmbasic -f create_fulltext_objects_ha.ebs -e HAPostInstallStep -- repositorysuperuser password


Note:



Creates a new dm_fulltext_index object and a new dm_ftindex_agent_config objectfor the secondary xPlore deployment.

9. Restart both Content Servers.

10. Confirm that the scripts created two of each of these objects:

dm_fulltext_index

dm_ftindex_agent_config

35


dm_ftengine_config

Each set of these objects should correspond to each xPlore deployment.

a. Using IAPI, connect to the repository as a user with superuser privileges.

b. To verify that there are two dm_fulltext_index objects (each one is associated with oneof the two indexes and each has an associated dm_ftengine_config object), execute thefollowing Documentum API queries:

For the default engine:

API> ?,c,select e.r_object_id,e.object_name from dm_ftengine_config e,dm_fulltext_index fi where e.r_object_id=fi.ft_engine_idand fi.is_standby=0

For the standby engine:

API> ?,c,select e.r_object_id,e.object_name from dm_ftengine_config e,dm_fulltext_index fi where e.r_object_id=fi.ft_engine_idand fi.is_standby=1

c. To verify that there are dm_ftindex_agent_config objects for each index and that thequeue user is correctly set for the second index, execute the following Documentum APIqueries:

For the default index agent:

API> ?,c,select ia.r_object_id,ia.object_name,ia.queue_userfrom dm_ftindex_agent_config ia,dm_fulltext_index fiwhere ia.index_name=fi.index_name and fi.is_standby=0

For the standby index agent:

API> ?,c,select ia.r_object_id,ia.object_name,ia.queue_userfrom dm_ftindex_agent_config ia,dm_fulltext_index fiwhere ia.index_name=fi.index_name and fi.is_standby=1

Note: The object_name records the name of the index agent and index_name is the name ofthe index.

To initialize and verify indexing:

1. Start both index agents.

Note: Starting the index agents creates the indexes.

2. To verify that the indexes have been created correctly, run ftintegrity or the State of theIndex job on each Content Server. See the Documentum xPlore Administration Guide.

Enable full-text queries to be serviced by the secondary xPloredeployment

When the primary xPlore deployment fails, although indexing continues on the secondary xPloredeployment, you must manually enable queries to be serviced by the secondary xPlore deployment.

1. To retrieve the secondary dm_fulltext_index object ID, execute this DQL statement:

select ft_engine_id from dm_fulltext_index where is_standby=1

36


2. On both the primary and secondary Content Server, edit the server.ini file locatedin documentum_home/dba/config/servername. Under [SERVER_STARTUP], set theftengine_to_use value to the secondary dm_fulltext_index object ID.

change ftengine_to_use value in server.ini to the secondary dm_fulltext_indexobject ID.


xPlore dual mode with an existing FAST HA deployment

If you have an existing FAST HA (with two Content Servers and one shared repository) and want toset up FAST-xPlore dual mode, then follow the instructions in this section.

Note: For more information about dual mode, see the Documentum xPlore Installation Guide.

In this FAST HA configuration, there are these relevant objects:

One dm_fulltext_index object with an object_name of _ftindex_00

where:

Name of the Documentum repository.

One full-text index user named dm_fulltext_index_user_01

xPlore dual mode must reserve the same objects as FAST HA. You must change the FAST HA objectnames, so that you can create the xPlore objects with those names.

1. Shut down the FAST index agent and index servers.

2. Execute these commands to check whether these objects exist and create new ones with differentnames.

a. Use this command to check if dm_fulltext_index_user_01 exists:

API> retrieve,c,dm_user where user_name='dm_fulltext_index_user_01'

b. If dm_fulltext_index_user_01 exists, then execute these commands to create a new user:

API >create,c,dm_userAPI >set,c,l,user_nameSET>dm_fulltext_index_user_02API > set,c,l,user_login_nameSET> dm_fulltext_index_user_02API> set,c,l,user_privilegesSET> 16API> save,c,l

A unique, full-text index user (which is used when generating queue item events) is requiredfor each index.

Note: Make sure that a dm_fulltext_index_user_02 user does not exist.

3. Reserve a new dm_fulltext_index name for xPlore as follows:

a. Retrieve the index_name of dm_fulltext_index object that is to be reserved for thexPlore dm_fulltext_index by executing the following command:

37


API> ?,c,select index_name from dm_fulltext_index where index_name like '%_ftindex_00'

The index_name is retrieved. In these instructions, the variable for this index_nameis .

b. Change to _02 by executing the followingcommand:

API>execsql,c,update dm_fulltext_index_s set index_name='_02' where index_name =''

4. Change the dm_ftindex_agent_config object where its queue user is dm_fulltext_index_user_01 and index_name is as follows:

API>?,c,update dm_ftindex_agent_config object set queue_user='dm_fulltext_index_user_02' where queue_user='dm_fulltext_index_user_01'

API>?,c,update dm_ftindex_agent_config object set index_name='_02' where index_name =''

5. Update the event in dmi_registry where its user_name is dm_fulltext_index_user_01by executing the following command:

SQL> update dmi_registry_s set user_name ='dm_fulltext_index_user_02' whereuser_name ='dm_fulltext_index_user_01'

.

6. Set up xPlore for dual mode.

Note: For instructions on setting up dual mode, see the Documentum xPlore Installation Guide.

Setting up xPlore for HA after dual mode has been configured

Once you have set up xPlore in dual mode, you can set up xPlore for HA.

Note: Steps 1 - 5 are the same as in the previous procedure except that you must specify a differentfull-text index user and dm_fulltext_index.

1. Shut down the following:

xPlore index agents

xPlore instances

FAST index agents

FAST index servers

2. Execute these commands to check wether these objects exist and create new ones with differentnames.

a. Use this command to check if dm_fulltext_index_user_01 exists:

API> retrieve,c,dm_user where user_name=dm_fulltext_index_user_01

b. If dm_fulltext_index_user_01 exists, then execute these commands to create a new user:

API >create,c,dm_userAPI >set,c,l,user_nameSET>dm_fulltext_index_user_03API > set,c,l,user_login_nameSET> dm_fulltext_index_user_03

38


API> set,c,l,user_privilegesSET> 16API> save,c,l

A unique, full-text index user (which is used when generating queue item events) is requiredfor each index.

Note: Because you have already created dm_fulltext_index_user_02 whenyou set up dual mode, create dm_fulltext_index_user_03. Make sure that adm_fulltext_index_user_03 user does not exist.

3. Reserve an index_name for xPlore as follows:

a. Retrieve the index_name of dm_fulltext_index object that is to be reserved for thexPlore dm_fulltext_index by executing the following command:

API> ?,c,select index_name from dm_fulltext_index where index_name like '%_ftindex_00'

The index_name is retrieved. In these instructions, the variable for this index_nameis .

b. Change to _03 by executing the followingcommand:

API>execsql,c,update dm_fulltext_index_s set index_name='_03' where index_name =''

Note: Because you have already created _02 when youset up dual mode, create _03. Make sure that a_03 dm_fulltext_index does not exist.

4. Change the dm_ftindex_agent_config object where its queue user is dm_fulltext_index_user_01 and index_name is as follows:

API>?,c,update dm_ftindex_agent_config object set queue_user='dm_fulltext_index_user_03' where queue_user='dm_fulltext_index_user_01'

API>?,c,update dm_ftindex_agent_config object set index_name='_03' where index_name =''

Note: Because you have already created _02when you set up dual mode,create _03. Make sure that a _03dm_fulltext_index does not exist.

5. Update the event in dmi_registry where its user_name is dm_fulltext_index_user_01by executing the following command:

SQL> update dmi_registry_s set user_name ='dm_fulltext_index_user_03' whereuser_name ='dm_fulltext_index_user_01'

6. Change the dm_ftengine_config objects object_name attribute as follows:

API> ?,c,update dm_ftengine_config object set object_name='DSearch Fulltext Engine Configuration 00'where object_name='DSearch Fulltext Engine Configuration'

7. Install a second xPlore deployment.

8. Install an index agent, configure its dsearch_home\setup\indexagent\config.properties file as follows and point it to the second xPlore deployment:

indexagent.dualmode=true

Note: For instructions on setting up dual mode, see the Documentum xPlore Installation Guide.

39


xPlore HA dual mode with an existing FAST HAdeployment

If you have an existing FAST HA (with two Content Servers and one shared repository) and want toset up FAST-xPlore dual mode on each Content Server with xPlore HA, then follow the instructionsin this section.

Note: For more information about dual mode, see the Documentum xPlore Installation Guide.

In this FAST HA configuration, there are these relevant objects:

Two dm_fulltext_index objects each with an object_name of _ftindex_00 and _ftindex_01

where:

Name of the Documentum repository.

Two fulltext users named dm_fulltext_index_user and dm_fulltext_index_user_01

xPlore HA with dual mode on a single Content Server uses the same object names as FAST HA.You must change the FAST HA object names, so that you can create the xPlore HA objects withthose names.

1. Shut down the FAST index agent and index servers.

2. Rename the _ftindex_00 index name to _ftindex_00_00 and update the object that references it by executing these IAPI commands:

execsql,c,update dm_fulltext_index_s set i_vstamp=i_vstamp+1,index_name='' where index_name=''

execsql,c,update dm_ftindex_agent_config_s set index_name='' where index_name='

3. Shut down the second Content Server, which ensures that the following command updatesonly the first Content Servers serverconfig.

dmbasic f fulltext_setup_for_dss.ebs eSetupDualModeForSingleCS --

where:

Name of the repository

Administrator or superuser name

Password of the administrator or superuser

Fulltext home. For example, c:\documentum\fulltext

40


Note: Do not use a macro like $DM_DOCUMENT; use the full file system patch. Do not add abackslash ( \ )at the end of the path.

A new fulltext user. For example, dm_fulltext_index_user_02

This command creates and updates the following objects:

New dm_fulltext_index_user_02 user.

For any dm_ftindex_agent_config object that has a queue_user valueof dm_fulltext_index_user, dm_fulltext_index_user is changed todm_fulltext_index_user_02.

For any dmi_registry object that has a user_name value of dm_fulltext_index_user,dm_fulltext_index_user is changed to dm_fulltext_index_user_02.

For any dm_fulltext_index object with an index_name value of _ftindex_01, _ftindex_01 is changed to _ftindex_01_00.

Note: To process (to the FAST index server) any existing queue items that have beengenerated for dm_fulltext_index_user but not completed in dmi_queue_item,update dmi_queue_item and set all items with name of dm_fulltext_index_user todm_fulltext_index_user_02.

4. Restart the first Content Server so that the xPlore query plugin is loaded.

5. Install xPlore and configure the xPlore server and xPlore index agent to point to the first ContentServer.

See the EMC Documentum xPlore Installation Guide.

After index agent configuration, dm_fulltext_index_user is re-used and adm_fulltext_index object with index_name of _ftindex_01 iscreated.

6. Start the second Content Server and stop the first Content Server.

7. (Optional) If the path of fulltextHome on the second Content Server is different from thefirst Content Server, create a new dsearch dm_location object with an object_name ofdsearch_01 for the second Content Server as follows:

a. Using IAPI, connect to the repository as a superuser.

b. Create a new dm_location object by executing these commands:

iapi>create,c,dm_locationiapi>set,c,l,object_nameSET>dsearch_01iapi>set,c,l,path_typeSET>directoryiapi>set,c,l,iapi>set,c,l,no_validationSET>Fiapi>save,c,l

41


The fulltextHome of the secondary Content Server with \dsearch appended to it (forexample, c:\documentum\fulltext\dsearch).

c. Update the docbaseconfig to point to the dsearch_01 location object by executing thesecommands:

iapi>append,c,docbaseconfig,fulltext_install_locsSET>dsearch_01iapi>save,c,docbaseconfig

d. Update the second Content Servers serverconfig to point to the dsearch_01 locationobject by executing these commands:

iapi>set,c,serverconfig,fulltext_locationSET>dsearch_01iapi>save,c,serverconfig

e. Restart the second Content Server to load the xPlore query plugin.

Do not start the first Content Server.

8. Execute create_fulltext_objects_ha.ebs with the HAPreInstallStep entry pointas follows:

dmbasic -f create_fulltext_objects_ha.ebs -e HAPreInstallStep -- repositorysuperuser password

where repository is the name of the repository, superuser is the user name of a user with Superuserprivileges in the repository, and password is the Superusers password.

Note:



Running HAPreInstallStep performs the following actions:

Renames index_name from _ftindex_01 to _ftindex_00.

Appends 00 to the end of the xPlore dm_ftengine_config objects object_name.

9. Install and configure xPlore and an xPlore index agent to point to the second Content Server.

See the EMC Documentum xPlore Installation Guide.

10. Edit the create_fulltext_objects_ha.ebs script as follows and then execute it specifyingthe HAPostInstallStep entry point:

a. On the regUserName = "dm_fulltext_index_user_01" line, change dm_fulltext_index_user_01 to a new user (for example, dm_fulltext_index_user_03(becausedm_fulltext_index_user_01 was used in FAST HA).


dmbasic -f create_fulltext_objects_ha.ebs -e HAPostInstallStep -- repositorysuperuser password


Note:



42


11. Start the first Content Server.

Note:

Search requests are sent to the primary xPlore index server.

To send search requests to the first FAST index server:

1. Update the fulltext_location of both serverconfigs to the FAST full-text location.

2. Set ftengine_to_use in both Content Servers server.ini to the first FASTdm_ftengine_config object ID.


The xPlore full-text registered events are the defaults: dm_sysobject with the dm_save, dm_checkin, dm_destroy, dm_readonlysave,

dm_move_content events

dm_acl with the dm_save, dm_destroy, dm_saveasnew events

dm_group with the dm_save, dm_destroy events.

If you have customized the dmi_registry for full-text activity for FAST, you must make thecorresponding changes in dmi_registry for new xPlore full-text users.

43


44

Index

Bbackup

overview, 15window, 11

Ddm_ftengine_config objects, 29dm_ftindex_agent_config objects, 30dmfulltext.ini file, 29

Ffull-text index

dmfulltext.ini file, 29full-text indexing

dm_ftengine_config objects, 29dm_ftindex_agent_config objects, 30high-availability deployments, 27location objects, 29properties supporting, 28server.ini file entries, 29

fulltext_location property, 29

Hhigh-availability deployments

default index, 27described, 27failover, 27indexing, 27installing, 30

querying, 27queue items, 27redundancy, 27standby index, 27

Iindex agent

representation in repository, 30index server

representation in repository, 29installation

high-availability deployments, 30

Llocation objects

full-text indexing, 29

Rredundancy

high-availability deployments, 27restore time, 12RPO, 11RTO, 12

Sserver.ini file

full-text indexing entires, 29

45

EMC Documentum xPlore High Availability and Disaster Recovery GuidePrefaceIntended AudienceRevision History

OverviewIntroductionBackup and restore processBackup levels and typesBackup process

Recovery Point Objective (RPO)Restore Time Objective (RTO)Balancing RPO and RTOBackup riskMulti-site disaster recoveryHigh availability and disaster recovery cost

Backup and Disaster RecoveryIntroductionContent Server recovery overviewFigure 1. Backup for Object ConsistencyFigure 2. RPO for disaster recovery site (including data replica

xPlore backup configurationsRestore Content ServerSynchronize the restored dataRestore if content is newerFigure 3. Storage Area Backup is Newer

Restoring if metadata is newerFigure 4. Database Backup is Newer

Restore an index at a particular point in timeFigure 5. Restoring xPlore Instances Decision Tree

Restore data after an instance fails

xPlore High Availability DeploymentsOverviewContent Server full-text objects and initialization filesFigure 6. Full-text Index Objects

Set up Content Server and xPlore HASpare instance HA (standby)Figure 7. xPlore Spare Instance HA

Dual primary instance HA (active-active)Figure 8. Two xPlore Primary Instances HAProcedureEnable full-text queries to be serviced by the secondary xPlore

xPlore dual mode with an existing FAST HA deploymentSetting up xPlore for HA after dual mode has been configured

xPlore HA dual mode with an existing FAST HA deployment

Documents

Docu36066 Documentum XPlore 1.2 High Availability and Disaster Recovery Guide