Checklist for LiveCache Recovery

Checklist for liveCache Recovery liveCache >= 7.4

Best Practice for Solution Management

Version Date: July 2005 The newest version of this Best Practice can always be

obtained through the SAP Solution Manager

Contents Applicability, Goals, and Requirements....................................................................................................2 Best Practice Procedure and Verification .................................................................................................4

Preliminary Tasks ...............................................................................................................................4 Procedure...........................................................................................................................................4

Backup and Recovery for liveCache >= 7.4.................................................................................4 Miscellaneous.............................................................................................................................20

Further Information .................................................................................................................................22

Best Practice: Checklist for liveCache Recovery liveCache >= 7.4

© 2004 SAP AG

2

Applicability, Goals, and Requirements

To ensure that this Best Practice is the one you need, consider the following goals and requirements.

Goal of Using this Service This document contains a condensed set of instructions for a liveCache backup and recovery for all applications using SAP liveCache with the exception of Business Intelligence. The SAP liveCache is used within SAP APO 3.0, 3.1, and mySAP SCM 4.0, 4.1 and 5.0 and also for WFD Server (Software component WFMCORE 200). Some parts may be only relevant for APO/SCM or WFD Server. These sections are marked accordingly.

It uses a checklist approach to describe all steps necessary before going live, during system operation, and immediately after an abnormal termination of liveCache. For detailed information on BI please see note 816730.

Alternative Practices

APO/SCM only In general, this document is a summary of the more comprehensive document APO 3.0A Backup and Recovery. However, in some places this document also provides supplementary information.

WFD Server only This is the standard document for backup and recovery for the WFD Server (Software component WFMCORE 200).

Staff and Skills Requirements The IT department (liveCache and database administrator) leads in the implementation of this Best Practice. Knowledge and experience in the area of database backup and recovery are required.

APO/SCM only Normally, APO exchanges data with one or several SAP R/3 Systems. This more complex landscape of distributed and redundantly stored data has some impact on the recovery procedure for an APO system. Therefore, it is necessary to involve the team in your company that is responsible for the Core Interface (CIF) or the interface to non-SAP systems.

WFD Server only The WFD Server does not use the Core Interface (CIF) and therefore all sections regarding CIF can be ignored.

System Requirements SAP APO 3.0 was originally delivered with liveCache 7.2. Now, liveCache 7.4.2 is available for SAP APO 3.0 – see note 610316.

SAP APO 3.1 and mySAP SCM 4.0 are delivered with liveCache 7.4.

mySAP SCM 4.1 is delivered with liveCache 7.5.

mySAP SCM 5.0 is delivered with liveCache 7.6.

WFD Server (Software component WFMCORE 200) is delivered with liveCache 7.6.


© 2004 SAP AG

3

Duration and Timing Implementing this Best Practice including corresponding tests takes about a week. The main efforts are required in testing.

The best time to apply this Best Practice is during the implementation phase of your mySAP solution.

How to Use this Best Practice Backup and recovery protects against loss of data in case of liveCache crashes. Therefore, it is important to read and execute carefully all steps described in this Best Practice. It is also important to verify the whole backup and recovery procedure before going live during an intensive test phase.


© 2004 SAP AG

4

Best Practice Procedure and Verification

Preliminary Tasks Before performing this Best Practice, ensure that you perform the following preliminary tasks or checks in the system:

• Create a backup concept that meets your general security requirements. LiveCache allows you to save data completely (Save Data) or incrementally (Save Pages), and to save logs. For more information, see the following documents:

– SAP liveCache 7.4 Backup & Recovery and Administration

– MaxDB Library: Database Manager GUI

– MaxDB Library: Database Manager CLI

This document shows you how to complete the necessary actions using the Database Manager GUI (DBMGUI) Version 7.6. All the actions performed in the DBMGUI can also be performed using the Database Manager Command Line Interface (DBMCLI).

Procedure

Backup and Recovery for liveCache >= 7.4

Activities before Going Live Backup Media When you create a liveCache backup, you need to specify backup media. You can specify backup media using the DBMGUI. The backup can be written to a file, to a tape, or in a pipe. Backups on parallel media are supported. Data backups can be transferred directly to external backup tools. For information about this, see SAP Note 119863.


© 2004 SAP AG

5

To define the backup media using the DBMGUI, choose Configuration >> Backup Media

Data Backups It is advisable to back up your data once a day. This may also be an incremental data backup that writes all changes made since the last complete save to the backup media.

Log Backups Log backups copy the content of the log area into version files. These version files are located in a file system and are archived using file system backup or the DBMCLI-command archive_stage.

Set up liveCache to back up the log automatically. This prevents the log volume becoming overloaded. Switch on the automatic log backup in the DBMGUI by choosing Instance >> Set AutoLog on/off and selecting a medium in the upcoming window.


© 2004 SAP AG

6

Backup History Data backup and the associated log backups should last for at least four generations. Keep the backups available for at least four weeks. If you run a restore and the data backup has errors, you can go back to a previous data backup, so long as it and all the log backups that were created since then are still available.

Schedule the Data Backup Transactions DB13 or DB13C support the scheduling of data backups.

liveCache >= 7.6

Call transaction DB13 for the first time via transaction LC10 >> liveCache: Monitoring >> Tools >> DBA Planning Calendar. Choose the liveCache instance LCA and double-click on the action item. Select the desired backup medium and define the Recurrence Pattern in the next window.


© 2004 SAP AG

7

liveCache < 7.6

Use transaction DB13C for the data backups. Select the liveCache connection under Goto >> Local Calendar. Define the liveCache connection with Configuration >> Add System if it’s not yet available. Double-click the Planning Calendar and schedule the action “Complete data backup”. Select the desired backup medium in the next window.


© 2004 SAP AG

8

Use the report RSLVCBACKUP and schedule it in transaction SM36 if transaction DB13C is not available in your system.


© 2004 SAP AG

9

Check and Test Recovery SAP strongly advises that you test the liveCache recovery a number of times in both the test and production systems. You can do this in a number of ways, for example:

APO/SCM only - By simulating a liveCache crash in a system that is not being used:

No transactions open, no users in the system, CIF queues stopped. Before the crash, use consistency check programs (/SAPAPO/OM17, /SAPAPO/CIF_DELTAREPORT3) to check the internal and external data consistency for SAP APO. After the recovery, these reports must show the same consistencies or the same inconsistencies as before the crash.

- By simulating a liveCache crash with a full system load:

Start a background job (MRP run, for example) and generate CIF activities (data exchange between SAP APO and SAP R/3). During this, carry out a “hard” liveCache stop. After the recovery, verify the internal and external APO data using the consistency check programs mentioned above. There are several options for carrying out a “hard” liveCache stop: Pull out the network connection, boot the server, or (for Unix systems) use the command: dbmcli -d <lc-sid> -n <host> -u control,<password> db_clear Do not use the UNIX command “kill -9” to end the liveCache process (“kernel.exe”).

WFD Server only - By simulating a liveCache crash in a system that is not being used:

No transactions open, no users in the system. Before the crash, use consistency check program /SAPAPO/WFM17 to check the data consistency for WFD Server. After the recovery, these reports must show the same consistencies or the same inconsistencies as before the crash.

- By simulating a liveCache crash with a full system load:

Start a background job. During this, carry out a “hard” liveCache stop. After the recovery, verify the data using the consistency check programs mentioned above. There are several options for carrying out a “hard” liveCache stop: Pull out the network connection, boot the server, or (for Unix systems) use the command: dbmcli -d <lc-sid> -n <host> -u control,<password> db_clear Do not use the UNIX command “kill -9” to end the liveCache process (“kernel.exe”).

For both servers: Be sure to test a complete restore run including data and log.

Verify Periodically execute a “verify”. Schedule the action “Check database structure (all objects)” in the DBA Planning Calendar of transaction DB13 or DB13C. The verify checks physical consistency; that is, that data pages are linked in class containers and table trees in liveCache. Backup generations should not be deleted before an intermediate verify has been run successfully.

The “verify” can result in queues due to locks in liveCache. If you have large liveCache instances, SAP recommends that you run the “verify” on a second system. The data backup is imported into the second system from the production system. The verify is then performed in this second system. This


© 2004 SAP AG

10

procedure checks the data backup and consistency in liveCache. For information about this, see SAP Note 521870.

Activities During System Operation

2.2.1 Check the Backups Make daily checks to see whether the required backups have been created successfully. To perform this check, in the DBMGUI choose Information >> Backup History. In the “Result” column for the relevant backup, you should see “OK”:

Also, take this opportunity to check the level of usage of the liveCache instance. If liveCache looks like it is approaching full capacity, you can increase its size online.


© 2004 SAP AG

11

Check the Verify Check when the last verify ran and if it was successful. To view the log, in the DBMGUI choose Check >> Diagnosis Files >> Utility Statements.

Activities After a liveCache Crash If liveCache crashes, you need to determine whether the failure has been caused by disk errors or by other hardware or software problems. LiveCache can be restarted if no disk errors have occurred and the server is available.

Activities after a Crash without Disk Errors After a crash, start the liveCache instance.

APO/SCM only Then activate the cancelled CIF queues that could not be processed during the downtime.


© 2004 SAP AG

12

Start the liveCache Instance After a crash without disk errors, you can use transaction LC10 to bring the liveCache instance back online. To do this, choose Administration >> Operating, and choose pushbutton “Start liveCache”.

All transaction data that had been committed at the time of the crash is available after the restart. Changes caused by transactions that had not been completed at the time of the crash are rolled back.

The system application processes automatically reconnect to liveCache. Use transaction LC10 (not the DBMGUI or DBMCLI) to start liveCache and to inform all the work processes about the liveCache restart.


© 2004 SAP AG

13

You can tell that liveCache has crashed if it does not have state Admin or Online and has not been shut down. To view the shutdown log, choose Check >> Diagnosis Files >> Utility Statements.

If liveCache crashes for unknown reasons, open an OSS message on component BC-DB-LVC.

APO/SCM only

Activate CIF Queues After the liveCache restart the CIF queues have to be activated. The program /SAPAPO/RCIFRESTART can be used to automatically restart all relevant CIF Queues. See note 604053 for details. Alternatively, you can manually restart the CIF queues. Depending on the queue type, use transaction SMQ2 for inbound queues and / or SMQ1 for outbound queues (or use /SAPAPO/CQ, the SCM Queue Manager). You should check the status of the CIF queues in all relevant systems, APO and R/3.

SYSFAILs due to liveCache Crash A liveCache crash may have caused queue entries with status SYSFAIL if the processing of these entries was interrupted by the crash. After liveCache is available again, these transfers can be easily re-executed. The program /SAPAPO/RCIFRESTART will do this automatically. If you manually restart the queues, you should also execute the SYSFAIL entries in SMQ1 / SMQ2 or use the programs RSQOWKEX (outbound) and RSQIWKEX (inbound) to start several queues in parallel. Of course, errors not caused by the liveCache crash will result in status SYSFAIL again.

Online transfer from APO to R/3 If an APO transaction transfers changes from APO to R/3 in online mode it first writes so-called change pointers (i.e. type and keys of orders) into the APO database. A succeeding and asynchronous process reads the corresponding orders from liveCache and transfers them to R/3. If a


© 2004 SAP AG

14

liveCache crash interrupts the APO transaction right after the change pointers have been committed into the APO database the asynchronous process (reading and transferring the data to R/3) cannot be started. Consequently, the change pointers would remain unprocessed in the APO database. The program /SAPAPO/RCIFRESTART will reprocess the entries which were interrupted by the crash. The manual way is to trigger the transfer of such kind of change pointers via transaction /SAPAPO/C5 after a liveCache restart. However, note that this transaction might display a lot of regular change pointers that are stored for the next periodic data transfer to R/3. Often, for instance, SNP is configured to publish its planning results periodically (once a week) rather than online.

If SAP Note 587062 has been applied in your system, you can distinguish in transaction /SAPAPO/C5 between change pointers stored for online transfer from change pointers stored for periodic transfer. Use checkbox ‘Interrupted online-transfers’ in order to select the online change pointers which have not been transferred due to a liveCache crash or a program termination as described in the paragraph above. If SAP Note 587062 is not applied, select the relevant change pointers using a time interval shortly before the liveCache crash for the creation date select-option.

The APO system should not be released until the CIF queues have been processed. Since the connected SAP R/3 Systems normally send data constantly to the SAP APO system, the CIF queues are never empty for a long period of time. However, before you release the APO system, the CIF queues that grew in size during the liveCache downtime should be processed until they are back to a normal size. An indicator for this is that the CIF queue entries only have time stamps after the recovery.

Depending on the size of the CIF queues, CIF queue processing may take several hours. If you ignore the above recommendation and release the APO system before the CIF queues have returned to a normal size, you should note the following:

- The data in the APO system is not concurrent with the R/3 data; it is outdated in comparison with the R/3 data.

- The synchronous online ATP check from a connected SAP R/3 System bypasses the CIF queues and thus does not see any current data in APO. This can lead to incorrect check results.

- If certain APO planning processes were not executed due to the APO downtime, there is a danger that transactions in R/3 will run on an unplanned or obsolete dataset.

It is the responsibility of the systems group operator for APO and the connected SAP R/3 or non-SAP systems to decide when the APO system can be released again.

Activities After a Crash Caused By Disk Errors Disk or disk system errors can have a variety of effects. They can prevent liveCache from restarting. If the structure of individual blocks on the disks is destroyed, it may still be possible to start the liveCache instance. However, data from individual class containers or SQL tables is then no longer accessible.

If the disks of the liveCache instance have errors, data is not lost if a data backup and all associated log backups and the log volumes are available.

How Do You Recognize I/O Errors? You generally notice data volume errors when you start the instance. Check the knldiag file if you cannot start the instance. To find the file, in the DBMGUI choose Check >> Diagnosis Files >> Database Messages.


© 2004 SAP AG

15

Error message –902 initially indicates an I/O error. To find the cause of the I/O error, see the knldiag file. Error messages related to the operating system are logged here. Check in the system to see whether the disk peripherals have reported any errors.

Error messages –9026 “BD Bad data page” or –9028 “BD Bad file” are displayed if individual blocks are destroyed. These errors do not necessarily cause the liveCache instance to crash. If they occur, the application writes a short dump. To find the error message with corresponding data volume, see the log file mentioned above.

Prepare for a liveCache Recovery Recovery usually takes quite a while. Therefore, it is advisable to make a few preparations in the APO system:

Ask all users in the system to log off as soon as possible. You can inform the users about the recovery using a system message (transaction SM02).

Prevent others from logging on to the system. To do this, use report /SAPAPO/OM_LOCKUSER to lock all users.

Deschedule all programs scheduled to run in the background, particularly planning runs.

APO/SCM only As of SAP APO SP 17, you can perform all these activities in transaction /SAPAPO/OM17. Simply use the first two pushbuttons on the initial screen of the transaction.

If you have activated inbound queues in SAP APO, you can continue to transfer data from the R/3 outbound queue to the APO inbound queue during the liveCache downtime. The APO inbound queue grows accordingly. If several SAP R/3 Systems are connected to a central APO system, the APO inbound queue can grow very rapidly. To prevent an overload of the APO inbound queue, it may be useful to use transaction “SMQ1” to stop the data transfer from SAP R/3 Systems to APO. (Via “SMQ1” the R/3 outbound queues are stopped.) This means that the data quantity is distributed over the outbound queues of the connected SAP R/3 Systems.

In order to avoid that during the liveCache downtime the CIF tries to book permanently data into APO it makes sense to stop additionally the APO inbound queue via transaction “SMQ2”.

SAP Note 505304 describes how to configure database space for stopped CIF queues.


© 2004 SAP AG

16

These considerations also apply if non-SAP systems are connected to APO.

Data and Log Recovery Change the defective disk. Use the DBMGUI to start the liveCache instance in state Cold or Admin. Close the result window and choose Action >> Admin or click the yellow traffic light.

Then start the Recovery Wizard by choosing Recovery >> Recovery Wizard.


© 2004 SAP AG

17

Do not mark “Initialize database instance before restore complete data backup” in the next menu.

If the most recent data backup is unusable, you can select “Restore a specified backup from history” to select an older data backup otherwise select “Restore last backup”.

The Recovery Wizard displays all the backups required. Log backups are only required for log entries that are no longer in the log volumes. Therefore, it is possible that fewer log backups are displayed here than have been created in the meantime.

Check the recovery strategy displayed by the Recovery Wizard, and then choose “Start” to continue. Start the restore for the data backup. If the log backups have been written to a file system and no longer exist there, they can be retrieved during the restoration of data.


© 2004 SAP AG

18

After the data is restored, start restoration of log backups. The Recovery Wizard now recovers the log entries of all displayed log backups, if the backups are available. It will ask you to provide log backups or change the file name of the media if not all the log backups could be retrieved.

APO/SCM only Do not select “Restore database until a specified time.” If the liveCache instance is not completely recovered, inconsistencies might occur between the APO database and liveCache. SAP does not support synchronization using time stamps.

WFD Server only Do not select “Restore database until a specified time.” This might lead to inconsistencies between liveCache and other data of WFD which are not stored in liveCache. SAP does not support synchronization using time stamps.

The recovery time depends on the hardware speed and the size of the backups. If online transactions are running in parallel, log entries can also be recovered in parallel. Therefore, the load distribution of the application also influences the recovery time.

The liveCache instance is automatically set to state Online once all the log entries have been processed. There can be a delay displaying the last log backups when the log entries are read from the log volumes and recovered.


© 2004 SAP AG

19

Follow-Up Work for a liveCache Recovery

APO/SCM only After liveCache recovery, activate the CIF queues as described in section 2.3.1. In addition, perform the following tasks:

- Remove system messages relating to liveCache recovery (SM02)

- Create an SAP support notification if you are unable to determine the reason for the disk failure

- Start/schedule background jobs and planning runs that were descheduled (as described in section 2.3.2) or that did not start due to the downtime (you can do this in transaction /SAPAPO/OM17)

- Unlock the APO users (report /SAPAPO/OM_LOCKUSER) (you can do this in transaction /SAPAPO/OM17)

- Unlock the APO system for the users

WFD Server only After liveCache recovery perform the following tasks:

- Remove system messages relating to liveCache recovery (SM02)

- Create an SAP support notification if you are unable to determine the reason for the disk failure

- Start/schedule background jobs and planning runs that were descheduled (as described in section 2.3.2) or that did not start due to the downtime (you can do this in transaction /SAPAPO/WFM17)


© 2004 SAP AG

20

- Unlock the APO users (report /SAPAPO/OM_LOCKUSER) (you can do this in transaction /SAPAPO/WFM17)

- Unlock the system for the users

Miscellaneous

Log Volume Loss Use appropriate safety measures to ensure that the log volumes are not lost. Log volumes must be mirrored in production systems. If the log volume with the current write position is lost, it is not possible to recover all the transaction entries during the log recovery. In this case, data is lost from liveCache.

With 7.4.03 and newer versions liveCache supports mirroring of the log volumes. The liveCache instance will shutdown when it detects an error during an I/O to a log volume. Check the file knldiag and find the corrupted log volume. To find the file, in the DBMGUI choose Check >> Diagnosis Files >> Database Messages.

Change the defective disk and restore the corrupted log volume. Use the DBMCLI command “util_execute RESTORE LOG VOLUME ‘<file name of the corrupted log volume>’”. Select Instance >> Command Line to get a dbmcli command line in the DBMGUI.


© 2004 SAP AG

21

Restart the liveCache with transaction LC10.

APO/SCM only If you do lose the log volumes that are currently being written although they are mirrored, you should initialize liveCache using transaction LC10. You then have to transfer all transaction data from the connected OLTP systems to the APO system in initial status.

Alternatively, you can reinstall liveCache from the DBMGUI by choosing Instance >> Create. Afterwards, you can import a data backup and additional log backups, if available. LiveCache data backups are consistent because they contain the “before” images of the open transactions.

However, it is likely that data inconsistencies will occur because the most recently made changes are missing from liveCache; that is, the data in the APO database is not completely consistent with the data in liveCache or the data in the connected OLTP systems. To check the internal consistency of the SAP APO system, you can use transaction /SAPAPO/OM17. To check its external consistency, you can use report /SAPAPO/CIF_DELTAREPORT3. If you are using SAP APO 3.0, see SAP Note 425825.

In the case of log volume loss, SAP is unable to guarantee data consistency because data has been lost.

WFD Server only If you do lose the log volumes that are currently being written although they are mirrored, you should initialize liveCache using transaction LC10. Afterwards you can reload the data into liveCache with the program /SAPAPO/WFD_ASSIGNMENT_SYNC_LC.


© 2004 SAP AG

22

Further Information

Troubleshooting If executing this Best Practice did not produce the desired results, please create a support notification describing your problem in detail.

Background Information and References All relevant information and references are given in the sections above.

Feedback and Questions The existing documents will be regularly extended and improved to make them more usable and customer-friendly.

We would therefore welcome your feedback.

If you would like to give us feedback on a particular document, proceed as follows:

Click on the corresponding document from our homepage in SAP Service Marketplace.

Click on the button "Feedback" above the document page. In the field "What is your feedback related to?” select "Contents".

Send us your feedback.


© 2004 SAP AG

23

© Copyright 2005, 2002 SAP AG. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft®, WINDOWS®, NT®, EXCEL®, Word®, PowerPoint® and SQL Server® are registered trademarks of Microsoft Corporation. IBM®, DB2®, OS/2®, DB2/6000®, Parallel Sysplex®, MVS/ESA®, RS/6000®, AIX®, S/390®, AS/400®, OS/390®, and OS/400® are registered trademarks of IBM Corporation. ORACLE® is a registered trademark of ORACLE Corporation. INFORMIX®-OnLine for SAP and Informix® Dynamic Server

TM are registered trademarks of Informix Software

Incorporated. UNIX®, X/Open®, OSF/1®, and Motif® are registered trademarks of the Open Group. HTML, DHTML, XML, XHTML are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology. JAVA® is a registered trademark of Sun Microsystems, Inc. JAVASCRIPT® is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. SAP, SAP Logo, R/2, RIVA, R/3, ABAP, SAP ArchiveLink, SAP Business Workflow, WebFlow, SAP EarlyWatch, BAPI, SAPPHIRE, Management Cockpit, mySAP.com Logo and mySAP.com are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other products mentioned are trademarks or registered trademarks of their respective companies. Disclaimer: SAP AG assumes no responsibility for errors or omissions in these materials. These materials are provided “as is” without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP shall not be liable for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. SAP does not warrant the accuracy or completeness of the information, text, graphics, links or other items contained within these materials. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third party Web pages nor provide any warranty whatsoever relating to third party Web pages.

Documents

Checklist for LiveCache Recovery