SPLIT_MIRROR_UPGRADE.pdf

SAP Advanced Technology Group

Minimize Upgrade Downtime – Split Mirror Upgrade (SMU) Based on CBU, ICNV, Split Mirror Solution and ORACLE transportable Tablespace April 2002, Version 1.1

Split Mirror Upgrade (SMU) Version 1.1

SAP – Advanced Technology Group page 2

History:

Version Date Annotation

1.0 06 April 2002 Initial Version

1.1 24 April 2002 Chapter 5.3: Added “Mount Filesystems” paragraphs

© Copyright 2002 SAP AG. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft®, WINDOWS®, NT®, EXCEL®, Word®, PowerPoint® and SQL Server® are registered trademarks of Microsoft Corporation. IBM®, DB2®, OS/2®, DB2/6000®, Parallel Sysplex®, MVS/ESA®, RS/6000®, AIX®, S/390®, AS/400®, OS/390®, and OS/400® are registered trademarks of IBM Corporation. ORACLE® is a registered trademark of ORACLE Corporation. INFORMIX®-OnLine for SAP and Informix® Dynamic Server

TM are registered trademarks of Informix Software Incorporated.

UNIX®, X/Open®, OSF/1®, and Motif® are registered trademarks of the Open Group. HTML, DHTML, XML, XHTML are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology. JAVA® is a registered trademark of Sun Microsystems, Inc. JAVASCRIPT® is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. SAP, SAP Logo, R/2, RIVA, R/3, ABAP, SAP ArchiveLink, SAP Business Workflow, WebFlow, SAP EarlyWatch, BAPI, SAPPHIRE, Management Cockpit, mySAP.com Logo and mySAP.com are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other products mentioned are trademarks or registered trademarks of their respective companies. Disclaimer: SAP AG assumes no responsibility for errors or omissions in these materials. These materials are provided “as is” without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP shall not be liable for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. SAP does not warrant the accuracy or completeness of the information, text, graphics, links or other items contained within these materials. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third party web pages nor provide any warranty whatsoever relating to third party web pages.



Contents:

1 Abstract ...........................................................................................................................4

2 Introduction.....................................................................................................................4

3 Major Downtime Consumer ...........................................................................................5

4 Concepts .........................................................................................................................5

4.1 Split Mirror Upgrade Solution ..................................................................................................... 6

4.2 Split Mirror Solution ........................................................................................................................ 8

4.3 Customer Based Upgrade (CBU) .............................................................................................. 10

4.4 Incremental Data Conversion (ICNV).................................................................................... 12

4.5 Transportable Tablespace........................................................................................................... 18

4.6 Transfer Optimizer Statistics..................................................................................................... 20

5 The Split Mirror Upgrade Implementation ..................................................................22

5.1 SMU System Setup ......................................................................................................................... 22

5.2 SMU Process ..................................................................................................................................... 25 5.2.1 Standard Upgrade....................................................................................................................... 26 5.2.2 Preparation for CBU and Remote Conversion....................................................................... 27 5.2.3 Execution of CBU and Remote Conversion............................................................................ 27 5.2.4 Downtime ..................................................................................................................................... 28

5.3 Complete SMU Phase List............................................................................................................ 29 5.3.1 Split Mirror Copy (COPYNIMP, COPYITTS, COPYETTS).................................................................... 30 5.3.2 SAP Standard Upgrade (SPREPARE, SUPGRADE) ......................................................................... 38 5.3.3 Export ICNV candidates (EXPINCVC).......................................................................................... 38 5.3.4 Customer Based Upgrade (CBUEXREP, CBUPREPA, CBUPGRAD, CBUCOMPL)................................ 39 5.3.5 Import SMU Tool extension (ISMUTOOL) .................................................................................. 40 5.3.6 Incemental Conversion (ICNVSTOR, ICNVSTART, ICNVSETx, ICNVDATA, ICNVDELT) .................... 41 5.3.7 Unplug / Plug transportable Tablespace (UNPLUGTS, PLUGINTS) ........................................... 43 5.3.8 Transfer Optimizer Statistics (STATEXP, STATIMP).................................................................... 45

5.4 System Requirements Scenario ............................................................................................... 46

6 Future Directions..........................................................................................................47

7 References ....................................................................................................................49



1 Abstract This paper describes a recently implemented upgrade method to current SAP 4.6 releases that exploits ORACLE's "transportable tablespace" concept and the instantaneous - Split Mirror - copy capabilities of EMC’s Symmetrix Storage Systems. The storage and DB system based Split Mirror technique allows the cloning of an SAP system of any size within minutes and during this process the users of the production system do not miss a beat. Split Mirror Upgrade (SMU) is an extension of the SAP Customer Based Upgrade (CBU) method, developed to reduce the technical downtime of a standard Upgrade. Beyond CBU, Split Mirror Upgrade offloads data conversion workload and administration workload such as creation of indexes and Database statistics from the production system. Due to this procedure business crucial data will not be changed in the production system. Should the Upgrade not be successful for any reason, production can easily fall back to the original system. The downtime of a Split Mirror Upgrade will be as low as the downtime of a Customer Based Upgrade, but the SMU method offers the chance to further reduce downtime by means of “iterations”. This Split Mirror Upgrade solution [SMU01LA] was presented at SAP's Tech'Ed01, Los Angeles, Nov 6 - 9, 2001 targeted at the worldwide SAP Developer community.

2 Introduction With the increasing need for continuous system availability, minimization of the SAP Upgrade downtime is of paramount importance. Upgrading SAP systems is necessary, on the one hand due to the software lifecycle that will provide new functionality and on the other hand from customers view to fulfill additional business process requirements. For technical reasons there is a downtime that causes costs. Some customers will loose with every hour of downtime a considerable amount of money and therefore they are willing to spend additional resources, such as work force, servers and storage that help to minimize the downtime. Beyond minimization of downtime, SMU pursues the goals to leave business crucial data unchanged as long as possible, to use resources – hardware and work force – more efficient, to use well-known SAP Upgrade methods and to exploit special technologies of the used DB and Storage System. The major contributors of downtime for upgrades to current SAP 4.6 releases are data conversions. SMU offloads the data conversion workload form the production system.

The implemented SMU process starts with a standard SAP Upgrade on a complete copy (“clone”) of the production. The cloning process that has no impact to operation, exploits the DB systems online backup and the storage system instantaneous fast copy capabilities. The standard Upgrade enables testing and accurate planning of the real production upgrade. Beyond this SMU uses the SAP Customer Based Upgrade (CBU) method to export for later use the SAP repository from the successfully upgraded clone of the production.



As preparation of the real production upgrade, SMU initialises in the production system the SAP Incremental Data Conversion (ICNV), which creates target tables – tables with the new 4.6 structure – and their belonging indexes in an ORACLE transportable tablespace. ‘ICNV Initialisation’ also creates DB triggers used to record data modifications. The production system prepared of that kind will be cloned again. SMU will start in the clone the ‘ICNV Data Transfer’ that copies all converted data into the tables stored in the transportable tablespace. In the meantime in production system CBU will be allowed to continue – imports the repository – until the phase that requires production to be down. SMU then unplugs the transportable tablespace from the clone to save the data conversions, creates the clone again – but this time without copying the transportable tablespace – and plugs in again the saved data conversions. Based on the recorded information, ICNV will repeat the data conversions for records changed since the last cloning process and finally CBU will complete the upgrade. After the successful completion of SMU, the production will be started on the upgraded image. This document is mainly structured into the description of Concepts used for Split Mirror Upgrade (SMU) solution and the Implementation of the SMU solution.

3 Major Downtime Consumer

Due to permanent improvements of Upgrade tools and processes, such as mass activation of dictionary objects, pre-generation of ABAP program loads and parallel processing of table and language imports, the downtime was significantly reduced.

Today data conversions caused by structural changes or by application needs are the major consumers of downtime.

Structural changes are for example a new field length or an extension of a record by a new field. The conversions caused by these changes will be handled in phase PCON. Conversions caused by application needs will be executed by specific application programs (XPRA) – for example the value of a field must be changed depending on the content of other fields, from the same table or other tables.

For which table a structural or application change will cost the most conversion time, depends on the table size and is therefore highly customer dependend. In general one can not say in advance which conversion will take a long time, but there will be definitely a few conversions that dominate the downtime.

4 Concepts

We will at first introduce our new Upgrade methodology – the Split Mirror Upgrade – and afterwards the building-blocks as used for the SMU solution.

The building-blocks are based on technologies provided by Storage Systems: ‘Split Mirror Solution’, SAP: ‘Customer Based Upgrade’ and ‘Incremental Data Conversion’ and the ORACLE Database System: ‘Transportable Tablespace’ and ‘Transfer Optimizer Statistics’.



4.1 Split Mirror Upgrade Solution

This solution [SMU01LA] is based on the SAP Customer Based Upgrade process and requires like CBU enough disk storage to be able to store a complete copy of the production system.

Copying the production system is crucial to the SMU solution. The copy process will provide a second instance (‘clone’) of the production and will have no impact to operation. This will be achieved by a combination of the DB system online backup and storage system instantaneous fast copy capabilities (‘Split Mirror Solution’).

On the production copy – in the following this will be named upgrade system – a standard upgrade to the target release 4.6 will be processed. After all SAP modifications, support packages, add-on supplements and customer changes have been successfully applied to the upgrade system, the new 4.6 repository will be exported and saved to be able to use it later for the real production upgrade.

The standard upgrade process will deliver information about all tables that need to be converted due to technical changes. The large tables that will predominate the conversion process will be filtered out as candidates for a pre-downtime conversion. The upgrade tool ICNV (Incremental Data Conversion) was designed for the pre-downtime conversion. In its ‘Initialisation’ phase, ICNV will create for the ‘candidates’ target tables – tables with the new 4.6 structure – and their belonging indexes and will store them in a supplied ORACLE transportable tablespace. DB triggers will be created on the source tables and will immediately be used to record data modifications. During the initialisation phase access to the processed tables will be prevented for a short period.



The production system prepared of that kind will be cloned again. In the upgrade system the ICNV ‘Data Transfer’ phase will be started that copies all converted data into the tables stored in the transportable tablespace. After all data are converted the ‘un-plug’ feature of ORACLE’s transportable tablespace will be used to save the data conversions. In parallel to the conversion process, the SAP upgrade will be started in the production system. The upgrade process will import the repository created in the standard upgrade and continue until the phase that requires production to be down. All user activity will be stopped and the production system will finally be cloned, but this time the volumes that contain the transportable tablespace will be excluded from the copy process. The upgrade process will be completed in the upgrade system. For this the transportable tablespace with the saved converted data will be pluged in again and the upgrade process will be resumed. Based on the recorded modifications, ICNV will repeat the data conversions for records changed since the last cloning process and finally the upgrade will be completed.

At the end of this process the upgraded image will be used to verify the upgrade and if successful it will be released as the new production image. If for any reason the upgrade was not successful, production will be started on the old image. Essential to the SMU process is that the complete R/3 system will be handled as one entity and that the transportable tablespace will be used in the production system only for the creation of the target release table structures. The transportable tablespace in the production system will never contain any data. Consequently it must not be copied to the upgrade system after start of the data transfer to avoid conversions to be overwritten.



4.2 Split Mirror Solution

Making a copy of an SAP R/3 System where the operating and database system are the same on the target, is known in SAP terminology as Homogeneous System Copy [HOMCOPY] and sometimes referred to as a system clone or second instance.

The standard method for performing a Homogeneous System Copy is the use of the SAP R/3 Export/Import tools that export the data from the source database to flat files and then import them into the target database. Especially for large systems, this process may be very time consuming.

The fast copy capabilities of intelligent storage systems will considerably reduce the time needed for the copy process and place the clone much quicker to the user's disposal. There are several publications available that describe system cloning based on storage technologies – please refer to [SECNDINST].

The system cloning process used in the SMU solution will differ from a general cloning process in several aspects:

� The clone will always be managed on a different host. Depending on its size the cloned database can be stored on the same or a different storage system. This setup will allow a complete separation of the production workload from the workload caused by the upgrade.

� The setup must take into consideration that some objects are not always copied. The transportable tablespace will only be copied initially.

� The original system and the clone will be identical. This means the SMU cloning process will copy ORACLE datafiles as well as ORACLE and R/3 executables and control files. This is necessary because after the the successful Split Mirror Upgrade the clone will become the production system. Key system identifiers, such as ‘SAP R/3 system ID’ or ‘ORACLE instance name’ don’t need to be changed.

To be prepared for the system cloning, initially the operating system users <sid>adm and ora<sid> must be created on the second instance host. After that the environment for each user must be copied and in the copied environment the host names must be adjusted.



The following figure depicts the necessary steps to create a second instance – the SMU upgrade system.

Data will only be transferred to the second instance side, if a new copy is required. To minimize any impact to the production system, on storage level all volumes that belong to the R/3 and ORACLE system will at first be (1) synchronized and after completition of this step all tablespaces will be set into (2) ONLINE backup mode. Data modifications are still possible, they will be reflected in the LOG.

On the storage level (3) the relation between the production and second instance volumes – with exception of the LOG volumes – will be split and the (4) ONLINE backup mode for the tablespaces will be brought to an end. In step (5) ORACLE will be forced to perform a LOG switch and after that (6) the relation between the production and second instance LOG volumes will also be split.

The two files listener.ora and tnsnames.ora used by ORACLE to establish communication pathways between user processes and ORACLE contain the name of the DB host. The files reside in the /oracle subdirectory and will be (7) copied with the volume copy process. Therefore after the copy process the DB host name must be (8) adjusted to the name of the second instance host.

After the adjustments (9) ORACLE will be started. This ‘DB restart’ process will apply modifications recorded in the LOGs to the ‘mirrored’ tablespaces and provide a consistent database.

To make sure that the cloned R/3 system does not behave like the original system, (10) some SAP specific cleanup procedures must be processed, such as erasure of scheduled jobs or RFC destinations.



4.3 Customer Based Upgrade (CBU)

SAP provides an extension of the Upgrade suite that certified consultants can use to perform a Customer Based Upgrade [CBUUPG]. The CBU methodology is specially suitable for customers who make their own extensive developments and also for customers who can’t afford a longer downtime of their SAP system.

CBU will perform the time intensive Upgrade workload mostly on a copy of the production system. This means as prerequisite before performing a CBU the customer must provide the same hardware – DB server and Storage – for the upgrade site as used for the production system. The customer must also be aware that after the first copy has been taken, he must ‘freeze’ the production repository. Otherwise any transports and correction that will be made in production will be lost and need to be repeated manually after the CBU.

The production system will be (1) cloned with the non-impact split mirror solution and on the upgrade system (2) a standard upgrade will be performed.

The standard upgrade will import all modification made for the target release, support packages, add-on supplements and customer modifications. After that the modifications will be adjusted, dictionary objects activated and ABAP program loads generated. The upgrade process will reveal critical or long-running database modifications, such as conversions and creation of indexes or DB statistics, that can be analyzed and optimized for the production upgrade.

After the standard upgrade is completed, the repository containing all activated and generated objects will be exported (3) and stored on ‘CDs’ – actually in a directory of the file system.



CBU then recommends to test the exported repository before using it for the real production upgrade. For this the production system will be (4) cloned again and (5) the upgrade will be repeated, but this time the exported repository will be used.

The second upgrade will be much faster, due to the already activated and generated objects, but the database modifications (conversions, index creation) will take the same time as in the first upgrade.

After completition of the second upgrade, (6) the system will be verified. If the upgrade was not successful, for example some customer transports and corrections are still missing, these must be applied and the process will continue with step (3). Again the repository CDs will be created, the production cloned, the upgrade with the new repository repeated and finally verified.

The real production upgrade will be performed with the faultless repository CDs (7). Major contributors to the downtime will be the conversions and creation of indexes and DB statistics.

There are two types of conversions, those executed by an specific application program (XPRA) – for example the value of a field will be changed depending on the content of other fields, from the same table or other tables – and those that can be executed without application knowledge. For the latter conversions (PCON), SAP provides the Incremental Data Conversion (ICNV) that can be processed before the downtime. Due to the runtime information provided in the previous upgrade processes, long running XPRAs can be identified and optimized for example by parallel execution.



4.4 Incremental Data Conversion (ICNV)

The Incremental Data Conversion is a method for online restructuring and reorganization of database tables. The R/3 transaction ICNV will be processed in 4 major phases:

In the Initilization phase ICNV takes the meta data of the objects to be processed from the ABAP Dictionary and creates shadow/target tables with the new 4.6 structure, a flag for every original record that will be used to indicate data changes and database triggers to record modifications.

During Data Transfer phase ICNV reads all records that were not yet processed as well as all modified records, converts them to the new structure and copies them into the shadow tables.

In the Switch phase ICNV disables the access to the original table and processes a last time a data transfer for the modified records. Finally ICNV switches to the shadow table.

In the Delete Entry phase ICNV removes physically the original table and all auxiliary objects such as triggers and indexes from the database.

ICNV – Initialization

The following figure depicts the ICNV initialisation steps as used in the implemented SMU solution.

The shadow table with the new 4.6 structure will be (1) created in the supplied transportable tablespace. Following the ICNV naming convention the name of the shadow table starts with the letters QCM. All indexes for the shadow table will also be (2) created and stored in the transportable tablespace. The indexes are named like the originals, they merely differ in the character used to separate the index suffix (ex. ‘~X’ is changed to ‘_X’).



The creation of the secondary indexes in this step is an enhancement of the standard ICNV.

The standard ICNV will not create the secondary indexes in this step, they will be created in the Switch phase. Since this will increase the downtime, we decided to create the indexes already in this step and build them continuously with the data conversion.

In step (3) views – if there exist any – depending on the original tables will be dropped and the original table will be renamed. Starting from this point in time the table will no longer be available for production processing. The new table name starts with the letters QCM1.

The renamed original table will then be modified and (4) ICNV will add a new field to the table – the indicator flag. The indicator flag is added to every record and will have the initial value NULL.

To be able to record modifications on the original data, ICNV will create in step (5) and (6) database triggers on the QCM1 table.

The UPDATE trigger is executed whenever a field of the original table is modified. This means changes of the indicator flag are not tracked. A modification on a field that does not belong to the primary key, will change the state to ‘U’ which means ‘updated’. A modification on a field of the primary key will cause a DELETE followed by an INSERT operation and the indicator field will get the initial value NULL.

The DELETE trigger is executed whenever a record of the original table is deleted. The trigger will immediately delete the corresponding record from the shadow table.

An INSERT trigger is not necessary because the indicator flag of all new records gets the initial value NULL.

In step (7) ICNV will create a view (QCM2…) on the renamed original table that consists of the primary key fields and the indicatior flag. The view is used to determine the records to be converted and to compute the progress of the Data Transfer phase.

For perfomance optimization ICNV will create (8) an index on the indicator flag.

Finally ICNV will create (9) a view that consists of the fields of the original table. This view is named like the original table and is used by the applications to access the table data. If views depending on the original tables were dropped in step (3), they will now be recreated using the new name of the original table. After this step the SAP applications have again full READ/WRITE access to the data.

SMU – peculiarities:

Delete operations on the target table (QCMtab) that were released by the delete trigger will not be propagated to the upgrade system. Therefore the SMU solution must provide a function that removes deleted records from the upgrade system.



ICNV – Data Transfer

The ICNV data transfer phase will process all records of the original tables, where the indicator flag has the value NULL or ‘U’. NULL has the meaning that the record was inserted or not yet converted. ‘U’ means that one or more fields of the record were modified and therefore needs to be converted again.

The data transfer will read a record from the original table, convert it to the 4.6 target structure and copy it into the target table. The flag of a successfully processed record will get the value ‘C’ (converted).


The data transfer will not be processed in the production system. This means that no information about already converted records will be kept in the production system. Since the production system will be cloned completely (and sets all indicator flags back to NULL), the SMU solution must provide a function that avoids unnecessary repetitions of data transfers.

ICNV – Switch

During ICNV switch the data will not be available – this means DOWNTIME. To keep the switch as short as possible, the data transfer should have progressed quite far (at least 95% for each individual table).

The switch replaces the original table with the target table. The switch includes the following steps:

1. Disable access and lock the original table against further changes to the data.

2. Convert all records where the indicator flag has the value NULL or ‘U’.

3. Replace the virtual original table (view) with the new target table.

4. Reconstruct all views based on the original table.

After the successful switch to the converted table, the original data are still available.


Due to the cloning of the production to the upgrade system, the information that indicates the progress of the data transfer will be reset. Therefore SMU must provide a function that corrects the progress information to allow the succeeding upgrade phases to continue.

ICNV – Delete Enty

The ICNV delete entry phase removes physically the original table and all auxiliary objects such as triggers and indexes from the database.



Solutions for the SMU – peculiarities

The following figure depicts for the conversion indicator flags (CNV_FLAG) the transition of states during modification recording and the state of the original (QCM1tab) and target (QCMtab) table in the upgrade system after the final system cloning.

Avoid unnecessary data transfers

Before the production system will be copied to the upgrade system for the data transfer, the conversion flags of all records will be set to state CONVERTED (‘C’). Immediately after system cloning and before starting the data transfer, all conversion flags will be reset in the upgrade system to their initial value. Then ICNV data transfer will be started in the upgrade system.

With this procedure all records will initially be converted in the upgrade system. In the production system, modification recording will lead to an original table, where the most records are in state CONVERTED (‘C’) and others that are in state UPDATED (‘U’) or INSERTED (NULL). This means that after the final cloning, only UPDATED and INSERTED records will be converted.

Setting the state of the conversion flags during initialisation step (4) ‘Append indicator field’ would increase the time where the original table is not available. Therefore SMU provides a program that fulfills the following requirements:

� During the set flag process the tables are available for full READ/WRITE processing.

� Lock escalation will be avoided and even for very huge tables the process will not become a long running transaction.

� The process can perform many tables in parallel.

� If the resources run short the process can be stopped and restarted at any time without loosing work.

This new SMU program commits work frequently and allows the user to specify how many tables shall be processed in parallel.



Remove deleted records from the upgrade system

Since records will only be deleted in the production system, the data transfer target table may contain after the final system cloning more records than the original table. Information about deleted records are not reflected in any conversion state.

Therefore SMU provides a program that determines records in the target table that do not exist in the original table and removes them from the target table.

Set progress indicator A new SMU program will SET for ALL tables processed by ICNV the progress indicator greater than 95 %, to allow the upgrade to continue immediately with the 'switch' phase.

ICNV integration into Upgrade procedures

The incremental data conversion is fully integrated into the SAP upgrade suite (refer to [CBUUPG]) and the ICNV – software will be imported during the preparation for upgrade in the tool import phases (TOOLIMP). In particular in phase TOOLIMP_CBU the name tables ICNV30L and ICNV31L (the ‘nametab’ contains the table and field definitions) of the target release will be populated with the tables which are supposed to be converted.

In prepare phase ICNVXRQ the candidates for ICNV will be determined by comparing the active nametab (tables: DDNTT and DDNTF) with the target nametab (ICNV30L and ICNV31L). Several filters are executed and tables not passing any filter are deleted from the target nametab. This filter function will discard for example: tables that are pool tables on the start release; tables with names longer than 10 characters if start release < 4.0 or tables that are new in the target release.

In prepare phase ICNVINIT, candidates for ICNV with less than 10.000 rows will be deleted from the target nametab, because the ICNV process will be most efficient for large tables. Finally ICNV copies the remaining candidates into table ICNVCANDY.

In prepare phase ICNVREQ_PRP, it will be checked if table ICNVCANDY is not empty and if candidates are left, the user is asked to call transaction ICNV and select the tables that shall be converted.

Up to this point the process runs without any user interaction. If the user does not call transaction ICNV, the incremental data conversion will not be used.

If the user calls transaction ICNV, all the candidates are displayed and one can decide, which of the preselected tables shall be processed by ICNV. The screen will also show tables that have been modified by the user, and that would be converted during a return to the SAP standard. For these tables, the user can decide, which tables shall return to the standard and which shall be presented again for modification adjustments (SPDD).

After deciding about all tables, the user is brought to the 'real' ICNV transaction. The user can choose to be guided through the necessary steps by an ICNV-assistant. For the upgrade scenario, the initialization and data tranfer phases must be started manually. At each point in time, the transaction ICNV displays the status of the conversion progress for each table. For details about handling ICNV, please refer to the online documentation.



The remaining ICNV steps will be performed by R3up.

In upgrade phase MODPROF_TRANS – the beginning of down time – ICNV checks if enough entries in the tables (at least 95%) have already been converted to make sure, that the downtime will be as short as possible. If the total amount of converted data is less than 95%, the user is notified.

In upgrade phase PRUN_RADISWTC the ICNV-Switch will at first be processed for the tables that belong to the basis system.

All other tables will be switched to the new release in upgrade phase PCON_<rel>. In this phase it will be checked if the structure of the ICNV converted tables were changed in the meantime (the target nametab ICNV30L, ICNV31L will be compared with the currently inactive nametab DDXTT, DDXTF) and if so, they will be converted again, together with non ICNV processed tables.



4.5 Transportable Tablespace

The transportable tablespaces feature [ORACLETTS] – available with Enterprise Edition of Oracle8i (or higher) - can be used to move a subset of an Oracle database and "plug" it in to another Oracle database, essentially moving tablespaces between the databases. The tablespaces being transported can be either dictionary managed or locally managed.

A tablespace or a set of tablespaces are transportable, if they are self-contained. In this context "self-contained" means that there are no references from inside the set of tablespaces pointing outside of the set of tablespaces. For example in a self-contained set of tablespace there is no index that points to a table that is stored in tablespace not member of the set.

To be able to use the transportable tablespace feature a few limitations must be kept:

� The source and target database must be on the same hardware platform. For example, tablespaces can be transported between Sun Solaris Oracle databases, or between Windows NT Oracle databases. However, a tablespace cannot be transported from a Sun Solaris Oracle database to an Windows NT Oracle database.

� The source and target database must use the same character set and national character set.

� A tablespace cannot be transported to a target database in which a tablespace with the same name already exists.

A self-contained set of tablespaces can be transported between ORACLE databases after generation of a transportable tablespace set. A transportable tablespace set consists of datafiles for the set of tablespaces being transported and a file containing structural information (metadata) for the set of tablespaces.

In the SMU solution locally managed tablespaces were used. Locally managed tablespaces track all extent information in the tablespace itself, using bitmaps, resulting in the following benefits:

� Improved concurrency and speed of space operations, as space allocations and deallocations predominantly modify locally managed resources (bitmaps stored in header files) rather than requiring centrally managed resources such as enqueues.

� Improved performance, because recursive operations that are sometimes required during dictionary-managed space allocation are eliminated.

� Simplified space allocation – when the AUTOALLOCATE clause is specified, appropriate extent size is automatically selected.

� Reduced user reliance on the data dictionary because necessary information is stored in file headers and bitmap blocks.

Note:

In the SMU solution the above described transportable tablespace feature will not be used to copy tablespaces from one ORACLE database to another, the feature will rather be used to ‘unplug’ a tablespace from a database and to ‘plug-in’ the tablespace at a later point in time to the same database.



The following figure depicts roughly the steps necessary in the SMU solution to ‘unplug’ and ‘plug-in’ a transportable tablespace.

In step (1) all tablespaces in the set that should be unplugged will be set read-only. Afterwards (2) the Export utility will be invoked with a specification which tablespaces are in the transportable set. Although the Export utility is used, only data dictionary structural information (metadata) for the tablespaces is exported. Hence, this operation goes quickly even for a large tablespace. Using the (3) Import utility the structural information will be intergrated in the ORACLE database.

Be aware that these operations will be executed in the upgrade system, hence setting tablespaces read-only will not impact any operational tasks.



4.6 Transfer Optimizer Statistics

At the end of a successful upgrade process there will be new indexes – especially for the converted tables – with statistic values that are not up-to-date. Without statistic updates the ORACLE cost-based optimizer may choose wrong access paths and consequently the performance of the R/3 applications will be bad.

Usually the statistics will be derived by the ANALYZE command. To reduce the execution time, ANALYZE will mostly be started for R/3 tables with the option ESTIMATE STATISTICS … SAMPLE 20 PERCENT. Even with this option the calculation of up-to-date statistics at the end of the upgrade will take a long time. If the customer decides to update the statistics before start of the R/3 application, this will increase the downtime and if he decides to update the statistics while the R/3 applications are running there will be a severe impact on the system performance.

ORACLE provides the PL/SQL package DBMS_STATS [ORACLESTAT] with a bunch of subroutines that allow the DB administrator to manage optimizer statistics.

To update the statistics for all R/3 tables, the GATHER statistics command can be applied to all R/3 tables (schema: SAPR3). ORACLE is able to process the GATHER statistics task in parallel.

For the SMU we recommend to GATHER statistics with SAMPLE 100 PERCENT for all R/3 tables after the Upgrade system is completely upgraded and ready for CBU export. We will then use other subroutines (as depicted below) of the DBMS_STATS package to save the statistics for later use:



In step (2) we will use subroutine CREATE_STAT_TABLE to create in the SAPR3 schema the table UPG_STAT, capable to store the statistic information.

From the completely upgraded Upgrade system the statistic information for all tables of the SAPR3 schema will be transferred with subroutine EXPORT_SCHEMA_STATS to the supplied UPG_STAT table (step 3). In step (4) table UPG_STAT – data as well as structural information – will be exported into file expstat.dmp and stored in the file system.

At the end of the SMU process, the file expstat.dmp will be imported into table UPG_STAT (step 5). If table UPG_STAT does not exist, ORACLE will create it. Finally in step (6) the statistics will be transferred with subroutine IMPORT_SCHEMA_STATS from table UPG_STAT into ORACLE’s dictionary tables.

Be aware that this procedure – gathering up-to-date statistics, export and import statistics – takes place in the Upgrade system and has no impact on production. Since the time needed to transfer statistics from table UPG_STAT to the dictionary is much less than the time needed to gather statistics from the R/3 tables, this procedure will reduce the downtime.



5 The Split Mirror Upgrade Implementation

The description of the implemented solution is devided in two parts, the setup of the systems used in the SMU and the process itself.

The process description starts with a list of all SMU phases (section 5.3) in their required sequence and should be used as the reference to the sections that contain the details about the SMU phases.

At the end of this chapter we will discuss a possible system requirement scenario for a SMU solution used at a customer site.

5.1 SMU System Setup

The following figure depicts the ‘big picture’ of the setup used for the implemented SMU solution. It will be refined by detailed descriptions of the hardware (server, storage, connection), software and host device configuration.

Following the recommendations in the chapter 4, the upgrade system is absolutely identical – with respect to hardware and software – to the production system. The transportable tablespaces reside in an own BCV TimeFinder [SYMCLITF] device group and can therefore be handled – synchronized, splitted – individually.

To be able to repeat and test the whole SMU solution again and again, the original R/3 production system was saved (‘Golden Copy’) on a separate BCV device group. To prepare a new complete SMU run, we simply synchonized the devices used for the production system with the ‘Golden Copy’. Thanks to this storage system based procedure we saved a lot of time.



Hardware Configuration

Server System Fujitsu-Siemens Prime Power: 16 x 400 MHz CPUs, 8 GB Memory, devideable into 4 partitions (nodes)

Node Used for CPUs Memory us79d0 Production System 4 2 GB us79d2 Upgrade System 4 2 GB us79d1 Development System 4 2 GB

Storage System EMC Symmetrix 8430-73: 8 GB Memory Disk Characteristics Size Number Physical Drive RAID 1 73 GB 96 Hypervolume 6:1 Split

4 parallel IO requests; queue depth of 32 12 GB 6 per phys. drive

Metavolume (#) Striped with depth of 960 KB; Consists of 8 Hypervolumes

96 GB 4 per SAP/ ORACLE system

Connection EMC DS-16: Switched fibre fabric (2 Brocade 16 Port Switches) Switch to Characteristics Number Server-Node Emulex LP8K 4 Storage JNI Fibre Channel Adapters FC64-1063-EMC 8

# This is the best performing and easiest to manage set up. We used this for both systems – production and upgrade.

Software Configuration Type Software Version

Symmetrix Microcode 5566-4027S EMC Solutions Enabler SYMCLI Base (SRDF, Time Finder, Smmetrix Manager)

V4.1-131 Storage

Powerpath – Load Balancing, Path Fail-over 2.0.0_b101 Volume Management EMC Foundation Suite by VERITAS Volume Manager 3.1.1 Operating System Solaris 5.7 64-Bit Database System ORACLE Enterprise Server 8.1.7.0

YO0 – Production Central and DB Instance 4.0B SAP (+) YO0 – Upgrade Central and DB Instance 4.6C

+ The latest versions of SAP executables: DISP+WORK, R3trans and TP must be used, because SMU will transport DYNPROs of target release 4.6C to the 4.0B production system.

Host Device Configuration Device Group

Meta- vol ID

Host Volumes File Systems Size

/dev/vx/rdsk/rootdg/volORACLE /oracle 276 GB/dev/vx/rdsk/rootdg/volOrigLoga /oracle/YO0/origlogA /dev/vx/rdsk/rootdg/volOrigLogb /oracle/YO0/origlogB /dev/vx/rdsk/rootdg/volMirrorLoga /oracle/YO0/MirrorlogA /dev/vx/rdsk/rootdg/volMirrorLogb /oracle/YO0/mirrorlogB

500 MB

/dev/vx/rdsk/rootdg/volUsrSAP /usr/sap 10 GB

rootdg (*) 30

38

40 /dev/vx/rdsk/rootdg/volSAPmnt /sapmnt/YO0 798 MB

Saptranspts (**) 48 /dev/vx/rdsk/SAPTransTS/volTB /oracle/YO0/SAPtranspTS 96 GB

* All 7 host devices of this group will be striped (in 128 KB stripes) on 3 Metavolumes.

** The transportable tablespaces will be stored on 1 Metavolume. Only in case that ALL tables need to be converted, the storage supplied for the transportable tablespaces must be as large as the storage for the /oracle file system. But this is very unlikely and therefore we prepared our systems to be able to convert one third of the whole database into transportable tablespaces.



Supply locally managed tablespaces

As mentioned in section ‘Transportable Tablespace’, we use locally managed tablespaces as transportable tablespaces.

For the SMU we supplied two new locally managed tablespaces – PSAPSSMUD for all tables to be converted and PSAPSSMUI for all indexes based on these tables.

SAPDBA cannot handle the creation of locally managed tablespaces. Nevertheless there are two options to create locally managed tablespaces:

1. Use SAPDBA to create a standard (dictionary managed) tablespace and use then a ORACLE package to convert the tablespace to a locally managed one.

a. Use SAPDBA as user ora<sid> to create the dictionary managed tablespaces PSAPSSMUD and PSAPSSMUI. To make sure that SAPDBA will create both tablespaces in device group ‘SAPtranspTS’, provide the complete path: ‘/oracle/YO0/SAPtranspTS’ for the datafiles.

b. Convert the new tablespaces to locally managed tablespaces:

svrmgrl or sqlplus execute dbms_space_admin.tablespace_migrate_to_local('PSAPSSMUD'); execute dbms_space_admin.tablespace_migrate_to_local('PSAPSSMUI');

2. Create the transportable tablespaces manually in device group ‘SAPtranspTS’.

Data Tablespace:

svrmgrl or sqlplus create tablespace PSAPSSMUD datafile '/oracle/YO0/SAPtranspTS/psapssmud.data1' size 2000m,

'/oracle/YO0/SAPtranspTS/psapssmud.data2' size 2000m, '/oracle/YO0/SAPtranspTS/psapssmud.data3' size 2000m, '/oracle/YO0/SAPtranspTS/psapssmud.data4' size 2000m, '/oracle/YO0/SAPtranspTS/psapssmud.data5' size 2000m

extent management local uniform size 1024k;

Index Tablespace:

svrmgrl or sqlplus create tablespace PSAPSSMUI datafile '/oracle/YO0/SAPtranspTS/psapssmui.data1' size 2000m,

'/oracle/YO0/SAPtranspTS/psapssmui.data2' size 2000m, '/oracle/YO0/SAPtranspTS/psapssmui.data3' size 2000m, '/oracle/YO0/SAPtranspTS/psapssmui.data4' size 2000m, '/oracle/YO0/SAPtranspTS/psapssmui.data5' size 2000m

extent management local uniform size 1024k;

Adjust the R/3 dictionary and define for the SMU tablespaces PSAPSSMUD and PSAPSSMUI the new data class SSMU0 in the dictionary tables DDART, TAORA, IAORA and TSORA.

TABLE DDART SSMU0 USR SSMU Split Mirror Upgrade - transportable Tablespace



5.2 SMU Process

The following paragraphs will concentrate on aspects that are specific to our process.

The implemented process consists of the phases:

1. Standard Upgrade (1) – (5)

2. Preparation for CBU and Remote Conversion (6) – (12)

3. Execution of CBU and Remote Conversion (13) – (16)

4. Downtime (17) – (23)



5.2.1 Standard Upgrade

In step (1) the Split Mirror Solution will provide – without any impact – a second instance of the production system – the upgrade system.

Along with the upgrade CDs, SAP delivers the file CONVTABS.TUL that contains the names of all SAP standard tables that were changed up to the new release. Which of the SAP standard tables are used in the customers installation, will be determined in step (2) by the PREPARE tool in phase CNV_LIST. These table names together with their size will be listed in file TABCONV.LST. This file does not contain tables that might have to be converted due to customer modifications, these information can only be determined after step (3), the standard upgrade.

In step (4) Split Mirror Upgrade will import the new program ICNV2NDI (transport: YS8K000405) and create with programs RSCBUICNTC and ICNV2NDI the transport SAPKUUICNV.

The CBU program RSCBUICNTC was designed to determine all candidates that can be incrementaly converted – including customer modified tables. RSCBUICNTC clears the ‘Nametabs for Incremental Conversion’ (tables: ICNV30L, ICNV31L), reads file CONVTABS.TUL, populates the Nametabs with all tables that can be incrementaly converted and creates the transport SAPKUUICNV that contains the Nametabs.

The Split Mirror Upgrade program ICNV2NDI collects from the SAP repository for all tables named in the ‘Nametabs for Incremental Conversion’ information necessary to create the secondary indexes for these tables. This information will be stored in the tables UPG_DD12L and UPG_DD17S, and added to transport SAPKUUICNV.



After the successful completion of the standard upgrade – including all necessary support packages, add ons supplements and customer modifications – in step (5) the new SAP repository will be exported. To achieve this, the Upgrade Assistant allows the administrator to start ‘R3up with option’: CBU_EXP.

5.2.2 Preparation for CBU and Remote Conversion

Transport SAPKUUICNV will provide all possible ICNV candidates to the production system and in step (6) the ‘PREPARE for the Customer Based Upgrade’ will filter from all ICNV candidates those tables that can be processed before downtime phase PCON. These tables – their names will be stored in table ICNVCANDY – should have for example a certain size or new fieldnames must be less or equal than 8 charactes.

To complete the set of tools that enable the remote ICNV, all new and modified programs will be imported in step (7) (transport: YS8K000403).

During SMU setup, the database administrator supplied ORACLE transportable tablespaces, located in a volume group that can be separately copied by the storage system’s fast copy functions. The names of these tablespaces will be provided to the SAP repository, so that the ICNV generated shadow tables and indexes can be automatically placed into these tablespaces. The storage parameter adjustments will be processed in step (8) with program ICNVSTOR.

Along with all other objects necessary for the incremental conversion, in step (9) the shadow tables will be created with transaction ICNV (phase ‘Initialization’) and at the same time the modification recording will be started. With the Split Mirror Upgrade program ICNVSETF in step (10) the flag used for data modification recording will be set to the value ‘C’ (:= converted). In case that the resources of the production system run short, this program can be repeatedly stopped and restarted at any time.

The non-impact copy step (11) of the Split Mirror Solution will also include the volume group that contains the transportable tablespace. After the successful copy, all flags in the Upgrade System will be reset in step (12) with program ICNVSETF to be prepared for the initial conversion of all records.

At this point the Production System is prepared for the Customer Based Upgrade and the Upgrade System for the Remote Conversion.

5.2.3 Execution of CBU and Remote Conversion

CBU and Remote Conversion will be executed in parallel.

In the remote upgrade system the ICNV data transfer phase will be started in step (14). Transaction ICNV allows to monitor this process and as soon as all records are converted the transportable tablespaces (15) will be unpluged in order to save already converted data.

In the production system R3up will be started in step (13) for a Customer Based Upgrade with a strategy that will minimze the downtime – A_switch or A_on. Both strategies use database log archiving to ensure a database recovery to the current state only with DB system capabilities. But only strategy A_on continues log archiving during the entire upgrade and needs therefore more additional space than A_switch.

We recommend to use strategy A_switch and run an Offline Backup after R3up finished.



The Offline Backup will provide more safety. Consider, logs that were archived on the upgrade site during conversion will be overwritten with the next complete copy of the production system and therefore unusable for future database recoveries. If sufficient storage resources are available, the Offline Backup time can be significantly reduced by a Split Mirror Backup. This means, the offline database will be copied with the storage system’s fast copy functions and on the copy the DB system will be started to take the backup.

Among all the other steps, CBU will import the new repository that was generated in the standard upgrade phase. CBU will be executed till phase MODPROF_TRANS (16) – the point in time where production must be set offline for further upgrade processing.

5.2.4 Downtime

The SAP application as well as the DB system will be stopped (17). The Split Mirror Upgrade will copy in step (18) again the consistent production system to the upgrade site, but this time without the volume group that contains the transportable tablespace.

After the DB system on the upgrade site is up again, in step (19) the transportable tablespace will be plugged in and after the R/3 system is up, in step (20) all records that have no corresponding record in the original tables will be removed with program ICNVDELT from the shadow tables.

Since the last copy of the production system to the upgrade site has overwritten the indicators that show the progress of conversion – they have now their initial values again – in step (21) program ICNVSETP will be executed to correct the progress information for the succeeding CBU phases.

To be able to resume the CBU on the upgrade site, R3up will be started in step (22) with the option to change all references from production (d0) to the upgrade (d2) system. Conseqently CBU will continue with phase CONFCHK_BAS – the phase right after MODPROF_TRANS. CBU will automatically process all remaining phases and finish the upgrade. In Phase PRUN_RADISWTC the incremental conversion will be completed, this means ICNV will determine and convert all records that were changed since the last Split Mirror copy. Finally ICNV will switch the shadow tables to the new tables and delete the tables with the old structures.

When phase MODPROFP_46C is reached the Offline Backup must be started – as a fast Split Mirror Backup, if the resources are available – and after the backup successfully finished the DB system and then the Production System can be started on the upgraded image.



5.3 Complete SMU Phase List

Since Split Mirror Upgrade (SMU) is based on the standard SAP Upgrade [UPGRADE] and Customer Based Upgrade [CBUUPG] processes, their phases will NOT be described in the following list, please refer to the reference chapter and download the current versions of these documents form the SAP service marketplace.

The following list will show the scale of time needed for each SMU phase. The downtime that starts with phase MODPROF_TRANS (16) will be predominated by phase CBUCOMPL (22) and this phase in turn by the time needed for conversions – especially of the records that have been modified since phase COPYITTS (11).

Chp. Phase Description Scale

of User Action

1 5.3.1 COPYNIMP Non-impact copy sec Run cloning procedure

2 5.3.2 SPREPARE Standard Upgrade-PREPARE

3 5.3.2 SUPGRADE Standard Upgrade week

Perform standard Upgrade to 4.6x s. [UPGRADE]

4 5.3.3 EXPICNVC Export all INCV candidates min Import (STMS) and Run RSCBUICNTC, ICNV2NDI

5.3.4 CBUEXREP CBU export repository min R3up with option: CBU_EXP 5

5.3.8 STATEXP Export DB statistics min Run DB script

6 5.3.4 CBUPREPA CBU PREPARE h Perform PREPARE [CBUUPG]

7 5.3.5 ISMUTOOL Import SMU tool extensions min Transport Transact.: STMS

8 5.3.6 ICNVSTOR Change storage parameter sec Run ICNVSTOR

9 5.3.6 ICNVSTART ICNV initialization min Tansa.: ICNV – initialization

10 5.3.6 ICNVSETC Set conversion flag to C h Run ICNVSETF opt: C

11 5.3.1 COPYITTS Clone inclusive tran. Tablesp. sec Run cloning procedure

12 5.3.6 ICNVSETU Reset conversion flag to U h Run ICNVSETF opt: U

13 5.3.4 CBUPGRAD Customer Based Upgrade h Perform UPGRADE [CBUUPG]

14 5.3.6 ICNVDATA ICNV data transfer h Tansa.: ICNV – data transfer

15 5.3.7 UNPLUGTS Unplug transp. tablespace sec Run DB script

16 MODPROF_TRANS

17 STOPPROD Stop the production system min

Isolate central instance and stop all app-servers [CBUUPG]

18 5.3.1 COPYETTS Clone exclusive tran. Tablesp. sec Run cloning procedure

19 5.3.7 PLUGINTS Plug in transp. tablespace sec Run DB script

20 5.3.6 ICNVDELT Remove delete records min Run ICNVDELT

21 5.3.6 ICNVSETP Set progress indicator sec Run ICNVSETP

5.3.4 CBUCOMPL Complete CBU h R3up – change references

5.3.8 STATIMP Import DB statistics min Run DB script 22

5.3.4 CBUBACKUP Offline Backup with Split Tech min Run Split Mirror Backup

23 STARTPRD Start production system min Use upgraded image

The following sections will describe the SMU phases in detail. Phases that belong to the same topic are combined in the same section. Please use the list above to refer to the appropriate section.

The scripts provided are examples and must be adjusted according to the specific customer implementation.



5.3.1 Split Mirror Copy (COPYNIMP, COPYITTS, COPYETTS)

Initialization

1. Create BCV Device Group: ROOTDG

1. Create ROOTDG and add 3 ‘production’ meta devices to it command line production systemsymdg create ROOTDG symld -g ROOTDG add dev 030 symld -g ROOTDG add dev 038 symld -g ROOTDG add dev 040

2. Associate 3 ‘upgrade’ meta devices to ROOTDG command line production systemsymbcv associate dev 0C0 -g ROOTDG symbcv associate dev 0C8 -g ROOTDG symbcv associate dev 0D0 -g ROOTDG

2. Create BCV Device Group: SAPtransTS

1. Create SAPtransTS and add one ‘production’ meta device to it command line production systemsymdg create SAPtransTS symld -g SAPtransTS add dev 048

2. Associate one ‘upgrade’ meta device to SAPtranTS command line production systemsymbcv associate dev 0D8 -g SAPtransTS

3. Initiate a full establish on all the BCV pairs in groups: ROOTDG and SAPtransTS command line production systemsymmir -g ROOTDG establish –full symmir -g SAPtransTS establish –full

Since we do not specify any pairing options, the standard and BCV device pairing algorithm selects the devices in the group such that they are of equal storage size for a given pair. The content of the standard (production) devices are copied to the BCV (upgrade) devices. The BCV devices are not available for host use during the time that they are assigned as a BCV mirror on a standard devices.

4. Split all BCV pairs in device groups: ROOTDG and SAPtransTS command line production systemsymmir -g ROOTDG split –instant symmir -g SAPtransTS split –instant

Operation with the standard device is resumed and any tracks changed from write operations to the standard device are marked. (This is necessary for updating the BCV device if it is re-established with the same standard device at a later time).

The BCV device state is changed to Ready, enabling host access.

In case of storing production and upgrade system on separate Symmetrix systems, SRDF and the appropriate commands [SYMCLISRDF] must be used instead of TimeFinder commands.



5. Import device groups and mount filesystems

a. Get the Veritas Disk Group ID and import the device groups.

command line production systemvxdg list

will produce the following output.

NAME STATE ID ROOTDG Enabled 985448230.1025.us79d0 SAPTransTS Enabled 985450429.1191.us79d0

The original group ROOTDG will be imported with the new name TESTDG, because there can only be one ROOTDG diskgroup in the system. The volume group SAPtransTS can be imported with the original name.

command line upgrade systemvxdg -C -n TESTDG import 985448230.1025.us79d0 vxrecover -g TESTDG –sb vxdg -C import 985450429.1191.us79d0 vxrecover -g SAPtransTS –sb

b. Finally the file systems will be checked and mounted:

Adjust file /etc/vfstab - maintain the same mount parameters for the filesystems as on the production host.

command line upgrade systemfsck /oracle fsck /oracle/YO0/origlogA fsck /oracle/YO0/mirrlogA fsck /oracle/YO0/origlogB fsck /oracle/YO0/mirrlogB fsck /usr/sap fsck /sapmnt/YO0 fsck /oracle/YO0/SAPtranspTS mount -a



c. Adjustments for Upgrade System

1. Create users yo0adm and orayo0

Make sure that both users have the same User-Id, Group-Id and identical home directories as on the production host. Copy user environment files like .cshrc , .dbenv* and *.sapenv and modify the hostname.

command line upgrade systemgroupadd –g 103 sapsys / groupadd –g 104 oper / groupadd –g 105 dba useradd –d /home/yo0adm –g 103 –G 104 105 –m –s /usr/bin/csh –u 101 useradd –d /oracle/YO0 –g 105 –G 104 –s /usr/bin/csh –u 1002

2. Create environment for users yo0adm and orayo0

The homedirectory of user yo0adm will never be copied to the upgrade host, therefore the files for user yo0adm need to be copied to the upgrade host and adjusted there. Files for user orayo0 can be copied and adjusted on production host, because they will be copied with the split mirror procedure.

command line – user orayo0 production systemcd /oracle/YO0 cp .dbenv_us79d0.csh .dbenv_us79d2.csh cp .sapenv_us79d0.csh .sapenv_us79d2.csh vi .dbenv_us79d2.csh / .sapenv_us79d2.csh :1,$ s/us79d0/us79d2/ #replace prod.host with upgrade host

command line – user yo0adm upgrade systemftp us79d0 cd /home/yo0adm get .dbenv_us79d0.csh / get .sapenv_us79d0.csh / get .cshrc cd /home/yo0adm cp .sapenv_us79d0.csh .sapenv_us79d2.csh cp .dbenv_us79d0.csh .dbenv_us79d2.csh vi .dbenv_us79d2.csh / .sapenv_us79d2.csh :1,$ s/us79d0/us79d2/ #replace prod.host with upgrade host

3. Adjust /etc/system

Maintain the same system parameters as on the production host. Reboot the system to activate the settings.

4. Copy SAP profiles

Copy START and INSTANCE profile into the SAP profile directory and adjust hostnames from production to upgrade host.

command line – user yo0adm production systemcdpro cp START_DVEBMGS00_us79d0 START_DVEBMGS00_us79d2 vi START_DVEBMGS00_us79d2 :1,$ s/us79d0/us79d2/ #replace prod.host with upgrade host cp YO0_DVEBMGS00_us79d0 YO0_DVEBMGS00_us79d2 vi YO0_DVEBMGS00_us79d2 :1,$ s/us79d0/us79d2/ #replace prod.host with upgrade host



COPYNIMP – Non-impact Copy Executed on Execution Type Previous Phase Next Phase Both Systems manually SPREPARE

1. Unmount all filesystems on upgrade system

Stop the R3 system and the database and all other processes using a file from the filesystems.

command line – user yo0adm ugrade systemstopsap saposcol -k

command line – user orayo0 ugrade systemlsnrctl stop

command line ugrade systemumount -a

2. Synchronize Production and Upgrade volume groups

Initiate an incremental establish on all the BCV pairs in group ROOTDG:

command line production systemsymmir -g ROOTDG establish symmir -g SAPtransTS establish

Incrementally establishing a BCV pair accomplishes the same as the full establish process, with a major time saving exception: the standard device copies to the BCV device only the new data that was updated on the standard device while the BCV pair was split. Any changed tracks on the BCV are also overwritten by the data on the corresponding tracks on the standard device.

3. Start Production tablespace in ONLINE Backup Mode

svrmgrl or sqlplus production systemALTER TABLESPACE <…> BEGIN BACKUP;

Allows READ/WRITE operations to the production system to continue, while the split operations are performed.

4. Flush filesystem buffers

command line production systemsync sync

5. Split Production from Upgrade volume group: ROOTDG and SAPtransTS

command line production systemsymmir -g ROOTDG split -instant symmir -g SAPtransTS split -instant

After successful completion the BCV device state is changed to Ready, enabling host access.



6. End ONLINE Backup Mode

svrmgrl or sqlplus production systemALTER TABLESPACE <…> END BACKUP;

7. Force Production LOG switch

svrmgrl or sqlplus production systemALTER SYSTEM SWITCH LOGFILE;

8. Mount filesystems

command line upgrade systemvxrecover -g TESTDG -sb vxrecover -g SAPtransTS –sb fsck /oracle fsck /oracle/YO0/origlogA fsck /oracle/YO0/mirrlogA fsck /oracle/YO0/origlogB fsck /oracle/YO0/mirrlogB fsck /usr/sap fsck /sapmnt/YO0 fsck /oracle/YO0/SAPtranspTS mount -a

9. Copy Production Archive LOGs to Upgrade System

In our set up we did not place the archive logs on a separate meta device – they are rather spreaded with all other data on the 3 production meta devices. Although this is not recommended, we did that due to our limited resources to save two meta devices.

Therefore we must copy the LOGs with ftp from the prodution to the upgrade host. Logon on upgrade host as user orayo0 and execute:

command line upgrade systemcd /oracle/YO0/origlogA ftp us79d0 cd /oracle/YO0/origlogA mget *.dbf lcd /oracle/YO0/origlogB cd /oracle/YO0/origlogB mget *.dbf

or determine the last logfile from the Oracle views V$LOG and V$LOGFILE.

sqlplus or svrmgrl production system

select v$logfile.member , v$log.status , v$log.sequence# from v$log , v$logfile where v$log.group# = v$logfile.group#;



This will produce a output like this: MEMBER STATUS SEQUENCE# /oracle/YO0/origlogA/log_g11m1.dbf INACTIVE 577 /oracle/YO0/mirrlogA/log_g11m2.dbf INACTIVE 577 /oracle/YO0/origlogB/log_g12m1.dbf CURRENT 578 /oracle/YO0/mirrlogB/log_g12m2.dbf CURRENT 578 /oracle/YO0/origlogA/log_g13m1.dbf INACTIVE 575 /oracle/YO0/mirrlogA/log_g13m2.dbf INACTIVE 575 /oracle/YO0/origlogB/log_g14m1.dbf INACTIVE 576 /oracle/YO0/mirrlogB/log_g14m2.dbf INACTIVE 576

Take the logfile with the highest sequence number of all numbers lower than the current sequence number. In our example it would be the sequence number 577 and we would copy the file ‘/oracle/YO0/origlogA/log_g11m1.dbf’.

10. Modify listener.ora and tnsnames.ora

These two files used by ORACLE to establish communication pathways between user processes and ORACLE contain the name of the DB host. The files will be copied with the establish/split ROOTDG process and need therefore to be adjusted:

command line – user orayo0 upgrade systemcd /oracle/YO0/817_32/network/admin vi listener.ora :1,$ s/us79d0/us79d2/ #replace prod.host with upgrade host vi tnsnames.ora :1,$ s/us79d0/us79d2/ #replace prod.host with upgrade host

11. Modify SAP DEFAULT.PFL profile and adjust hostnames

command line – user yo0adm upgrade systemcd /sapmnt/YO0/profile vi DEFAULT.PFL :1,$ s/us79d0/us79d2/ #replace prod.host with upgrade host

12. Start ORACLE System

command line – user orayo0 upgrade systemlsnrctl start

command line – user yo0adm upgrade systemstartsap db

13. Clean up SAP System

The system cloning process as described in [SECNDINST] creates a second instance that can be used in parallel to the production system for various purposes. To make sure that the so cloned system does not behave like the production system, the following steps will be executed:

1. adjust the system name in tables TCESYST and TCESYSTT.

2. adjust the correction and transport system (tables: TCECPSTAT, TCEDELI,TADIR, E070, E070L, E071 and TLOCK).

3. disable RFC connections (table: RFCDES).



4. disable all running, released and ready batch jobs (tables: TBTCS, TBTCO)

5. delete all print jobs and their lists (tables: TSP01, TSP02, TSP02F, TSP02L, TSP0E, TSPEVJOB, TST01 and TST03).

6. clean the information for application-buffer synchronization (table: DDLOG).

7. set a system message that the clone is a copy of the production system (tables: TEMSG, TEMSI).

In the SMU solution the clone will become the production system therefore after cloning in phases COPYNIMP and COPYITTS, we merely must disable RFC connections and batch jobs:

� disable RFC (R/3) connections

sqlplus or svrmgrl upgrade system Update sapr3.rfcdes Set RFCDEST = ‘#’||RFCDEST Where RFCTYPE = ‘3’;

� disable all running, released and ready batch jobs

sqlplus or svrmgrl upgrade system Delete from sapr3.tbtcs where (jobname, jobcount) in (select jobname, jobcount from sapr3.tbtco where (status in (‘R’,’S’,’Y’) and not jobname like ‘RDDIMPDP%’) or jobname in (‘SISM_COLL_AND_TRANS_DATA)) ; Update sapr3.tbtco Set status = ‘P’ where (status in (‘R’,’S’,’Y’) and not jobname like ‘RDDIMPDP%’) or jobname in (‘SISM_COLL_AND_TRANS_DATA)) ;



COPYITTS – Clone inclusive transportable Tablespace Executed on Execution Type Previous Phase Next Phase Both Systems Manually ICNVSETC ICNVSETU

You will find the details about these steps in the description of phase COPYNIMP.


2. Synchronize Production and Upgrade volume group: ROOTDG and SAPtransTS

3. Start Production tablespaces in ONLINE Backup Mode


5. Split Production from Upgrade volume group: ROOTDG and SAPtransTS



8. Mount Filesystems




12. Start ORACLE System

13. Clean up SAP System

COPYETTS – Clone exclusive transportable Tablespace Executed on Execution Type Previous Phase Next Phase Both Systems manually STOPPROD PLUGINTS

You will find the details about these steps in the description of phase COPYNIMP. Here we will describe only the differences.

DO NOT SYNCHRONIZE DISK GROUP SAPtransTS, otherwise you will loose all your already converted data.


2. Synchronize Production volume group: ROOTDG

command line production systemsymmir -g ROOTDG establish

3. Start Production tablespace in ONLINE Backup Mode


5. Split Production from Upgrade volume group: ROOTDG

command line production systemsymmir -g ROOTDG



8. Mount Filesystems






5.3.2 SAP Standard Upgrade (SPREPARE, SUPGRADE)

SPREPARE – Standard Upgrade – Prepare Executed on Execution Type Previous Phase Next Phase Upgrade System Standard SAP PREAPARE COPYNIMP SUPGRADE

This phase does not comprise any SMU peculiarities. The preparation phases of a standard SAP upgrade must be processed as described in detail in [UPGRADE].

All phases relevant to ICNV conversion, such as checks for incomplete conversions from previous upgrades and determination of ICNV candidates will be executed during the standard process.

SUPGRADE – Standard Upgrade Executed on Execution Type Previous Phase Next Phase Upgrade System Standard SAP UPGRADE SPREPARE EXPICNVC

This phase does not comprise any SMU peculiarities. R3up will be started to process all standard UPGRADE phases as described in detail in [UPGRADE].

The UPGRADE must be completed with SAP Support Packages, Customer Modifications and Add-On supplements. If ICNV will be used it must be ensured that all conversion have been completed. Missing program loads must be generated with transaction SGEN.

5.3.3 Export ICNV candidates (EXPINCVC)

EXPINCVC – Export all ICNV Candidates Executed on Execution Type Previous Phase Next Phase Upgrade System manually SUPGRADE CBUEXREP

This phase will be processed in two steps, at first the programs (RSCBUICNTC and ICNV2NDI) that collect all relevant information about the ICNV candidates will be imported and then the programs will be executed to provide the information in transport SAPKUUICNV.

1. Add transport YS8K000405 to the transport directory and import the request (transaction: STMS or TP on command line)

2. Start with transaction SE38 programs: RSCBUICNTC and ICNV2NDI

RSCBUICNTC Populates the name tables: ICNV30L, ICNV31L with all tables that can be incrementally converted, creates transport SAPKUUICNV and adds ICNV30L, ICNV31L to the transport.

ICNV2NDI Collects from SAP repository for all tables named ICNV30L, ICNV31L information, necessary to create the secondary indexes for these tables, stores the information in tables: UPG_DD12L, UPG_DD17S and adds them to transport SAPKUUICNV. Program RSCBUICNTC has already released transport SAPKUUICNV. Therefore we have to export the transport again and place the transportfiles to the CBU <refdir> and <expdir> directories.

command line – user yo0adm upgrade systemcd <transportdir>/bin tp export SAPKUUICNV YO0 u1 cp ../cofiles/KUUICNV.SAP <CBU/refdir/cofiles> cp ../data/RUUICNV.SAP <CBU/refdir/data>



5.3.4 Customer Based Upgrade (CBUEXREP, CBUPREPA, CBUPGRAD, CBUCOMPL)

CBUEXREP – CBU Export Repository Executed on Execution Type Previous Phase Next Phase Upgrade System Standard CBU Export EXPINCVC STATEXP

This phase must be processed as described in [CBUUPG], Step 4: Exporting the Substitution Set and Creating New Repository CDs.

As preparation for the repository export, it is required to create an Export Directory <expdir> that will later contain the repository – the CBU CDs. We recommend to place <expdir> not on host devices used by the upgrade system, because these will be overwritten by the cloning process.

Start R3up with option CBU_EXP

command line cd <putexpdir>/bin R3up CBU_EXP upgdir=<expdir>

CBUPREPA – CBU Prepare Executed on Execution Type Previous Phase Next Phase Production System Standard CBU PREPARE STATEXP ISMUTOOL

This phase must be processed as described in [CBUUPG], Step 6: Customer-Based Upgrade - Prepare.

We recommend using the Upgrade Assistant to execute PREPARE. Copy and unpack the assistant from the supplied CDs and verify wheather the correct R3up is used:

command line cd <upgrade directory>/bin R3up check As result the following will be displayed: R/3 UPGRADE CONTROL PROGRAM >>> Customer Based Upgrade <<< R3up prepare

The CBU specific tools will be imported. In phases ICNVXRQ and ICNVINIT the candidates for the incremental data conversion will be determined and in phase ICNVREQ_PRP the user will be asked to call transaction ICNV and select the tables that shall be converted.

Start transaction ICNV.

ICNV will display a list of all candidates the shall be processed by ICNV. Decide which tables shall finally be processed and Save decision.

CBUPGRAD – Customer Based Upgrade Executed on Execution Type Previous Phase Next Phase Production System Standard CBU UPGRADE ICNVSETU ICNVDATA

This phase must be processed as described in [CBUUPG], Step 6: Customer Based Upgrade.



STOPPROD – Stop Production System Executed on Execution Type Previous Phase Next Phase Production System Standard CBU UPGRADE UNPLUGTS COPYETTS

The upgrade process reached successfully phase MODPROF_TRANS. Since we will use upgrade strategy A_switch, R3up will prompt us to:

� make sure that all production work in the SAP System is stopped and that no users are logged on to the system.

� isolate the central instance.

� back up the upgrade directory to make sure that the upgrade can be reset to this point at a later time.

CBUCOMPL – Complete CBU Executed on Execution Type Previous Phase Next Phase Upgrade System Standard CBU UPGRADE ICNVSETP STATIMP

This phase must be processed as described in [CBUUPG], Step 6: Customer Based Upgrade.

Since the upgrade process was executed and interrupted on the production host us79d0 and the final cloning phase copied also the upgrade directory, we must now start R3up with the option that allows us to change all host references from us79d0 to us79d2.

The upgrade will continue with phase CONFCHK_BAS and progress until phase MODPROF_<rel> where it prompts a backup.

CBUBACKUP – Offline Backup with Split Mirror Technology Executed on Execution Type Previous Phase Next Phase Upgrade System manually STATIMP STARTPRD

5.3.5 Import SMU Tool extension (ISMUTOOL)

ISMUTOOL – Import SMU Tool Extensions Executed on Execution Type Previous Phase Next Phase Production System manually CBUPREPA ICNVSTOR

The complete set of programs that enable the remote ICNV will be provided in transport YS8K000403 and YS8K900022.

Add transport YS8K900022 and YS8K000403 to the transport directory and import the request (transaction: STMS or TP on command line).



5.3.6 Incemental Conversion (ICNVSTOR, ICNVSTART, ICNVSETx, ICNVDATA, ICNVDELT)

ICNVSTOR – Change Storage Parameters Executed on Execution Type Previous Phase Next Phase Production System manually ISMUTOOL ICNVSTART

Start with transaction SE38 program: ICNVSTOR

For all tables selected for incremental data conversions the storage parameters will be adjusted, so that the new 4.6 target tables and indexes will be stored in the transportable tablespaces.

ICNVSTART – ICNV Initialization Executed on Execution Type Previous Phase Next Phase Production System manually ICNVSTOR ICNVSETC

Start transaction ICNV.

ICNV will bring you now to the ICNV entry screen. ICNV offers an assistant that will guide the user through the necessary steps.

Start the Assistant and then execute the 1st step: Initialization. Select all offered tables for the initialization process. The progess of the initialization will be displayed in the ‘Worklist’ of the ICNV main screen. Please wait until for all tables the process finished successfully – this means: Prosess = stop, Production = Y and Status = Conversion. If there is for any table an Error, analyze the Log (F9, Spectacles) and fix the problem.

ICNVSETC – Set conversion Flag to C Executed on Execution Type Previous Phase Next Phase Production System manually ICNVSTART COPYITTS

Start with transaction SE38 program: ICNVSETF set parameter New FLAG Value = C.

There are two other parameters that influence the runtime and availability of the tables to be processed. No. parallel processed tables

Specify for how many tables shall be processed in parallel. Be aware that for each process a dialog work process will be used.

Blocksize

Specifies the number of records to be processed between two COMMITs. To avoid deadlocks or timeouts in the production system, do not choose a large size. We set in our tests Blocksize = 1000.

If the resources are not available or the impact on production is too high, the program can be stopped and restarted at any time without loosing work. The program will set only conversion flags that have the value NULL.

Start the next phase after this phase is successfully finished.

Note: If there is for any reason a delay between this and the next phase (Clone inclusive transportable Tablespace), start ICNVSETF again. This will ensure that records inserted during the delay will be converted in the initial conversion phase and consequently reduce the conversion workload during the downtime crucial ICNV switch phase.



ICNVSETU – Set conversion Flag to U Executed on Execution Type Previous Phase Next Phase Upgrade System manually COPYITTS CBUPGRAD

Start with transaction SE38 program: ICNVSETF set parameter New FLAG Value = U.

The parameters No. parallel processed tables and Blocksize can be set according to the hints in phase ICNVSETC. During this phase usually the upgrade system will not be used for anything else, therefore use as much resources (dialog work processes) as available to reduce the runtime and commit less frequently to reduce the DB workload.

In this phase the program will set only conversion flags that have the value ‘C’. Continue with the next phase after this is successfully finished.

ICNVDATA – ICNV Data Transfer Executed on Execution Type Previous Phase Next Phase Upgrade System manually CBUPGRAD UNPLUGTS

Start transaction ICNV and then the Assistant. Choose 2nd step and start the Data Transfer.

To monitor the the progress of the conversions, choose Performance Analysis. For monitoring it is absolutely sufficient just to refresh (F8) the statistics. We do not recommend to use the Compute progress option, because this will stop all conversions, do the computations and then restart the conversions.

The data transfer is successful, if for all tables the conversion process has status ‘Stop’ and is 100 % finished.

ICNVDELT – Remove Deleted Records Executed on Execution Type Previous Phase Next Phase Upgrade System manually PLUGINTS ICNVSETP

Start with transaction SE38 program: ICNVDELT and specify parameter: No. parallel processed tables

Determines how many 'delete programs' shall run in parallel. This parameter should be adjusted to the number of dialog work processes.

Start the next phase after this phase is successfully finished.

ICNVSETP – Set Progress Indicator Executed on Execution Type Previous Phase Next Phase Upgrade System manually ICNVDELT CBUCOMPL

Start with transaction SE38 program: ICNVSETP

Sets for ALL tables processed by ICNV the PROGRESS INDICATOR (in table DDICNVDIST) greater than 95 %, to allow the upgrade to continue.



5.3.7 Unplug / Plug transportable Tablespace (UNPLUGTS, PLUGINTS)

UNPLUGTS - Unplug transportable tablespace Executed on Execution Type Previous Phase Next Phase Upgrade System manually ICNVDATA STOPPROD

The UNPLUGTS phase generates a transportable tablespace set that consists of the tablespaces PSAPSSMUD, PSAPSSMUI and the structural information (metadata) for these tablespaces. It is performed in two steps:

1. Make all tablespaces in the set to be unpluged read-only:

svrmgrl or sqlplus alter tablespace psapssmud read only; alter tablespace psapssmui read only;

2. Invoke the Export utility, specify which tablespaces are in the transportable set and make sure that the metadata will be stored in the file system used for the transportable tablespaces. The file that contains the metdata will be named expdat.dmp:

command line – user orayo0 cd /oracle/YO0/saptranspTS exp transport_tablespace=y tablespaces=PSAPSSMUD,PSAPSSMUI triggers=y

constraints=y grants=y FILE=expdat.dmp

We set the parameter in a way that all triggers, referential integrity constraints and grants will also be exported. When prompted, connect as SYS (or other administrative user) with the SYSDBA system privilege: CONNECT SYS/password AS SYSDBA

PLUGINTS – Plug-in transportable Tablespace Executed on Execution Type Previous Phase Next Phase Upgrade System manually COPYETTS ICNVDELT

The PLUGINTS phase will be executed immediately after cloning the production system. Since the tablespaces can only be pluged in, if there exist no tablespaces with the same name in ORACLE’s dictionary, the tablespaces PSAPSSMUD and PSAPSSMUI must first be discarded from the cloned production system. After that the transportable tablespace will be pluged in and finally set into READ/WRITE mode.

1. Discard tablespaces PSAPSSMUD and PSAPSSMUI from cloned database:

a. Mount the upgrade database, but leave it closed svrmgrl STARTUP MOUNT



b. Alter datafiles OFFLINE alter database datafile '/oracle/YO0/SAPtranspTS/psapssmud.data1' offline drop; alter database datafile '/oracle/YO0/SAPtranspTS/psapssmud.data2' offline drop; alter database datafile '/oracle/YO0/SAPtranspTS/psapssmud.data3' offline drop; alter database datafile '/oracle/YO0/SAPtranspTS/psapssmud.data4' offline drop; alter database datafile '/oracle/YO0/SAPtranspTS/psapssmud.data5' offline drop;

alter database datafile '/oracle/YO0/SAPtranspTS/psapssmui.data1' offline drop; alter database datafile '/oracle/YO0/SAPtranspTS/psapssmui.data2' offline drop; alter database datafile '/oracle/YO0/SAPtranspTS/psapssmui.data3' offline drop; alter database datafile '/oracle/YO0/SAPtranspTS/psapssmui.data4' offline drop; alter database datafile '/oracle/YO0/SAPtranspTS/psapssmui.data5' offline drop;

c. Drop transportable tablespaces alter database open; drop tablespace PSAPSSMUD including contents; drop tablespace PSAPSSMUI including contents;

This will only remove meta-information about the tablespaces in the ORACLE dictionary. The physical datafiles will not be deleted.

2. Plug in the saved PSAPSSMUD and PSAPSSMUI using metadata expdat.dmp:

command line – user orayo0 cd /oracle/YO0/saptranspTS imp parfile=import.sql [import.sql] transport_tablespace=y datafiles= ‘/oracle/YO0/SAPtranspTS/psapssmud.data1', '/oracle/YO0/SAPtranspTS/psapssmud.data2', '/oracle/YO0/SAPtranspTS/psapssmud.data3', '/oracle/YO0/SAPtranspTS/psapssmud.data4', '/oracle/YO0/SAPtranspTS/psapssmud.data5', '/oracle/YO0/SAPtranspTS/psapssmui.data1', '/oracle/YO0/SAPtranspTS/psapssmui.data2', '/oracle/YO0/SAPtranspTS/psapssmui.data3', '/oracle/YO0/SAPtranspTS/psapssmui.data4', '/oracle/YO0/SAPtranspTS/psapssmui.data5' file=expdat.dmp When prompted, connect as SYS (or other administrative user) with the SYSDBA system privilege: CONNECT SYS/password AS SYSDBA

3. Set PSAPSSMUD and PSAPSSMUI into READ/WRITE mode:

svrmgrl alter database open; alter tablespace psapssmud read write; alter tablespace psapssmui read write;



5.3.8 Transfer Optimizer Statistics (STATEXP, STATIMP)

STATEXP – Export Database Statisics Executed on Execution Type Previous Phase Next Phase Upgrade System manually CBUEXREP CBUPREPA

The STATEXP phase gathers optimizer statistics from the upgraded system and exports the information to the filesystem. It is performed in three steps:

1. Gather statistics for the whole SAPR3 schema:

svrmgrl or sqlplus upgrade systemexec dbms_stats.gather_schema_stats( 'sapr3' , NULL , FALSE , 'FOR ALL COLUMNS SIZE 1' , 4 );

This will compute the statistics for the schema SAPR3 in parallel, here we used a parallel degree of 4.

2. Create a table which is capable of holding statistics and export the dictionary statistic information to that table.

svrmgrl or sqlplus upgrade system

exec dbms_stats.create_stat_table('sapr3','UPG_STAT','PSAPSSMUD');

exec dbms_stats.export_schema_stats('sapr3','UPG_STAT','SMU');

This creates the table UPG_STAT in the tablespace PSAPSSMUD. Afterwards we retrieve the statistics for all objects in the schema ‘sapr3’ and store them in the table ‘UPG_STAT’. As a generell rule, we recommend to always supply the parameter STATID (here ‘SMU’) in the DBMS_STATS procedures, otherwise you will gain bad performance.

3. Export statistic table to the filesystem:

command line – yo0adm upgrade systemexp FILE=expstat.dmp ROWS=Y TABLES=UPG_STAT Keep the file expstat.dmp in a filesystem which will never be overwritten by the upgrade or the cloning procedures.

STATIMP – Import Database Statisics Executed on Execution Type Previous Phase Next Phase Upgrade System manually CBUCOMPL CBUBACKUP

The STATIMP phase imports the statistic information gathered and exported in phase STATEXP. It is performed in two steps:

1. Import statistic table from the filesystem:

command line – yo0adm upgrade systemimp FILE=expstat.dmp ROWS=Y TABLES=UPG_STAT This will create the table UPG_STAT in the tablespace PSAPSSMUD. Afterwards the data from file expstat.dmp will be imported.

2. Import statistics into the oracle dictionary.

svrmgrl or sqlplus upgrade system

exec dbms_stats.import_schema_stats('sapr3','UPG_STAT','SMU');

This procedure retrieves statistics for all objects in the schema ‘SAPR3’ from the table 'UPG_STAT' and stores them in the dictionary.



5.4 System Requirements Scenario

In this section we want to point out that there may be a system operation drawback when using the SMU solution.

The following figure depicts the storage requirements of a customer with a large SAP database. Here we assume that the size of the production system is 3.5 TB. Usually the production system is a link of a system chain with two more systems, a development and a quality assurance system.

In this example all three SAP systems are consolidated in one storage system and to be prepared against a lost of the primary computing center, the production system is mirrored to another storage system located in a remote secondary computing center. The mirror will be constanly synchronized and does not require any host computing power for this task.

An additional host is available to take over the work if the primary should fail. Failover procedures are established that allow the primary and secondary host to access either the primary or secondary storage system.

Automated procedures are established that allow to back up the production system in the secondary computing center. Furthermore tapes will be used to create a copy of the production system in a third computing center, based on the recent backup.

This seems to be a complicated environment, but it is not unsusal for customers that have to provide a 24 x 7 service.



In this scenario we assume that there is no spare space left for SMU on the primary storage system and that a second storage and host system will be provided in the primary computing center.

The SMU cloning process will exploit the fast remote copy capability of the storage system and after the successful completition of SMU, production will be started on the upgraded image while the original system will be released on the primary storage system.

Note: If proceeded as described above, the high availability failover and backup procedures have to be adjusted to the new host-storage configuration and – more important – the procedures must be carefully tested – this may be a crucial drawback.

If this is not desired, we have the option to copy the complete upgraded system back to the primary storage system, but despite the fast storage based copy, this will increase the upgrade downtime.

6 Future Directions

Process automation

The SAP Upgrade work done by the SMU solution is mainly based on the standard Upgrade and CBU procedures - as shown in this document – and therefore as automated as these standard procedures. To ease the handling it is recommended to use the SAP Upgrade Assistant.

Automation of the SMU solution means an integration of the few new SMU phases into the standard procedures. The majority of these phases are simple executions of programs in the SAP system. Merely the handling of the transportable tablespaces and the system cloning need more user interaction, but this can be simplified by scripts.

EMC offers a new product, the Symmetrix Data Mobility Manager (SDMM) that allows the user to configure and contol the replication of objects. Furthermore SDMM provides external interfaces to call on the one hand scripts that contol other systems (e.g. a DB system) and on the other hand to allow other systems to call the SDMM replication policies.

In a future project system cloning should be automized by SDMM and it should be investigated how SDMM can be integrated into the SAP upgrade procedures.

Iteration

If the time between system cloning in phase COPYITTS and the stop of production (MODPROF_TRANS) takes long, we may end in worst case in a situation where all original records were changed. This means that after cloning production to the upgrade system, all records must be converted again.

A solution to avoid this could be to iterate again and again the phases: cloning, plug-in tablespace, data transfer and unplug tablespace until the amount of modified records will be negligible.



But due to our process which never propagates the information about converted records from the upgrade system back to production, we will end in a situation as depicted below:

With every iteration more records must be processed on the upgrade site.

To be able to perform iterations we need a process that is able to determine those records that were updated, deleted and inserted since the last system cloning.

There is a new ICNV feature under development that records with DB triggers for UPDATE, DELETE and INSERT operations without modifications of the original table – no indicator field. The keys of the modified records along with information about the kind of modification will be stored in a new table.

In a future project it should be investigated how this new feature can be exploited for a SMU solution with iterations.

Extension to XPRAs

For those XPRAs (eXecution of PRograms After Import) which change in one table the value of a field (or fields) depending on the content of other fields, from the same table or other tables, the described SMU solution can be applied.

The idea is to use ICNV for the table that contains the field(s) to be changed and to modify the XPRA–program so that instead of the original table, the ICNV created QCM… tables will be used.

Since the XPRA itself must be changed, this solution can’t be provided for all possible XPRAs. We think that after the standard upgrade the conversion runtimes should be analysed and ICNV should only be applied to the long running XPRAs.



7 References

[CBUUPG] Customer-Based Upgrade to Release 4.6C SR1/SR2,

http://service.sap.com/instguides (- SAP R/3 - Release 4.6C SR 2)

[HOMCOPY] SAP R/3 Homogeneous System Copy, http://service.sap.com/instguides (- SAP R/3 - Release 4.6C SR 2)

[ORACLESTAT] ORACLE Supplied PL/SQL Packages and Types Reference, DBMS_STATS http://download-west.oracle.com/otndoc/oracle9i/901_doc/appdev.901/a89852/dbms_sta.htm - 999107

[ORACLETTS] ORACLE Database Adminstrator’s Guide, Managing Tablespaces, Transporting Tablespaces Between Databases http://download-west.oracle.com/otndoc/oracle9i/901_doc/server.901/a90117/tspaces.htm#5697

[SMU01LA] Minimize Upgrade Downtime – Split Mirror Upgrade (SMU); TechEd’01 LA, http://service.sap.com/split-mirror (- SAP Upgrade)

[SECNDINST] SAP R/3 Second Instance using ORACLE on UNIX Platforms http://service.sap.com/split-mirror (- System Cloning)

[SYMCLIB] EMC Solution Enabler SYMCLI Base Component, Version 4.3, Product Guide P/N300-000-048

[SYMCLISRDF] EMC Solution Enabler SYMCLI SRDF Component, Version 4.3, Product Guide P/N300-000-049 (Symmetrix Remote Data Facility)

[SYMCLITF] EMC Solution Enabler SYMCLI TimeFinder Component, Version 4.3, Product Guide P/N300-000-050

[UPGRADE] Upgrade to 4.6C Support Release 2, http://service.sap.com/instguides (- SAP R/3 - Release 4.6C SR 2)

http://service.sap.com/instguides


http://download-west.oracle.com/otndoc/oracle9i/901_doc/appdev.901/a89852/dbms_sta.htm#999107



http://download-west.oracle.com/otndoc/oracle9i/901_doc/server.901/a90117/tspaces.htm#5697

http://download-west.oracle.com/otndoc/oracle9i/901_doc/server.901/a90117/tspaces.htm#5697

http://service.sap.com/split-mirror

http://service.sap.com/split-mirror


Documents

SPLIT_MIRROR_UPGRADE.pdf