Landscape Virtualization Management

IBM SAP International Competence Center

Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 1 of 169

IBM SAP Technical Brief

Implementing and Using

SAP NetWeaver Landscape Virtualization Management

on IBM Power Systems and IBM PureFlex Power-nodes


Walldorf, Germany

Version: 1.0

Status: July 2012



Preface Edition Notice (July 2012)

SAP NetWeaver Landscape Virtualization Management software (SAP LVM) is the

successor and extension of the SAP Adaptive Computing Controller. This document is

complement to the ISICC Implementation Guide “SAP‟s Adaptive Computing on Power

Systems”, Version 3.0, August 2011. The paper focuses on the new features and uses-cases

provided with SAP LVM in conjunction with IBM Systems Director managed Power

Systems. It describes the IBM infrastructure specifics required to implement and operate an

on-premise Cloud Solution for SAP landscapes.

The cookbook covers a solution stack tested during a Proof-of-Concept at the ISICC. This

includes SAP Business Suite 7 with IBM DB2 UDB LUW 9.7 database on IBM Power

Systems using AIX 6 and AIX 7 operating systems, IBM Systems Director, IBM Tivoli

Storage Flashcopy Manager and IBM XIV Storage System. At the time of writing, SAP LVM

V1.0 SP3 is generally available.

Scope and Audience

People reading this document should understand core components and pre-requisites of the

SAP and Systems Software solution stack in an IBM POWER environment including their

installation and handling. They should understand the typical use-cases and their execution

and effects on a POWER/AIX based SAP infrastructure.

The document is intended for IBM (Practitioners from Pre- & Post-Sales and Services,

GBS/GTS, ATS, eTS,) and customer personnel involved in the implementation of SAP

Landscape Virtualization Management software on IBM Power Systems and IBM Systems

Director. Since we do not repeat existing documentation and technical information from both

IBM and SAP the readers are expected to have a solid knowledge in AIX, PowerVM

virtualization, IBM Systems Director and its plug-ins, FlexSystems Manager, IBM Tivoli

Storage FlashCopy Manager and SAP Basis.

Authors

Dr. Edmund Haefele, IBM Technical Sales for SAP

Dr. Thomas Hebert, IBM Technical Sales for SAP

Maik Gasterstaedt, IBM SAP International Competence Center

Walter Orb, IBM SAP International Competence Center

Matthias Koechl, IBM SAP International Competence Center

Arnold Beilmann, IBM R&D

Feedback

We are interested in any feedback you have. Please send your comments to

[email protected].

Disclaimer

This document is subject to change without notification and will not comprehensively cover

the issues encountered in every customer situation. It should be used only in conjunction with

the product literature accompanying the products listed above. The information contained in

this document has not been submitted to any formal IBM test and is distributed AS IS.

mailto:[email protected]



1 INTRODUCTION ....................................................................................................................................... 9

1.1 BACKGROUND AND BUSINESS CONTEXT .............................................................................................. 9 1.2 SCOPE OF THIS COOKBOOK ................................................................................................................... 9 1.3 SAP NETWEAVER LANDSCAPE VIRTUALIZATION MANAGEMENT SOFTWARE (SAP LVM) ............... 10

2 OVERVIEW AND PLANNING ............................................................................................................... 13

2.1 ARCHITECTURE OVERVIEW ................................................................................................................. 13 2.2 COMBINING THE IBM AND SAP LAYERS ............................................................................................ 15 2.3 PLANNING PROJECT RESOURCES AND TIMING .................................................................................... 20 2.4 TESTED CONFIGURATION .................................................................................................................... 22 2.5 SAP LVM ENVIRONMENT .................................................................................................................. 24

3 INSTALLATION ...................................................................................................................................... 32

3.1 NAMING CONVENTIONS ...................................................................................................................... 33 3.2 INSTALL STORAGE INFRASTRUCTURE ................................................................................................. 34 3.3 INSTALL MANAGEMENT ENVIRONMENT ............................................................................................. 44 3.4 INSTALL MANAGED ENVIRONMENTS .................................................................................................. 52

4 INITIAL LVM CONFIGURATION ....................................................................................................... 71

4.1 ENGINE SETTINGS ............................................................................................................................... 71 4.2 STORAGE MANAGER CONFIGURATION ............................................................................................... 71 4.3 DISCOVERY OF RESOURCES (LPARS) ................................................................................................. 74 4.4 SERVICE CONFIGURATION/ DISCOVERY IN SAP LVM ........................................................................ 75 4.5 SETUP FOR THE ENVIRONMENT IN IBM SYSTEMS DIRECTOR .............................................................. 92 4.6 VIRTUALIZATION MANAGER CONFIGURATION IN SAP LVM ........................................................... 104 4.7 ADDITIONAL CONFIGURATION STEPS ............................................................................................... 106

5 DETAILED DESCRIPTION / EXECUTION OF USE CASES ......................................................... 109

5.1 LVM STANDARD EDITION USE CASES ............................................................................................. 109 5.2 SAP LVM ENTERPRISE EDITION USE CASES .................................................................................... 126

6 TROUBLESHOOTING HINTS AND TRICKS ................................................................................... 165

6.1 PHASE “PREPARE CLONE VOLUMES” ................................................................................................ 165 6.2 CLEANUP ACTIVITIES ON A RESOURCE LPAR ................................................................................... 165 6.3 CLEAN-UP ORPHANED VOLUMES IN THE IBM XIV ........................................................................... 166 6.4 DELETE THE SYSTEM IN SAP LVM .................................................................................................. 166

7 RESOURCES ........................................................................................................................................... 167

7.1 IBM DOCUMENTATION ..................................................................................................................... 167 7.2 SAP DOCUMENTATION AND SAP NOTES.......................................................................................... 167



List of figures

Figure 1 Logical connections of IBM and SAP components ..................................................... 9

Figure 2 Generic SAP NetWeaver LVM capabilities .............................................................. 11

Figure 3 SAP LVM Block Diagram (Source: SAP) ................................................................ 13

Figure 4 IBM Systems Director and Plug-Ins .......................................................................... 14

Figure 5 Software structure to integrate IBM IT layers ........................................................... 16

Figure 6 Generic SAP System Copy Workflow ....................................................................... 17

Figure 7 Architecture overview diagram ................................................................................. 23

Figure 8 redundant virtual fibre channel attachment................................................................ 28

Figure 9 Service configuration in SAP LVM with distributed file system layout ................... 34

Figure 10 Example configuration of an SAP system using GPFS ........................................... 39

Figure 11 Volume Group and Filesystem layout ..................................................................... 41

Figure 12 Volumes of SAP system CL6 .................................................................................. 43

Figure 13 SAP System CL6 in LVM and mount points .......................................................... 44

Figure 14 Overview of the FlashCopy Manager installation ................................................... 46

Figure 15 Sample LDAP Directory Information Tree for AIX user management .................. 49

Figure 16 Start IPSec ................................................................................................................ 55

Figure 17 Define the Resource as “Isolation Ready” in SAP LVM ........................................ 56

Figure 18 Configuration of host_profile .................................................................................. 59

Figure 19 Content of .conf file FCMCLI.conf ......................................................................... 59

Figure 20 Content of .conf file XCLI.conf ............................................................................... 60

Figure 21 LVM Engine Settings .............................................................................................. 71

Figure 22 Storage Manager Configuration entry panel ............................................................ 72

Figure 23 Storage Manager Configuration: Select Storage Manager type .............................. 72

Figure 24 Storage Manager Configuration: Define FlashCopy Manager settings ................... 73

Figure 25 Storage Manager Configuration: user and password ............................................... 73

Figure 26 Storage Manager Configuration Summary .............................................................. 74

Figure 27 Add resource – find Resources ................................................................................ 74

Figure 28 Add resource - assign pool ....................................................................................... 75

Figure 29 Add resource – define properties ............................................................................. 75

Figure 30 Add new SAP services ............................................................................................. 76

Figure 31 Expert Mode: Detect new services using host agent and instance agent ................. 77

Figure 32 Add Services View .................................................................................................. 77

Figure 33 Selection of the Resource Pool ................................................................................ 78

Figure 34 Enter SAP System Description ................................................................................ 78

Figure 35 Add Services sub-menu ........................................................................................... 78

Figure 36 Summary view for the new system .......................................................................... 79

Figure 37 SAP System configuration ....................................................................................... 79

Figure 38 SAP System configuration: general settings ............................................................ 80

Figure 39 SAP System configuration. Overall system details ................................................. 80

Figure 40 SAP system configuration: ACM ............................................................................ 81

Figure 41 SAP System configuration: Network fencing defaults ............................................ 81

Figure 42 SAP System configuration: Add allowed communications ..................................... 82

Figure 43 SAP System configuration: Add allowed communications ..................................... 82

Figure 44 DB Configuration: Entry screen .............................................................................. 83

Figure 45 DB Configuration: Basic configuration ................................................................... 83

Figure 46 DB Configuration: Add resource details ................................................................. 84

Figure 47 SAP LVM Storage Definition Screen ...................................................................... 85

Figure 48 Create a list of the DB filesystems ........................................................................... 85

Figure 49 Create XML-File for mountconfig .......................................................................... 86



Figure 50 Paste Mountconfig in LVM storage config ............................................................. 86

Figure 51 Extract Mount Points ............................................................................................... 87

Figure 52 Final LVM Mount point configuration .................................................................... 87

Figure 53 Service Details Central Instance .............................................................................. 88

Figure 54 Basic Configuration for Central Instance ................................................................ 88

Figure 55 Service properties for Central Instance .................................................................... 89

Figure 56 Mount Config for Central Instance .......................................................................... 90

Figure 57 Create List of filesystems for central instance ......................................................... 90

Figure 58 Extract Mount Config for Central Instance ............................................................. 91

Figure 59 Transfer Mount Config to SAP LVM ...................................................................... 91

Figure 60 Apply Mount Config to SAP LVM ......................................................................... 92

Figure 61 HMC Discovery in IBM Systems Director .............................................................. 93

Figure 62 Request access to the HMC ..................................................................................... 93

Figure 63 HMC inventory collection ....................................................................................... 94

Figure 64 View HMC inventory .............................................................................................. 94

Figure 65 View VIO server inventory ...................................................................................... 95

Figure 66 View NIM server inventory ..................................................................................... 96

Figure 67 Remove NIM server ................................................................................................. 97

Figure 68 Discover NIM server ............................................................................................... 97

Figure 69 Collect Inventory of NIM server.............................................................................. 98

Figure 70 Installation of NIM subagent on the NIM server with VMControl ......................... 99

Figure 71 Export mksysb directory .......................................................................................... 99

Figure 72 Trigger of Virtual Appliance Deployment ............................................................. 101

Figure 73 Setup of IBM Systems Director Security – Authorize Groups .............................. 102

Figure 74 Setup of IBM Systems Director Security – Copy Predefined Roles ..................... 102

Figure 75 Setup of IBM Systems Director Security – Modify Copied Roles ........................ 103

Figure 76 Setup of IBM Systems Director Security – Define Resource Groups ................... 103

Figure 77 Setup of IBM Systems Director Security – Assign Roles to User ......................... 104

Figure 78 Setup of IBM Systems Director Security – Assign Groups and Roles to User ..... 104

Figure 79 Virtualization Manager Configuration ................................................................... 105

Figure 80 Virtualization Manager: IBM Systems Director VMControl ................................ 105

Figure 81 Virtualization Manager properties ......................................................................... 106

Figure 82 Virtualization manager summary ........................................................................... 106

Figure 83 Configure Network ................................................................................................ 107

Figure 84 Customizing SAPInst rename path ........................................................................ 108

Figure 85 IBM PowerVM based relocation ........................................................................... 111

Figure 86 SAP adaptive based relocation .............................................................................. 111

Figure 87 Prepare SAP instance ............................................................................................. 113

Figure 88 Start SAP system as mass operation ...................................................................... 114

Figure 89 Logs of the 'Start SAP system' operation ............................................................... 114

Figure 90 Relocation of SAP system as mass operation ........................................................ 115

Figure 91 Stop LPAR ............................................................................................................. 116

Figure 92 Dashboard in SAP LVM ........................................................................................ 117

Figure 93 Infrastructure Visualization in SAP LVM ............................................................. 118

Figure 94 Performance monitoring in SAP LVM .................................................................. 119

Figure 95 OS Provisioning – Template Selection .................................................................. 121

Figure 96 OS Provisioning – LPAR Name Definition ........................................................... 121

Figure 97 OS Provisioning – Target Server Selection ........................................................... 122

Figure 98 OS Provisioning – Storage Pool Selection ............................................................ 122

Figure 99 Provisioning – Definition of Provisioning Parameters .......................................... 123

Figure 100 Provisioning – Final Parameter Confirmation ..................................................... 125



Figure 101 OS Provisioning – Process Monitoring ............................................................... 125

Figure 102 SAP System Copy scenarios supported by SAP LVM (Source SAP) ................. 126

Figure 103 Enable the cloned System for SAP System Rename ........................................... 128

Figure 104 Invoke SAP System Clone ................................................................................... 130

Figure 105 SAP System Clone: Basic data for the target system .......................................... 130

Figure 106 SAP System Clone: Resource selection ............................................................... 131

Figure 107 SAP System Clone: Network definition .............................................................. 133

Figure 108 SAP System Clone: Storage Definitions ............................................................. 134

Figure 109 SAP System Clone: Network Fencing ................................................................. 135

Figure 110 SAP System Clone: Final parameter confirmation .............................................. 135

Figure 111 Define RFC destination for PCA execution ........................................................ 137

Figure 112 SAP System Copy – Basic data ........................................................................... 138

Figure 113 SAP System Copy - Resources ............................................................................ 138

Figure 114 SAP System Copy – Network settings ................................................................ 139

Figure 115 SAP System Copy – Instance Number ................................................................ 139

Figure 116 SAP System Copy – Storage Definition .............................................................. 140

Figure 117 SAP System Copy – User definitions .................................................................. 141

Figure 118 SAP System Copy – Rename SAP system ID ..................................................... 141

Figure 119 SAP System Copy – Network fencing ................................................................. 142

Figure 120 SAP System Copy - Select RFC destination for PCA ......................................... 142

Figure 121 SAP System Copy – Summary view ................................................................... 143

Figure 122 SAP System Refresh: Start system refresh .......................................................... 143

Figure 123 SAP System Refresh: Basic Data ........................................................................ 144

Figure 124 SAP System Refresh: Resource Data .................................................................. 144

Figure 125 SAP System Refresh: Host Names ...................................................................... 145

Figure 126 SAP System Refresh: Storage Settings ................................................................ 145

Figure 127 SAP System Refresh: Users ................................................................................. 146

Figure 128 SAP System Refresh: DB Schema ....................................................................... 146

Figure 129 SAP System Refresh: Network Isolation ............................................................. 147

Figure 130 SAP System Refresh: PCA settings ..................................................................... 147

Figure 131 SAP System Refresh: Parameter summary .......................................................... 148

Figure 132 Start SAP System Rename ................................................................................... 148

Figure 133 SAP system Rename: Basic Data ........................................................................ 149

Figure 134 SAP System Rename: Virtual Hostnames ........................................................... 150

Figure 135 SAP System Rename: Instance numbers ............................................................. 150

Figure 136 SAP System Rename: Mount points .................................................................... 151

Figure 137 SAP System Rename: Specify users for target system ........................................ 151

Figure 138 SAP System Rename: Specify database parameters ............................................ 152

Figure 139 SAP System Rename: Configure network isolation ............................................ 153

Figure 140 SAP System Rename: Disable Post Copy Automation (I) .................................. 153

Figure 141 SAP System Rename: Disable Post Copy Automation (II) ................................. 154

Figure 142 SAP System Rename: Summary Screen .............................................................. 154

Figure 143 Provider Implementation Definition (I) ............................................................... 157

Figure 144 Provider Implementation Definition (II) .............................................................. 157

Figure 145 Provider Implementation Definition (III) ............................................................ 158

Figure 146 Provider Implementation Definition (IV) ............................................................ 158

Figure 147 Custom Hook Definition (I) ................................................................................. 159

Figure 148 Custom Hook Definition (II) ............................................................................... 159

Figure 149 Custom Hook Definition (III) .............................................................................. 160

Figure 150 System State in Operations view ......................................................................... 161

Figure 151 Mass operation „Stop and Unprepare” started ..................................................... 161



Figure 152 Monitoring view .................................................................................................. 162

Figure 153 Mass Stop completed ........................................................................................... 162

Figure 154 Destroy System .................................................................................................... 163

Figure 155 Destroy operation – delete storage volumes ........................................................ 163

Figure 156 Destroy operation – delete host names ................................................................ 164

Figure 157 Destroy Operation – summary screen .................................................................. 164

Figure 158 Remove System in SAP LVM ............................................................................. 166



List of tables

Table 1 Functionalities in SAP LVM V1.0 .............................................................................. 12

Table 2 Supported SAP releases for the SAP LVM scenario .................................................. 13

Table 3 Overview of the Scenarios .......................................................................................... 20

Table 4 Project Phases and estimated time .............................................................................. 22

Table 5 Software Stack ............................................................................................................ 23

Table 6 Operational Model ...................................................................................................... 32

Table 7 Sample directory structure for a distributed file system implementation ................... 33

Table 8 Entries in /etc/services ................................................................................................. 57

Table 9 Local Filesystem layout .............................................................................................. 57

Table 10 XIVGUI Install directory .......................................................................................... 60

Table 11 Minimum filesystem layout (DB2 UDB) .................................................................. 65

Table 12 Extended filesystem layout (DB2 UDB) ................................................................... 66

Table 13 Required setup procedures ...................................................................................... 129

Table 14 Examples for IP addresses ...................................................................................... 131

Table 15 List of SAP Notes .................................................................................................... 168



1 Introduction

1.1 Background and Business Context

In the era of worldwide business relations and processes, the flexible and non-disruptive

operation of business applications and their underlying middleware and hardware becomes an

essential factor for success. Even unforeseen fluctuations in transaction volumes and changing

business processes must be reflected immediately by the supporting system capacities. On the

other hand, planned interventions such as hardware or application maintenance and testing

must not disrupt SAP Business Suite availability to worldwide partners and consumers. In

order to accelerate innovation and fast adoption of new business processes, non-production

stages of SAP systems need to be deployed on demand, without latency for procurement and

installation.

Consequently, manually operating and administrating a growing number of individual

systems or landscape components is no longer an option. Here, the concepts of cloud

computing not only promise to save costs but also to increase flexibility, elasticity, and

automation of system operations to efficiently serve the needs of the business.

This paper describes how to design and implement an on-premise cloud environment by

combining IBM platform technologies, namely IBM Systems Director and Tivoli Flash Copy

Manager, with the SAP NetWeaver® Landscape Virtualization Management (SAP LVM)

environment. Finally the paper shows, how all the use-cases, including cloning, copying, and

refreshing of SAP systems, are managed via the SAP LVM Administrator interface.

1.2 Scope of this Cookbook

The technical integration of SAP LVM and IBM Systems and Systems Management Software

components is basis for the solution. Figure 1 illustrates all the major components.

The integration includes IBM Power Systems in combination with PowerVM and

VMControl, selected IBM Storage Systems respectively SAN Volume Controller in

combination with IBM Tivoli Storage FlashCopy Manager, and finally the IBM Systems

Director as common base for plug-ins and interfaces.

Figure 1 Logical connections of IBM and SAP components



This cookbook describes the specifics and installation steps for implementing the SAP

NetWeaver Landscape Virtualization Management software V1.0 on an IBM PowerVM /

AIX based infrastructure. It also provides how-to‟s for using the resulting administration

environment for launching and monitoring the most popular SAP “cloud use cases”.

The installation sequence has been tested in a Proof-of-Concept including

SAP Business Suite 7

DB2 UDB LUW

IBM POWER

AIX 6 and AIX 7

IBM Systems Director,

IBM Tivoli FlashCopy Manager

SAN-attached IBM XIV storage subsystem.

Because of the technical affinity of FlexSystem Manager (FSM) and IBM Systems Director,

we assume that the content applies to a PureFlex system containing virtualized POWER

nodes, too.

At the time of writing SAP LVM V1.0 has been made GA (June 2012) and is available with

SP3. Relevant to IBM infrastructure, this new support package added support for SAN

Volume Controller and V7000 in addition to the already supported IBM XIV Storage.

The fact that we describe a POWER based implementation does in no way mean other

hardware and virtualization platforms (including IBM System x) do not offer a comparable

degree of SAP LVM integration. However, the described scenario and its functionality are

most advanced and thus can provide an important differentiator for the IBM POWER and

storage platforms.

The cookbook is not intended to replace a solid technical knowledge of any of the involved

layers and is no substitute for existing literature by IBM and SAP. We will point to applicable

literature where needed.

1.3 SAP NetWeaver Landscape Virtualization Management software (SAP LVM)

SAP NetWeaver® Landscape Virtualization Management software lets customers monitor

and manage entire SAP software landscapes and their associated heterogeneous IT

infrastructures. SAP LVM provides a single point of control, which allows visualizing,

monitoring, and managing data center tasks. The managing aspect includes resource

allocation and de-allocation on IT, but also on SAP Business Suite level.



Figure 2 Generic SAP NetWeaver LVM capabilities

SAP LVM is an evolution of the previously established SAP Adaptive Computing Controller

(ACC 7.3). As such SAP LVM inherits functions like “Landscape-wide Visualization and

Monitoring” and “Automated Capacity Management” from the former ACC.

In addition, SAP LVM provides new and more powerful functions like “SAP System

Cloning” and “SAP System Copy / Refresh”, which are frequently performed operations at

SAP sites. They combine numerous steps on infrastructure and SAP application level

including Post Copy Automation (PCA) sequences. SAP PCA is only available with the SAP

LVM “Enterprise” Edition.

SAP LVM Standard is available for download from SAP Service Market Place as a default

component of any NetWeaver 7.3ff stack with no additional fee; the SAP LVM Enterprise

Edition is associated with additional SAP license fees.

SAP LVM Version 1.0 provides all the administrative functionalities as listed in Table 1.

Functions that require SAP LVM Enterprise Edition are highlighted in blue and italic font.

The Standard Edition establishes the basic framework and includes administrative functions

that have been available with SAP Adaptive Compute Controller (SAP ACC) in the past.

These are related to landscape monitoring and automated startup, stop, and relocate of SAP

instances.

The Enterprise Edition extends these core capabilities with the advanced functions including/

involving SAP Post Copy Automation sequences. One can mostly automate complex and

frequently applied SAP system management tasks. This includes generation of isolated clones

of a SAP Production system for testing purposes, but also replication of a Production system

into a QA- or Pre-Production-System that stay fully integrated within the existing transport

stages. For latter purpose, business data and system settings need to be replicated, while at

any time avoiding conflicts with the online production environment.



SAP LVM 1.0 Functionalities, GA June 2012

End-to-End SAP System Clone / Copy / Refresh for ABAP

End-to-End SAP System Clone / Copy / Refresh for JAVA

Post-Copy Automation (PCA) for ABAP and JAVA

Automatic Capacity Management (ACM)

Application Server Installation / Un-installation

Mass Operations (start / stop / relocate)

User Configurable Dashboards

Reporting

Landscape Visualization

Custom Operations and Custom Hooks (Extensibility)

Custom Services (Management of Non-SAP Components)

Table 1 Functionalities in SAP LVM V1.0

Although SAP LVM today supports a high degree of automation for individual systems, it

does not reflect interdependencies among logically linked systems on an application level.

Future versions should improve the way in which management of application topologies can

be automated also. SAP LVM will also become aware of complementary systems attached to

a backend system - like TREX, liveCache etc. - and consider those systems when performing

landscape change actions.

The next section will provide a closer look into the involved components, overall solution

architecture, and details about the technical integration of SAP LVM with IBM cloud

enabling technology.



2 Overview and Planning

2.1 Architecture overview

This section provides an introduction where and how individual functions are implemented.

2.1.1 SAP NetWeaver Landscape Virtualization Management

2.1.1.1 Technical Implementation

SAP LVM is an add-on to SAP NetWeaver 7.30/7.31 Application Server Java. The SAP

LVM software is available for any platform that supports the SAP NetWeaver releases

mentioned above.

The SAP systems managed by SAP LVM can run far older SAP releases. Table 2 lists the

supported releases for the managed SAP systems in a SAP LVM scenario. SAP note 1527538

describes the details on restrictions that may apply to some of the older SAP releases.

Supported Managed SAP systems (See SAP Note 1527538)

NetWeaver 7.00ff Web AS 6.40 Web AS 6.20 Web AS 6.10 R/3 4.6D R/3 4.6C

Table 2 Supported SAP releases for the SAP LVM scenario

Figure 3 illustrates the communication between all the different components. SAP LVM

software (both the SAP NetWeaver AS Java and the SAP LVM Add-on) is installed on a

single system designated as “Management Server. SAP LVM communicates with the

managed nodes via the SAP Host Agent (a process running on each managed node) for

landscape-wide monitoring and execution of tasks on the systems. Additionally there are

direct interfaces (black boxes in the diagram) to the IBM PowerVM virtualization layer and

the IBM Storage Layer. The IBM solution described here uses of all those paths.

Figure 3 SAP LVM Block Diagram (Source: SAP)



SAP LVM provides a set of APIs as interface. In order to support specific hardware,

virtualization, OS and DB layers, each SAP partner needs to supply a set of libraries that

translate the generic SAP LVM operations to platform specific commands and executes them

as integrated solution on the appropriate component. SAP Note 1527538 describes all the

supported platforms for SAP LVM: Not all platform providers have implemented specific

libraries for all of their platforms, or partly they support just a subset of the functionality.

Thus, the supported use case for the SAP LVM solution may vary by platform.

2.1.2 IBM Systems Management Components

IBM Systems Director® is the platform management backbone, providing all the building

blocks for integrated services management. Systems Director provides the IT administrator

with the features:

Unified management of physical and virtual resources for IBM servers, storage, and

networks.

Automated data center operations by implementing cloud-ready virtual infrastructures.

Figure 4 IBM Systems Director and Plug-Ins

Key value of IBM Systems Director is its ability to work in diverse IT environments, so it

reduces the number of required management tools and interfaces. This is one reason, why it

was selected as programmatic SAP LVM counterpart: Instead of creating and maintaining

multiple interfaces for several IBM platforms, one generic interface to SAP LVM is

sufficient.

The components highlighted by a red frame are the base for this document and were explicitly

tested in the environment. This does not mean that other combinations and stacks are not

supported by SAP or IBM. For example, IBM SAN Volume Controller (SVC) has become

available with the GA of SAP LVM and can support a large variety of IBM and non-IBM

“backend storage” subsystems attached.

https://service.sap.com/sap/support/notes/1527538



In case customers plan to implement the documented scenario on top of an IBM PureFlex

platform with POWER nodes and V7000 storage, this document should apply too. The Flex

System Manager (FSM) then substitutes IBM Systems Director. In respect to functions and

technical interfaces, both management layers are consistent.

Optional plug-ins can extend IBM Systems Director. The SAP LVM integration exploits:

IBM® Systems Director VMControl™ is the multi-platform virtualization management

solution. This application provides the following features that are also relevant in any SAP

landscape:

Create and manage virtual machines

Relocate virtual machines

Import, edit, create and delete virtual images

Deploy virtual images

The VMControl plug-in is used to actually perform the LPAR and AIX related steps along the

overall processing sequence.

IBM® Systems Director Storage Control extends the management of systems to include the

storage layer. It is based on IBM Tivoli Storage Productivity and enables functions as:

Extends storage management of IBM Systems Director to cover most IBM storage

systems

Storage device discovery and coverage in integrated physical and logical topology views

Show relationships between storage and server resources

Ability to configure logical and physical configuration

Ability to view controller and volume status and to set notification alerts

Integration with IBM Systems Director VMControl storage provisioning for image

creation, deployment, and cloning

Storage Control is used in the end-to-end provisioning sequence to allow “bare-metal”

LPAR creation and AIX installation: During the provisioning process, Storage Control

takes care that the storage for the OS is provisioned and attached to the LPAR.

IBM Tivoli® Storage FlashCopy® Manager. This software provides fast application-aware

backups and restores by exploiting snapshot technologies in IBM storage systems.

In the context of SAP landscape management, IBM Tivoli FlashCopy Manager is used to

create instantaneously consistent copies of online SAP systems. Instead of using these copies

as source for backups, SAP LVM uses the flashes as source for further processing system

replica by subsequent post-processing steps.

2.2 Combining the IBM and SAP Layers

2.2.1 Integration by Platform Libraries

Figure 3shows how the SAP LVM architecture is implemented for IBM POWER and IBM

storage platforms. The integration consists of a set of libraries, which transfer generic SAP



LVM commands into the specific syntax for the IBM environment. IBM provides these

libraries as part of the SAP LVM software stack.

Different communications paths exist between the managing server and the managed nodes

One communication path is via the SAP Host Agent

The SAP Host agent can execute OS commands on each managed SAP node. SAP

LVM communicates with this process, which calls the vendor libraries in order to map

the LVM commands to an OS, DB, or storage specific language and syntax.

The second communication path uses the SAP LVM virtualization adapters.

The managing system interfaces directly with the VMControl plug-in of IBM Systems

Director to control PowerVM and OS layers,

Another communication path uses the SAP LVM storage adapter (IBM Tivoli Storage

FlashCopy Manager). For non-disruptive copies of running SAP instances, SAP LVM

interacts with the Tivoli Storage Flash Copy Manager, which interfaces to the storage

system for the cloning, and handles then handles all storage activities required on the

managed nodes also. This implementation guarantees a consistent replication of SAP

instances by synchronizing the required steps among the involved infrastructure

layers (storage, AIX, DB).

For any supported IBM component own libraries need to be created: The introduction of IBM

Systems Director and SAN Volume Controller as common interface and abstraction layers

towards various underlying virtualization and hardware layers makes this step easier.

Figure 5 Software structure to integrate IBM IT layers



2.2.2 Example Workflow “SAP System Copy”

In this section, a walk-through through a sample “SAP System Copy” use case will

demonstrate how the integration works together. The workflow is composed of two phases

(See Figure 6):

System Cloning Phase

The system cloning phase begins by creating host name mappings on a name server

and is finished when the clones SAP systems is ready to be started in an isolated

network segment. These steps are the same for the system copy and system cloning

scenarios. At the end of this phase, a 1:1 clone of the source SAP system exists which

per default cannot be accessed from other system systems from outside that fenced

segment.

System Copy Phase

After the system-cloning phase is completed, SAP LVM triggers additional automated

tasks that make up the system copy phase. This includes renaming the SAP system

clone. The Post Copy Automation (PCA-) sequence adjusts the complete SAP

configuration so that finally a unique SAP system (SID) is established.

Figure 6 Generic SAP System Copy Workflow

For a detailed description of the workflow, the following landscape example is used:

The source system is an SAP WebAS (Application Server) ABAP system

It is deployed in an AIX LPAR.

The SAP system is configured with virtual IP addresses.

The operating system, SAP system, and database are installed on separate AIX logical

volume groups.

The physical disks for these volume groups are allocated on SAN storage systems.

The source system is configured in SAP NetWeaver Landscape Virtualization

Management.

The administrator navigates to the “Provision view” in the SAP LVM console and selects the

source SAP system (All the detailed steps are shown later in chapter 5.2). After pressing the

Copy button, SAP LVM guides the administrator through dialogue steps defining the end-to-

end process. SAP LVM knows all the configuration settings of the registered source system;

however, the administrator has to specify configuration parameters for the target system.



Reasonable values for some parameters are calculated automatically and are pre-set as default

during the configuration dialogue. Following parameters can be changed/ overridden (this is a

non-exhaustive list):

Target resource: SAP NetWeaver Landscape Virtualization Management

provides the option to choose an already existing AIX LPAR or to create a new

AIX partition as target for the system. (For the latter the virtualization manager

configured in SAP NetWeaver Landscape Virtualization Management needs to

be set-up for creating and configuring such an operating system partition).

Host names: The copied SAP system is given a new virtual IP address and

host name. The host name can be configured manually. Alternatively, SAP

NetWeaver Landscape Virtualization Management is able to create new

hostnames and to update the domain name server automatically.

Storage volumes: SAP NetWeaver Landscape Virtualization Management

uses the registered storage adapter to gather the data about the volumes

attached to the source system. The information is gathered directly from the

storage management system. The administrator can specify target volume

names and mount points.

Database consistency: SAP NetWeaver Landscape Virtualization

Management asks whether the SAP source system is to be stopped during the

cloning process or whether the system should stay up and running (for

example, in case of a production system). With the online mode, the

administrator decides whether SAP NetWeaver Landscape Virtualization

Management is responsible for database consistency. Otherwise, there is no

guarantee that the cloned database can be recovered and brought online. When

selecting database consistency, SAP NetWeaver Landscape Virtualization

Management switches the source database into online backup mode (Oracle) or

into suspend I/O mode (DB2). This prevents write operations to the storage

volumes during the actual cloning step.

Target isolation: On the target system network, network fencing has to be

established to ensure that the cloned system can be started without interfering

with other systems in the data center. This is achieved by using the IPsec

feature of AIX. SAP NetWeaver Landscape Virtualization Management allows

the definition of permitted outgoing network connections. All other outbound

connections are blocked while incoming connections are permitted.

SAP copy parameters: SAP NetWeaver Landscape Virtualization

Management queries information about the copied SAP system, such as the

new SAP system identifier (SAP SID) and master password.

SAP copy users: SAP NetWeaver Landscape Virtualization Management

allows new users to be created and defined for the copied SAP system on a

central user repository.

SAP post copy automation: SAP NetWeaver Landscape Virtualization

Management allows the selection of task lists and task list variants to be used



for post copy automation. This is possible for multiple clients in the SAP

system (such as basis tasks in „000‟ and BDLS in „100‟).

After all the configuration parameters have been defined, the administrator reviews the input

and then starts the SAP system copy workflow.

There is a narrated ScreenCam video available at TechDocs, in case you want get a real

impression about the look and feel of the SAP LVM administrator console.

2.2.3 Supported Use-Cases and involved components

The use-cases for

Landscape-wide Overview and System monitoring

VM activate, deactivate and relocate

Provision new Resources (AIX LPARs)

are included in the standard edition of SAP LVM and will leverage one of the virtualization

adapters during execution. These use-cases are not dependent on special storage, as long as

the storage satisfies the requirements of the virtual environment. IBM Tivoli Storage

FlashCopy is not required for execution of these use-cases. Enhanced functionalities (e.g.

customized dashboards, generation of reports) will require the enterprise edition.

The use-cases for

SAP Storage volumes: Attach, Detach, Relocate

SAP Clone/ Copy/ Refresh

have some dependencies on the storage infrastructure.

The implementation of Attach/ Detach/ Relocate depends on the actual storage type: For NAS

and GPFS the functionality is included in the platform library. For SAN-based IBM storage,

the Storage Adapter is invoked during execution of the use-cases.

Table 3 summarizes the different use case families in SAP LVM and illustrates which Edition

for SAP LVM is required, which SAP LVM adapters needs to be invoked, and which

boundary conditions need to be fulfilled for the execution.

http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4949



Use Case Landscape-

wide Overview

and System monitoring

VM activate, deactivate

and relocate

Provision new Resources

(AIX LPARs)

SAP Storage volumes: Attach, Detach, relocate

SAP Clone/ Copy/

Refresh

SAP LVM Adapter

Virtualization Adapter

Storage Adapter or Platform Library

Storage Adapter

SAP Landscape Virtualization Management Edition

Standard Edition

(advanced

scenarios require Enterprise

Edition)

Standard Edition

Standard Edition

Standard Edition (NFS and GPFS), Enterprise Edition for SAN based

storage.

Enterprise Edition

Virtu

aliz

ation

Mana

ge

ment

IBM Power Systems with IBM Systems Director

Yes, requires VMControl Express Edition

Yes, requires VMControl Express Edition

Yes, requires VMControl Express Edition, requires Storage Control

not required not required IBM PureSystems with IBM Flex System Manager

HMC Yes Yes No

Sto

rage

Mana

ge

ment

IBM Tivoli Storage FlashCopy Manager

not required not required not required not required required

Supported Storage

Any Any Any

NAS (NFS), GPFS, or IBM

XIV, IBM SVC, IBM

V7000

IBM XIV, IBM SVC, IBM

V7000

Table 3 Overview of the Scenarios

2.3 Planning Project Resources and Timing

Table 4 summarizes the activities in order to implement a SAP LVM managed POWER

landscape comparable to our PoC setup described in the following section 2.4. The estimated

duration per phase provides a rough guideline only. In detail, the times are dependent on the

complexity of the SAP landscape (IT and SAP systems) and its operation as well as on the

availability of all the required skills for all involved disciplines.



Project Phase Included Tasks / Accomplishments Estimated time

Planning workshop, Verification of landscape prerequisites in client environment

Network and disk design requirements, SW prerequisites

Validate that customer has obtained the necessary SAP and AIX software, validate that dedicated IBM hardware infrastructure is available

SAP Landscape needs to support virtual host names, and a proper range of IP addresses needs to be available

2-3 Day workshop with customer

Plan and Prepare setup of storage infrastructure

Storage LUNs

Network, SAN zoning

Eventually map SVC to existing storage

Plan and create filesystems

3-4 days working with customer

Installation and configuration steps for the management environment (IBM Systems Director, SAP NetWeaver Landscape Virtualization Management)

Define LPARs on the managed systems, and install AIX OS on the LPARs

Set-up virtualization environment o Install and configure IBM Systems

Director including VMControl and Storage Control

o Integrate NIM server into System Director VMControl environment

o Set-up Storage Control for all storage entities (SAN, Storage System)

Install and configure storage management software (FlashCopy Manager, Storage CLI) on the management system

Plan and configure shared infrastructure services

o Install and configure LDAP (if not to be integrated with existing LDAP environment

o Plan for DNS

Install SAP LVM system

Two weeks working with customer

Installation and configuration steps of managed environment incl. basic testing Installation from scratch for one SAP System

Define AIX “golden image” content

Install first AIX LPAR with golden image

Install and configure storage management software (FlashCopy Manager, SVC CLI)

Install first “source” SAP system

LVM base configuration (Virtualization Manager, Storage Manager configuration, discover resource LPARs and SAP systems, users)

Capture image in Systems Director VMControl for further deployment

Additional LVM use case specific configuration (storage, post processing, steps per use case) Note: configuration of PCA post processing requires skilled SAP specialist with both SAP Basis and Application/Business Process knowledge)

Two weeks with customer

Installation and configuration steps of managed environment incl. basic testing Migrate/ Customize one already existing

Adapt existing SAP system to virtualised environment

Eventually Upgrade SAP Hostagent

Install and configure agents and storage management

Install and configure storage management

At least 2 weeks with customer



Project Phase Included Tasks / Accomplishments Estimated time

SAP system

software (FlashCopy Manager, SVC CLI)

Discovery and setup of LPARs and SAP systems in IBM Systems Director and SAP LVM

Additional LVM use case specific configuration (storage, post processing, steps per use case) Note: configuration of PCA post processing requires skilled SAP specialist with both SAP Basis and Application/Business Process knowledge)

Verification Steps using the customer’s SAP system

Perform functional verification

Execute and document each use case o Start / Stop / Relocate of LPARs and SAP

systems o Clone SAP Systems (target LPAR available) o Copy SAP Systems (target LPAR available) o Refresh SAP system o Clone SAP Systems (create target LPAR

from scratch) o Copy SAP Systems (create target LPAR

from scratch)

At least 2 weeks working with customer

Acceptance Tests

Support customer in LVM acceptance testing from basis or application level

At least 1 week

Document Solution or Handover

at least 2 days for handover, documentation at least 1 week

Table 4 Project Phases and estimated time

2.4 Tested configuration

The installation hints in this document are based on a PoC test installation in the ISICC demo

landscape using IBM Power Systems and IBM XIV Storage Systems.

Figure 7 illustrates the environment in the PoC.

In the management environment:

Virtualization Manager is IBM Systems Director 6.3.1 which is installed on one

LPAR. Plug-in VMControl 2.4.1 is deployed to that node also. The Storage Control

plug-in was not part of the PoC environment: All the “end-to-end” deployment use-

cases for provisioning of a new resource LPAR (as shown in section 5.1.3) were

executed within a different environment.

SAP NetWeaver Landscape Virtualization Management software V1.0 SP3 is

deployed on top of a SAP NetWeaver Java AS 7.3 engine in a second LPAR

Additional infrastructure servers like DNS, LDAP server, and NIM server are

available and configured.

In the managed environment

at least one SAP source system is existing



all resources (AIX LPARs) for both the SAP source system(s) and SAP target

system(s) are registered within SAP LVM. The resources are attached via virtual fibre

channel to an IBM XIV Storage system.

On each resource LPARs the software components

SAP Host Agent package

IBM Tivoli Storage FlashCopy Manager

IBM XIV XCLI

are installed and configured.

Systems DirectorAIX

NIM ServerAIX

Management Environment

IBM XIV

SVC / Storwize

SAP LVMAIX

LDAP

IBM Tivoli Directory Server

Source

Target

VIOS

vFC (NPIV)

IBM Systems Director 6.3.1

VM Control 2.4.1

DNS

SAP NetWeaver Landscape

Virtualization Management V1.0 SP3

SAP NetWeaver Java AS 7.3

SAP SourceAIXSAP Source

AIXSAP SourceAIXSAP Source

AIXSAP SourceAIXSAP Source

AIX

SAP Component - SAP NetWeaver

Managed Environment

VIOS

vFC (NPIV)

SAP TargetAIX

• Storage connected

via vFC (NPIV)

• Compliant Storage

Layout

IBM Tivoli FlashCopy Manager 3.1.0.1

Storage Manager native Interfacee.g. IBM XIV XCLI 3.1 build 18





Figure 7 Architecture overview diagram

Table 5 reflects the detailed software stack tested in the scenario:

Software Version

SAP Landscape Virtualization Management Software

1.0 SP3

IBM Systems Director / VMControl 6.3.1 / 2.4.1

IBM Script Package

SAP Host Agent 7.20 PatchLevel 103

SAPINST (Rename)

abi version : 722 make variant: 720_REL

build: 1287791 compile time: Dec 19 2011 22:27:22

AIX 6.1 TL7 SP2 AIX 7.1 TL1 SP3

6100-07-02-1150 7100-01-03-1207

IBM Tivoli Storage FlashCopy Manager 3.1.0.1

IBM XIV 10.2.4.a

XIVGUI XCLI 3.1 build 18

DB2 UDB DB2 v9.7.0.5 Special Build 28492 (FP5SAP)

Table 5 Software Stack



2.5 SAP LVM Environment

2.5.1 Storage Virtualization

A SAP LVM infrastructure requires a centralized storage solution. The storage infrastructure

in the ISICC sample environment is based on IBM System Storage™ solutions using a

Storage Area Network (SAN).

IBM System Storage virtualization products achieve the abstraction from physical volumes of

data storage to a logical level. It addresses the increasing complexity of managing storage,

while reducing the associated costs. Its main purpose is the full exploitation of the benefits

promised by a SAN. Virtualization enables data sharing, ensuring higher availability,

providing disaster tolerance, improving performance, allowing for consolidation of resources,

providing policy-based automation, and much more besides, which do not automatically result

from the implementation of today‟s SAN hardware components. Storage virtualization is

possible on several levels of the storage network components, meaning that it is not limited to

the disk subsystem. Virtualization separates the representation of storage to the operating

system and its users, from the actual physical components.

Storage virtualization accumulates the storage into storage pools, which are independent of

the actual layout of the storage (that is, the overall file system structure). Because of this

independence, new disk systems can be added to a storage network, and data migrated to

them, without causing disruption to applications. Since the storage is no longer controlled by

individual servers, it can be used by any server as needed. In addition, it can allow capacity to

be added or removed on demand without affecting the application servers. Storage

virtualization will simplify storage management, which has been an escalating expense in the

traditional SAN environment.

IBM SAN Volume Controller (SVC), IBM Storwize V7000, or IBM XIV System Storage is

not mandatory for all the use-cases in an adaptive computing environment, but they help in

the overall management of the storage landscape. For the SAP System Copy Use-Cases

however the Storage Adapter and Storage Library require one of those systems.

2.5.1.1 SAN Volume Controller System Storage

The IBM (SVC) is an in-band, block-based virtualization product that minimizes the

dependency on unique hardware and software, decoupling the storage functions expected in a

SAN environment from the storage subsystems and managing storage resources. SVC

combines software and hardware into a comprehensive, modular appliance that uses

symmetric virtualization.

Symmetric virtualization is achieved by creating a pool of managed disks (MDisks) from the

attached backend storage systems. Those storage systems are then mapped to a set of volumes

for use by attached host systems. System administrators can view and access a common pool

of storage on the storage area network (SAN). This functionality helps administrators to use

storage resources more efficiently and provides a common base for advanced functions.

Key feature exploited for the SAP System Copy scenarios is the ability to create and manage

FlashCopies of existing (SAP application) volumes. IBM FlashCopy is a part of the SVC

Copy Services and provides a point-in-time copy function. This includes thin-provisioned

FlashCopy to make multiple targets affordable.



2.5.1.2 IBM Storwize V7000 Storage System

The IBM Storwize V7000 is a storage server with internal disks, providing the same storage

virtualization functions and features like SAN Volume Controller.

2.5.1.3 IBM XIV System Storage

The IBM® XIV® Storage System is a high-end disk storage series designed to address

storage challenges across the application spectrum, including for virtualization, email,

database, and analytics and data protection solutions. The IBM XIV Storage System is a fully

virtualized system designed to eliminate the need for performance tuning and numerous other

storage management activities. IBM XIV provides consistent, hotspot-free enterprise

performance and exceptional ease of use. As virtualized storage that meshes tightly with

hypervisors, XIV offers optimal agility for cloud and virtualized environments.

Key feature exploited for the SAP System Copy scenarios is the ability to create and manage

snapshots of existing (SAP application) volumes. The IBM XIV Storage System has taken

this concept one step further, offering a completely innovative approach to snapshot creation

and management. The XIV system offers clear advantages, including the following:

Unlimited number of snapshots in the system.

Snapshot creation in virtually zero time, regardless of the size of replicated volumes.

Unaffected performance levels in a system that supports snapshots, regardless of the

number of snapshots currently defined in the system.

2.5.1.4 General Parallel File System (GPFS)

Exploiting GPFS is the easiest and most convenient way to implement the „classic“ adaptive

scenarios for relocating SAP services between different resources.

IBM's General Parallel File System (GPFS) provides file system services to parallel and serial

applications. GPFS allows parallel applications simultaneous access to the same files, or

different files, from any node, which has the GPFS file system mounted, while managing a

high level of control over all file system operations. GPFS is particularly appropriate in an

environment where the aggregate peak need for data bandwidth exceeds the capability of a

distributed file system server.

GPFS allows users shared file access within a single GPFS cluster and across multiple GPFS

clusters. A GPFS cluster consists of:

AIX nodes, a node may be:

o An individual operating system image on a single computer within a cluster.

o A system partition containing an operating system.

Network shared disks (NSDs) created and maintained by the NSD component of

GPFS

A shared network for GPFS communications allowing a single network view of the

configuration. A single network is used for GPFS communication, including the NSD

communication.



All disks utilized by GPFS must first be given a globally accessible NSD name. On AIX

server running GPFS, an existing virtual shared disk or physical disk may be given an NSD

name.

2.5.2 SAP NetWeaver Landscape Virtualisation Management

All the base considerations are described in the SAP Notes Note 1527538 - SAP NetWeaver

Landscape Virtualization Management 1.0 and Note 1644520 - Integration of IBM

Components into SAP LVM

SAP LVM distinguishes in its configuration between:

Services

A service is a whole or a part of a SAP System (database service, central instance

service, application server service). The SAP LVM use-cases typically are executed on

the services.

and

Resources

A resource describes an operating system installed on a physical or virtual server. SAP

services may be relocated between resources, or a resource may be the target for a new

SAP system created via SAP system copy or cloning. A new resource may be

deployed (via SAP LVM and IBM Systems Director) so that it can be used as a target

for a relocate operation or a SAP system copy use-case. SAP LVM monitors resources

through SAP Host Agents running on the operating systems.

2.5.2.1 SAP LVM Virtualization Management

To extend resource monitoring data available via SAP Host Agents, to provision new

resources, or to perform load balancing by starting, stopping and relocating the resources,

SAP LVM interacts with the virtualization managers:

SAP LVM virtualization managers (adapters) for

IBM Systems Director/ VMControl (respectively IBM Flex Systems Manager)

or

IBM Hardware Management Console for IBM Power

Both allow

Monitoring of IBM Power physical servers and logical partitions (LPARs)

Management of LPARs (Power on/ Power off)

Relocation of LPARs using Live Partition Mobility

Additionally the “IBM Systems Director/ VMControl” adapter provides provisioning

capabilities for both AIX OS and Storage.

See SAP Note 1644520 - Integration of IBM Components into SAP LVM for detailed

description of supported configurations and features.

2.5.2.2 SAP LVM Storage Management

For cloning, copying and refreshing of SAP systems and for preparing, un-preparing and

relocation of SAP services on / between resources SAP LVM interact with the storage

manager:








SAP LVM storage manager (adapter) for IBM Tivoli Storage FlashCopy Manager

(FCM)

This adapter supports invocation of FlashCopies/ SnapShots within the storage system to

provide a fast, efficient, and application-aware cloning of the source system within the

storage. At the time of writing this document (July 2012) this adapter supports following

storage system types:

IBM XIV Storage System

IBM Storwize V7000 storage system

IBM SAN Volume Controller (incl. any supported backend storage system)

In addition, the adapter provides the required functionality for the management of the storage

to attach/ detach volumes and filesystems to/ from the LPARs during the prepare and

unprepared activities.

See SAP Note 1644520 - Integration of IBM Components into SAP LVM for detailed

description of supported configurations and features.

In this PoC we used the following configuration for SAP System Copy use cases:

IBM XIV Storage System, Version 10.2 with XIV CLI 3.0

Storage volumes in the storage system connected to the IBM Power system LPARs

either by dedicated physical fibre channel adapters or by virtual fibre channel adapters

(N-Port ID virtualization, NPIV).

The volumes of the source system are connected to the source LPAR only. The source

volumes must not be connected to the target LPAR.

The SAP Host Agent (located in /usr/sap/hostctrl) is installed on a storage volume that

is not part of any cloning scenario: We recommend creating a /usr/sap filesystem in

“rootvg”.

The target host LPAR is defined as host in the storage system. We recommend

deploying the AIX operating system of the target LPAR to volumes within the same

storage system: then this requirement is fulfilled automatically.

2.5.2.2.1 SAN zoning requirements

As a prerequisite for all the SAP System Copy use cases (SAP System Clone, SAP System

Copy, SAP System Refresh) SAN storage is required with the IBM Tivoli Storage FlashCopy

Manager (FCM) Adapter. SAN zoning on the target resource LPAR must be pre-configured:

This can be easily fulfilled if the volume(s) for the AIX operating system (rootvg) of the

target LPAR is located on the same storage like the SAP systems in scope of the System

Copy. Then the existing access to the rootvg on the target LPAR provides SAN access for the

SAP application volumes too. In this case, the “host” and “adapter” definition in the IBM XIV

storage system exist already, and the LPAR has an already zoned access in the SAN to the

storage system. The SAP application volumes are assigned to the LPAR by “LUN masking”

commands out of the SAP LVM storage adapter.




The storage system should be zoned according to “single initiator zones”. Each zone set

should contain one (active) virtual fibre channel host port only, but can include multiple

storage adapter ports. (E.g. between three and six adapter ports for an IBM XIV storage

system). For IBM XIV storage the maximum number of paths that can be zoned to an AIX

LPAR is 32. However there is no benefit in having more than 24 paths.

Although not mandatory it is best practise to define the name of the host in the IBM XIV

storage system identical to the name of the LPAR or its hostname. If virtual fibre channel

adapters are used, then all the WWPNs need to be included in the IBM XIV port definition, as

well as in the SAN zoning definition. This includes all “active” WWPNs via the VIOS pair, as

well as the secondary WWPNs used during a Live Partition Mobility relocation event.

2.5.3 Considerations for the SAP LVM resources

Several storage attachment considerations need to be obeyed, and additional software

components need to be installed on the resource LPARs. All these are described shortly in this

section

2.5.3.1 VIOS / PowerVM and SAN attachment

For all the SAP System clone/System Copy scenarios that invoke storage-based cloning, all

the storage volumes for both the source and the target system need to be attached via virtual

Fibre Channel (vFC) adapters and using NPIV. Figure 8 illustrates the redundant attachment:

two redundant VIO Servers are installed in the POWER system. Each of the VIO servers has

one or more physical dual-port fibre channel adapters assigned to the partition. The two ports

of the fibre channel adapter card are connected to two different SAN fabrics. Also, the XIV

storage system is connected to both fabrics. If a LPAR has only one fibre channel adapter,

then zone it to three IBM XIV modules. Zone it to six modules only in case the LPAR has

very high throughput requirements.

Typically, each client LPAR is defined with four virtual fibre channel adapters: On each VIO

server two virtual fibre channel server adapters are created for each client LPAR. These server

adapters are assigned to the corresponding virtual fibre channel client adapters for each

LPAR. Four pairs of WWPNs are created (WWPN1, WWPN2, WWPN3, WWPN4, plus the

additional WWPNs for the LPM scenario).

LPAR

IBM XIV Storage

connected via vFC

(NPIV)

vFC Client

WWPN1

vFC Client

WWPN2

vFC Client

WWPN3

vFC Client

WWPN4

VIOS #1 VIOS #2

vFC ServervFC Server vFC ServervFC Server

FC adapter

port 1

FC adapter

port 0

FC adapter

port 1

FC adapter

port 0

FC adapter

port 1

FC adapter

port 0

FC adapter

port 1

FC adapter

port 0

Fabric #1

Fabric #2

Figure 8 redundant virtual fibre channel attachment



Use single initiator zones: Each zone set contains one (active) virtual fibre channel host port

and between three and six XIV ports. The maximum number of paths that can be zoned to an

AIX LPAR is 32. However there is no benefit in having more than 24 paths.

2.5.3.2 AIX operating system specifics

The AIX operating system on the managed systems needs to fulfil certain criteria:

IPSec is required for network fencing

All required filesets need to be installed. IPSec needs to be activated in the OS

DNS should be exploited

For the sake of simplicity, all the virtual hostnames of the SAP services should be

provided by DNS.

A central user management is recommended.

In the test environment, LDAP is used for that purpose. The LDAP filesets need to be

installed, and the OS needs to be configured as LDAP client.

No central repository for the services entries was used.

The /etc/services file need to be prepared and distributed to all resource LPARs

2.5.3.3 IBM Tivoli Storage FlashCopy Manager

IBM Tivoli Storage FlashCopy Manager software provides fast application-aware backups

and restores, so leveraging advanced snapshot technologies in IBM storage systems. The

software provides following features in general:

Near-instant application-aware snapshot backups, with minimal performance impact

for IBM DB2, Oracle, SAP, Microsoft SQL Server, and Exchange

Improvement in application availability and service levels through high-performance,

near-instant restore capabilities that reduce downtime

Integration with IBM Storwize V7000, IBM System Storage DS8000, IBM System

Storage SAN Volume Controller and IBM XIV Storage System on AIX, Solaris,

Linux, and Microsoft Windows

Fulfilment of advanced data protection and data reduction needs with optional

integration with IBM Tivoli Storage Manager

In the context of SAP landscape management, IBM Tivoli FlashCopy Manager is utilized to

create instantaneously a consistent copy of an online SAP system.

2.5.3.4 XCLI

The IBM XIV Storage System command-line interface (XCLI) provides a mechanism for

issuing commands to manage and maintain the XIV storage systems. XCLI commands are

entered on the XCLI client either by a user or by another application. The XCLI client and the

XIV system communicate using Secure Sockets Layer (SSL) over TCP/IP.

2.5.3.5 SAP Host Agent

The SAP Host Agent package contains all the required elements for centrally monitoring of

any hosts. It is used by SAP LVM to monitor and manage SAP services and resources also.

So the SAP Host Agent package needs to be installed and configured on each host containing



an SAP component managed by the SAP NetWeaver Landscape Virtualization Management

software.

The SAP Host Agent package is automatically installed as part of the “SAPinst” installation

procedure for any components of SAP NetWeaver 7.0 EHP2 and higher. However it may

need to be updated to a higher patch level to exploit all SAP LVM management features. It is

also required to install SAP Host Agent manually on operating systems without any SAP

systems. In our case these are operating systems that act as targets in SAP System Copy and

SAP Service Relocation use cases.

2.5.4 Considerations for the SAP LVM Services (SAP Systems)

All SAP systems need to be installed compliant to a SAP LVM environment. This includes

different considerations for user management, network, and storage. Some best practises are

discussed in the following.

2.5.4.1 User Management

Ensure that the administrative users are available with same UID and GID on all the

LPARs involved in the scenario.

A central user management solution provides the same UIDs, GIDs and passwords for

one user on all the computing nodes. In the environment described here this is

achieved by using LDAP.

2.5.4.2 Network considerations

The following guidelines for the network setup for the SAP system to be managed by SAP

LVM needs to be considered:

Provide a dedicated virtual hostname for each SAP service. The different services of

the SAP system (SAP database, SAP Central Services, SAP instances) shouldn‟t share

an IP address. Each service should own a dedicated IP address.

Provide a dedicated IP address for each virtual host name. Do not use multiple virtual

hostnames (IP alias names) referring to one and the same IP address.

Ensure that all the virtual host names can be resolved on the SAP LVM system.

Ensure that the reverse lookup on the SAP LVM server provides a one-to-one

mapping.

Ensure that all the virtual host names can be resolved on any resource LPAR. Ensure

that the reverse lookup on the LPAR provides a one-to-one mapping.

2.5.4.3 Storage Layout

Separate all storage for the different services. During the prepare/unprepare activities

is must be possible to handle the storage independent of the services:

o Create different AIX volume group for the filesystems related to the SAP

Central Instance and the Database

o If a Solution Manager Diagnostics agent exists for the SAP System, place its

filesystems to the Central Instance.



o If there is a dedicated Solution Manager diagnostics agent for the Database,

then its filesystems need to be part of the volume group of the database

o If you intend to use FlashCopy backup for the production database also, then

split the database content into three VGs: database data (tablespaces), database

online logs, and other database files (instance directory, executables, archive

logs etc …)



3 Installation

Table 6 gives an overview about the operational model. In the test environment, both the

management environment and the managed environment are hosted on IBM POWER using

AIX operating system. Most of the deployment units for the managed environments were

directly added to the AIX operating system image: By this means they are part of the NIM

deployment process for a new SAP LVM resource and are automatically available on the new

system.

Management environment

Node Deployment Unit

Management LPAR SAP LVM

AIX

LDAP Client (IBM Tivoli Directory Server Client)

SAP NetWeaver 7.3 Java System / DB2 UDB

SAP NetWeaver Landscape Virtualization Management 1.0 SP03

IBM Tivoli Storage FlashCopy Manager [acsd]

XIVGUI XCLI

SAPHOSTAGENT package and IBM partner libraries

Management LPAR IBM Systems Director

AIX

LDAP Client (IBM Tivoli Directory Server Client) optional

IBM Systems Director incl. plug-in VMControl

IBM Systems Director Storage Control required for end-to-end provisioning of new LPARs

NIM Server

AIX, setup as NIM Master

LDAP Client (IBM Tivoli Directory Server Client) optional

IBM Systems Director VMControl subagent for NIM

LDAP Server AIX

IBM Tivoli Directory Server

DNS Server AIX bind

Managed Environment

Node Deployment Unit Part of

“golden AIX image”

on all SAP LVM resource LPARs

AIX Yes

LDAP Client (IBM Tivoli Directory Server Client) Yes

SAPHOSTAGENT package and IBM partner libraries Yes

IBM Tivoli Storage FlashCopy Manager [acsgen] Yes

XIVGUI XCLI Yes

SAPInst rename required for SAP System Copy on “target resource”

No (accessed

from a central NFS share)

on one SAP LVM resource LPAR

SAP ECC 6.0 source system / DB2 UDB

SAP PostCopyAutomation tool optional, allows for key-functionality in SAP System Copy

Table 6 Operational Model

This chapter describes the installation of the components for both the management and

managed environments for an AIX/ POWER environment. Not all the installation steps will

be described in all details: It is assumed that the reader has sufficient background knowledge

to perform those.



In principle the SAP NetWeaver Landscape Virtualization Management and IBM Systems

Director could be deployed to another platform, e.g. LINUX/INTEL or MS Windows Intel:

Then the deployment unit “IBM Tivoli Storage FlashCopy Manager [acsd]” would need to be

moved to another AIX server (e.g. to the NIM server). XIVGUI XCLI and SAP Host Agent

package and partner libraries then would need to be installed on this server too.

3.1 Naming Conventions

In a SAP LVM environment SAP services can potentially run on any SAP LVM enabled

server. For example, you could install all service relevant data (database, binaries, and

working directories) on a distributed file system. In this case, we recommended putting this

data under a common root directory on that distributed file system.

A careful planning is required to avoid the collision of various names like IP names, user

names, user ids and group ids, directory structures, etc. One method is to include the SAP SID

(which is typically unique in a customer environment) in the required service IP names and

root directories of the service data. In addition, if you want to run multiple services on a single

compute node, you have to plan for unique SAP instance numbers as well, as you cannot run

two services with the same instance number (but different SIDs) within the same OS image.

3.1.1 Sample Service IP Names for SID C01

The following is one possible naming convention to build SAP service IP names. There are no

specific requirements for IP names in the SAP LVM infrastructure, however choosing and

sticking to one convention will help to simplify the administration.

c01db01 IP name for database server

c01ci01 IP name for central instance

c01as01 IP name for first application server instance

c01as02 IP name for second application server instance

3.1.2 Sample directory structure for SID C01 on a Distributed File System

Each SAP system should have a common high level subdirectory name to provide a standard

entry point to all data that belongs to the same system. For example if the distributed file

system is mounted at /sapfs, then all data belonging to SAP system C01 could be allocated

under the subdirectory /sapfs/sapC01. The “root” directory for the SAP system C01 would

then be /sapfs/sapC01. In that root directory, all instance relevant subdirectories are stored.

The mappings between the physical locations on the storage subsystem and the directory

locations as they are expected by the SAP kernel have to be defined in SAP LVM. The

controller will then create the required mapping between the new data location and the

required standard path before starting a service on a server. Table 7 shows a sample directory

structure for SID C01:

Directory Content Path in standard

installation

<root>/home/c01adm Home directory for SAP admin user /home/c01adm

<root>/db2 <root>/oracle

Database Server, Client, instance /db2 /oracle

<root>/sapmnt/C01 Directory for Executables, Profiles, Shared files /sapmnt/C01

<root>/usr/sap/c01ci01 Instance Directories for central instance /usr/sap/C01

<root>/usr/sap/c01as01 Instance Directories for application server 01 /usr/sap/C01

/sapfs/saptrans01 Common transport directory /usr/sap/trans

Table 7 Sample directory structure for a distributed file system implementation



The screenshot in Figure 9 shows the service configuration screen on the SAP LVM

Controller which illustrates the mapping between physical layout and the required layout for

the SAP database service EC1. The storage type is Distributed File System (DFS) with a

subtype of “gpfs", which refers to a General Parallel File System installation.

The entries with mount option dirmount define the required mapping between the physical

location and the mount point that is expected by the SAP kernel.

Figure 9 Service configuration in SAP LVM with distributed file system layout

3.1.3 Sample directory structure using Network Attached Storage (NAS)

The same concept as in the previous section can be used to layout the physical data locations

on a NAS device. A typical organization unit with NAS is called a volume where you store

directories and files. Those resources then have to be exported to the clients that need to

access them. The root directory for service C01 could for example look like

nas_server:/vol/sapvol1/sapC01, where nas_server is the IP name of the NAS device. Again

the mappings between physical locations and expected locations have to be defined in the

SAP LVM Controller. The controller will then create the required mappings (in this case

using client NFS mount instead of creating symbolic links) before starting a service on a

server.

3.2 Install Storage Infrastructure

3.2.1 Exploit GPFS as “adaptive storage”

This section provides a brief summary of setup tasks to get started with a GPFS cluster using

the ISICC SAP LVM landscape implementation as an example.

Using GPFS it is possible to carry out the use cases of the standard edition of LVM, that is in

general start, stop, and relocate SAP systems.

We do not cover in detail typical GPFS administration tasks like GPFS installation and

administration, the creation and management of storage pools, definition of file placement



policies, etc. For those tasks you need to refer to the standard product documentation and

available Redbooks. Especially to configure data availability and disaster recovery scenario,

you should carefully review the GPFS Base documentation (e.g. the “GPFS Concepts,

Planning, and Installation Guide” and the “GPFS Advanced Administration Guide”).

Additional information can be found in the GPFS Wiki

3.2.1.1 Planning for GPFS

3.2.1.1.1 Quorum Nodes

The first task is to decide on a node quorum strategy. As described in the “GPFS Concepts,

Planning, and Installation Guide”, GPFS enforces a node quorum rule to prevent multiple

nodes from assuming the role of the file system manager in the event of a network

communication problem in the cluster. A majority of quorum nodes must remain active in

order for the cluster to sustain normal file system usage (multi-node quorum).

The ISICC implementation uses a new node quorum mechanism that was introduced with

GPFS V2.3 – node quorum with tiebreaker disks, which allows you to run with as little as one

quorum node available as long as you have access to a majority of the quorum disks. Please

refer to Chapter 2 in the “GPFS Concepts, Planning, and Installation Guide” for a detailed

description of the quorum concept and guidelines to select quorum nodes. In the ISICC setup,

we use two quorum nodes and one tiebreaker disk.

3.2.1.2 Install and setup GPFS

3.2.1.2.1 Install GPFS software

You need to install the following filesets:

gpfs.base

gpfs.msg.en_US

This is best accomplished by including these filesets in a NIM installp bundle.

3.2.1.2.2 Create GPFS Cluster

The mmcrcluster is used to create a GPFS cluster. Our setup uses two quorum nodes: siccps10

and siccps12. The same nodes are also used as primary (option –p) and secondary (option –s)

configuration server to store the GPFS cluster configuration data. For cluster communication

we use ssh and scp instead of rsh and rcp, which has to be specified with the –r and –R

options. The –A option specifies that GPFS daemons are to be automatically started when

nodes come up. The default is not to start daemons automatically. Our cluster is named

siccfs.isicc (-C option).

root> mmcrcluster –N siccps12:manager-quorum,siccps10:manager-quorum –p siccps12 –s siccps10 –

r /usr/bin/ssh –R /usr/bin/scp –C siccfs.isicc –A

After a successful creation of the cluster, you can verify your setup using the mmlscluster

command.

root> mmlscluster

GPFS cluster information

========================

GPFS cluster name: siccfs.isicc

GPFS cluster id: 691765743288319456

GPFS UID domain: siccfs.isicc

Remote shell command: /usr/bin/ssh

Remote file copy command: /usr/bin/scp

http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp

http://publib.boulder.ibm.com/epubs/pdf/a7604135.pdf


http://publib.boulder.ibm.com/epubs/pdf/c2351825.pdf

http://www.ibm.com/developerworks/wikis/display/hpccentral/General+Parallel+File+System+%28GPFS%29






GPFS cluster configuration servers:

-----------------------------------

Primary server: siccps12

Secondary server: siccps10

Node Daemon node name IP address Admin node name Designation

----------------------------------------------------------------------------------------------

1 siccps12 9.153.165.71 siccps12 quorum-manager

2 siccps10 9.153.165.69 siccps10 quorum-manager

Then you need to startup GPFS on all nodes and you should check the state of the cluster:

root> mmstartup -a

root> mmgetstate -aL

Node number Node name Quorum Nodes up Total nodes GPFS state Remarks

------------------------------------------------------------------------------------

1 siccps12 2 2 8 active quorum node

2 siccps10 2 2 8 active quorum node

3.2.1.2.3 Define Network Shared Disk (NSD) Devices

GPFS uses Network Shared Disk (NSD) devices to provide access to the disks. A NSD device

can either be physically attached (using a SAN infrastructure) to a node or it can be accessed

using a virtual connection (through the network) using a primary and, if specified, backup

NSD server. GPFS determines if a node has physical or virtual connectivity to an underlying

NSD through a sequence of commands invoked from the GPFS daemon. This determination

is called disk discovery and occurs at both initial GPFS startup as well as whenever a file

system is mounted.

In the ISICC PoC landscape, we used the following configuration file to create the NSD

devices:

/var/mmfs/config/nsd.disks:

#Description of disk attributes

#<disk name>:<primary NSD server>:<2ndary NSD server>:<disk usage>:<failure group>:<NSD name>

#Tiebreaker disk

hdisk3:::descOnly:-1:nsd_tbrk_01:

#Data and metadata disk for /siccfs

hdisk4:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_01:

hdisk5:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_02:

The first disk (hdisk3) holds file descriptor data only and is used as a tiebreaker disk. hdisk4

and hdisk5 are designated as data disks for our LVM cluster file system. We use our quorum

nodes as primary and backup NSD servers as well, although all of our nodes do have physical

access to the disks. This provides some automatic fallback capability should the SAN access

become unavailable.

Create the NSD devices using the mmcrnsd command and verify your setup using the

mmlsnsd command (the option -v yes specifies that the NSD are to be created only if the disk

has not been formatted by a previous invocation of the mmcrnsd command):

root> mmcrnsd -F /var/mmfs/config/nsd.disks -v yes

root> mmlsnsd -aL

File system Disk name NSD volume ID Primary node Backup node

---------------------------------------------------------------------------------------------

siccfs nsd_siccfs_01 0999A547447D7EFE siccps12 siccps10

siccfs nsd_siccfs_02 0999A547447D7F00 siccps12 siccps10

(free disk) nsd_tbrk_01 0999A547447D7DA6 (directly attached)



The specified disk file is rewritten during NSD creation. Each hdisk entry is preceded by a

comment (# sign) and followed by the equivalent NSD entry. The new file can then be reused

for file system creation during subsequent configuration steps. The file system create

command uses NSD devices and not AIX hdisks.

3.2.1.2.4 Activate Node Quorum using Tiebreaker Disks

You need to shutdown the GPFS cluster and use the mmchconfig to activate the tiebreaker

disks. Use the following commands to activate the NSD device nsd_tbrk_01 as a tiebreaker

disk:

root> mmshutdown –a

root> mmchconfig tiebreakerDisks="nsd_tbrk_01"

root> mmstartup –a

You can use the mmlsconfig and mmgetstate commands to check your configuration after the

switch to tiebreaker mode:

root> mmlsconfig

Configuration data for cluster siccfs.isicc:

-------------------------------------------

clusterName siccfs.isicc

clusterId 691765743288319456

clusterType lc

autoload no

useDiskLease yes

maxFeatureLevelAllowed 903

tiebreakerDisks nsd_tbrk_01

[siccps12]

takeOverSdrServ yes

File systems in cluster siccfs.isicc:

------------------------------------

/dev/siccfs



------------------------------------------------------------------------------------

1 siccps12 1* 2 8 active quorum node


In the mmlsconfig output you can see that device nsd_tbrk_01 is used as tiebreaker disk. The

quorum information in the mmgetstate output is displayed as “1*”, which indicates that this is

a two node tiebreaker disk cluster.

3.2.1.2.5 Create GPFS File System

The last step is to create the file system itself. We used the following excerpt of the disk file

that was created by the mmcrnsd command as input to create file system command:

/var/mmfs/config/disks.siccfs:

#Description of disk attributes

#<disk name>:<primary NSD server>:<2ndary NSD server>:<disk usage>:<failure group>:<NSD name>

#Data and metadata disk for /siccfs

# hdisk4:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_01:

nsd_siccfs_01:::dataAndMetadata:1::

# hdisk5:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_02:

nsd_siccfs_02:::dataAndMetadata:1::

The mmcrfs command was used to create the /siccfs cluster file system:

root> mmcrfs /siccfs /dev/siccfs -F /var/mmfs/config/disks.siccfs –A automount -B 256K -v yes



We used the –A automount option to specify that the file system is automatically mounted

when its first accessed.

3.2.1.2.6 Mount GPFS File System

If you did not use the automount option during file system creation, you can use the standard

AIX mount command to mount the cluster filesystem:

root> mount /siccfs

3.2.1.3 Considerations for SAP Service Installation on GPFS

A SAP system has to be up and running before it can be registered and configured in the

Landscape Virtualization Management software. SAP LVM is not able to create the required

environment (mount target directories) for a new installation, this has to be done manually

(see also section 3.4.5). You should first create the directory structure on the GPFS file

system according to your naming conventions and then mount the subdirectories at the mount

points that are expected by the SAP installation routines.

The following commands create a sample directory structure for system C01:

mkdir /siccfs/sapC01

mkdir /siccfs/sapC01/oracle

mkdir /siccfs/sapC01/sapmnt

mkdir /siccfs/sapC01/usrsap

mkdir /siccfs/sapC01/trans

mkdir /siccfs/sapC01/c01adm

Then you need to create the required mapping before you can start the installation. This is

done with a normal mount command using the option –v mmfs (you may have to create some

of the directories for the mount points first, e.g. mkdir /home/c01adm).

mount -v mmfs /siccfs/sapC01/oracle /oracle

mount -v mmfs /siccfs/sapC01/sapmnt /sapmnt/C01

mount -v mmfs /siccfs/sapC01/usrsap /usr/sap/C01

mount -v mmfs /siccfs/sapC01/trans /usr/sap/trans

mount -v mmfs /siccfs/sapC01/c01adm /home/c01adm

After the installation is finished and the new SAP service is registered at the Solution

Manager, you should unmount all directories and proceed to the SAP LVM configuration

tasks.

3.2.1.4 Service Configuration in SAP LVM for GPFS

When you enable a SAP service in SAP LVM, you have to configure the data storage on the

Services tab in the Configuration menu (for further details see chapter 4.4).

Data residing on a GPFS file system has to be defined using the Storage Type DFS. The

Partner Id is ibm and FS/SRID Type is gpfs. Please note that these values are case sensitive

and have to be entered using lower case. The Export Path designates the directories as they

are stored in the GPFS file system and the Mount Point/SRID is the mount point as it is

expected by the SAP kernel. The last field is Mount Options, the available options for GPFS

file systems are fsmount and dirmount. The first one can be used to mount the GPFS file

system itself (in case you don‟t use the automount option) and the second option to specify

the subsequent subdirectory mounts.

For example to mount subdirectory /usr/sap/EC1 on a GPFS file system that doesn‟t use the

automount option, you would need the following two entries:



Storage Type DFS

Mount Point/SRID /siccfs

Export Path /dev/siccfs

Mount Options fsmount

FS/SRID Type gpfs

Partner ID ibm

Storage Type DFS

Mount Point/SRID /usr/sap/EC1

Export Path /siccfs/sapEC1/usrsap

Mount Options dirmount

FS/SRID Type gpfs

Partner ID ibm

Figure 10shows an example configuration of a GPFS based SAP system within the ISICC

SAP LVM landscape. The database used in this example. For DB2 based systems the

filesystems need to be created and mounted accordingly to the DB2 file system structure.

Note that a separate resource pool for SAP systems using the GPFS file systems was created

in SAP LVM.

Figure 10 Example configuration of an SAP system using GPFS

3.2.1.5 Add a node to GPFS Cluster

You need to add an AIX node to the GPFS cluster before you can mount the cluster file

system. This is done with the mmaddnode command after the GPFS software installation is

completed:



root> mmaddnode –N siccps18

You can remove a node from the cluster using the mmdelnode command and we recommend

that you remove the node before you start a NIM re-installation of a specific node.

You can list the state of cluster nodes with the mmgetstate command:



------------------------------------------------------------------------------------



3 siccps17 1* 2 8 active






3.2.2 SAN Storage Management

The SAP NetWeaver Landscape Virtualization Management allows the direct management of

natively attached storage without having the need of a shared file system like GPFS.

IBM provides a deep integration of IBM storage functionality with SAP LVM. This includes,

but is not limited to, creation and mapping of storage volumes to hosts or LPARs, discovering

the volumes within the operating system and create file systems, creation of snapshots of

volumes and mapping the snapshots to hosts / LPARs, and re-mapping of volumes.

This allows automated and fast moving, cloning and copying of SAP systems.

This chapter briefly describes the storage components and setup used in this PoC. You can

find the detailed description how the components were installed and configured in chapter

3.4.4.

3.2.2.1 Storage Components

Storage system:

IBM XIV Storage System

Firmware Version: 10.2.4a

79 TB usable space

A separate pool „ISICC_cloud‟ was created for the volumes of the test environment

Storage Management Software:

XIV CLI version (XCLI) version: 3.1 build 18

IBM Tivoli FlashCopy Manager: 3.1.0.1

3.2.2.2 SAP LVM SAN configuration and file systems

You need to install the SAP instances to be managed with SAP LVM according to the SAP

installation guides.

The following file system layout is recommended, and required if IBM Tivoli® Storage

FlashCopy® Manager is used to create a FlashCopy backup or clone of a SAP DB instance;

typically this applies to SAP production systems. The number of sapdata file systems, see

chapter 3.4.5.1, might vary, the default is four. Three different volume groups need to be

configured, as also shown in Figure 11:



Figure 11 Volume Group and Filesystem layout

List of file systems used: root> mount

node mounted mounted over vfs date options

-------- --------------- --------------- ------ ------------ ---------------

/dev/hd4 / jfs2 Jul 02 09:47 rw,log=/dev/hd8

/dev/hd2 /usr jfs2 Jul 02 09:47 rw,log=/dev/hd8

/dev/hd9var /var jfs2 Jul 02 09:47 rw,log=/dev/hd8

/dev/hd3 /tmp jfs2 Jul 02 09:47 rw,log=/dev/hd8

/dev/hd1 /home jfs2 Jul 02 09:48 rw,log=/dev/hd8

/dev/hd11admin /admin jfs2 Jul 02 09:48 rw,log=/dev/hd8

/proc /proc procfs Jul 02 09:48 rw

/dev/hd10opt /opt jfs2 Jul 02 09:48 rw,log=/dev/hd8

/dev/livedump /var/adm/ras/livedump jfs2 Jul 02 09:48 rw,log=/dev/hd8

siccserv /siccfs /home/siccfs nfs4 Jul 02 09:48 ro,bg,hard,intr,vers=4,sec=sys

/dev/lv10 /sapmnt/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv00

/dev/lv11 /usr/sap/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv00

/dev/lv13 /home/cl6adm jfs2 Jul 02 11:02 rw,log=/dev/loglv00

/dev/lv12 /sapinst jfs2 Jul 02 11:02 rw,log=/dev/loglv00

/dev/lv15 /usr/sap/DAA/SMDA97 jfs2 Jul 02 11:02 rw,log=/dev/loglv00

/dev/lv09 /db2 jfs2 Jul 02 11:02 rw,log=/dev/loglv01

/dev/lv16 /db2/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv01

/dev/lv01 /db2/CL6/db2cl6 jfs2 Jul 02 11:02 rw,log=/dev/loglv02

/dev/lv03 /db2/CL6/log_dir jfs2 Jul 02 11:02 rw,log=/dev/loglv03

/dev/lv04 /db2/CL6/log_archive jfs2 Jul 02 11:02 rw,log=/dev/loglv01

/dev/lv02 /db2/CL6/db2dump jfs2 Jul 02 11:02 rw,log=/dev/loglv01

/dev/lv05 /db2/CL6/sapdata1 jfs2 Jul 02 11:02 rw,log=/dev/loglv02




/dev/lv14 /var/db2 jfs2 Jul 02 11:02 rw,log=/dev/loglv01



List of volume groups used:

root> lsvg -l vg00 vg01 vg02 vg03

vg00:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

loglv00 jfs2log 1 1 1 open/syncd N/A

lv12 jfs2 256 256 1 open/syncd /sapinst

lv10 jfs2 251 251 1 open/syncd /sapmnt/CL6

lv11 jfs2 376 376 1 open/syncd /usr/sap/CL6

lv13 jfs2 8 8 1 open/syncd /home/cl6adm

lv15 jfs2 62 62 1 open/syncd /usr/sap/DAA/SMDA97

vg01:



lv14 jfs2 1 1 1 open/syncd /var/db2

lv02 jfs2 7 7 1 open/syncd /db2/CL6/db2dump

lv04 jfs2 254 254 1 open/syncd /db2/CL6/log_archive

lv09 jfs2 178 178 1 open/syncd /db2

lv16 jfs2 1 1 1 open/syncd /db2/CL6

vg02:



lv01 jfs2 1 1 1 open/syncd /db2/CL6/db2cl6

lv05 jfs2 313 313 1 open/syncd /db2/CL6/sapdata1




vg03:



lv03 jfs2 497 497 1 open/syncd /db2/CL6/log_dir

List of hdisks used:

root> lsvg -p vg00 vg01 vg02 vg03

vg00:

PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION

hdisk2 active 1019 65 00..00..00..00..65

vg01:


hdisk1 active 1019 577 77..00..92..204..204

vg02:


hdisk3 active 1279 25 00..00..00..00..25

vg03:


hdisk4 active 1019 521 113..00..00..204..204

The list of volumes defined on the XIV storage system for SID CL6 is shown in Figure 12.

It also shows the snapshots created during the cloning of SAP system CL6 as source system.



Figure 12 Volumes of SAP system CL6

The volume wwn is part of the hdisk unique_id.

Thisattribute is used to identify the hdisk on the storage system volume, as highlighted in

yellow and green:

XIV 1300360>>vol_list vol=CL6_datavg_1 -t wwn

WWN

00173800016800CA

root> lsattr -l hdisk3 -E -a unique_id

unique_id 2611200173800016800CA072810XIV03IBMfcp

Figure 13 shows the database instance of SAP system CL6 in LVM and the according mount

points.



Figure 13 SAP System CL6 in LVM and mount points

The host names defined on the XIV storage system to which the volumes are mapped must

exactly match the real host name / node name.

Every storage system must be accessible from any LVM managed node: For SAN Volume

Controller and Storwize V7000, enable and configure ssh. For XIV systems install the XIV

CLI on every node.

IBM Tivoli FlashCopy Manager (FCM) is used to manage the snapshots on the storage

systems for SAP system cloning and copying purposes.

The installation and configuration steps of the XIV CLI and the FCM are described in detail

in chapter 3.4.4).

3.3 Install Management Environment

3.3.1 Install SAP NetWeaver Landscape Virtualization Management Software

The current SAP NetWeaver Landscape Virtualization 1.0 Software is an add-on to a SAP

NetWeaver 7.3 system. You must install a SAP NetWeaver Application Server Java of usage

type “Basic” before you can deploy the SAP LVM software packages.

Please download the appropriate installation guide for your intended operating system and

database platform combination and follow the instructions in that guide to install a SAP

NetWeaver Java System:

http://service.sap.com/instguidesNW73

As always please pay particular attention to the SAP notes mentioned in the “SAP Notes for

the Installation” section of the installation guide.

After the installation and configuration of the SAP Java System is complete, you need to

download and deploy the SAP NetWeaver Landscape and Virtualization Management

Software.

http://service.sap.com/instguidesNW73



Please download the latest version of the installation guide for your licensed version (standard

edition or enterprise edition):

https://websmp201.sap-ag.de/instlvm

Again it is important to check all SAP notes mentioned on the download page and in the

installation guide itself.

This installation guide describes in detail the necessary steps to install LVM after the initial

setup of SAP NW 7.3 AS Java.

At a high level, the installation of a SAP LVM management system

1. Install SAP NW 7.3 AS Java with usage type “BASIC”

2. Download the required SCA files for SAP LVM from the SAP Service Marketplace

http://service.sap.com/swdc

Software Downloads Browse our Download Catalog SAP NetWeaver and

complementary products SAP NW LANDSC VIRT MGT ENT

3. Deploy the downloaded SCA files using one of the supported implementation tools

(JSPM or telnet)

4. Perform the initial setup using the configuration wizard as described in the post-

installation section

The last step is described in detail on the SAP Help Portal at:

http://help.sap.com/nwlvm10

Initial Configuration Carrying Out the Initial Setup Using the Configuration Wizard

Follow the procedure documented in the help portal to step through the configuration wizard.

After the wizard finished successfully, you should be able to access SAP LVM using the

following URL:

http://<host>:<port>/lvm

3.3.2 Install IBM Tivoli Storage FlashCopy Manager

SAP LVM exploits IBM Tivoli Storage FlashCopy Manager to perform snapshot backups for

file systems related to the SAP Systems. The FCM setup is done according to the FlashCopy

backup of "Custom application environments". The snapshots are managed as backup

versions using the IBM Tivoli Storage FlashCopy Manager management policies. The target

volumes are acquired and mounted on the target LPAR: These are the storage volumes for the

target system.

The Management Agent (acsd) of Tivoli Storage FlashCopy Manager is installed centrally on

one of the management LPARs: "acsd" coordinates the snapshot backup operation and

controls the backup flow and mediates between the other agents. The Management Agent

provides access to the snapshot backup repository, which contains information about the valid

snapshot backups and their relationships to the snapshot capable storage devices.

On each resource LPAR an “acsgen” agent is started during the SAP cloning/ SAP system

copy process: This agent perform the storage related tasks on the source and target resource.

https://websmp201.sap-ag.de/instlvm

http://service.sap.com/swdc

http://help.sap.com/nwlvm10



Figure 14 illustrates the environment:

VIOS

vFC (NPIV)

VIOS

vFC (NPIV)

Management Environment

SAP LVMAIX

Storage Manager native Interfacee.g. IBM XIV XCLI 3.1 build 18/opt/IBM/xiv/XIVGUI3.1/xcli

Managed Environment


acsd/home/sapadm/acs

SAP Hostagent/usr/sap/hostctrl

SAP SourceAIX



acsgen/home/sapadm/acs


SAP TargetAIX



acsgen/home/sapadm/acs


IBM XIV

SVC / Storwize

Source

Target

SAN Storage

• IBM XIV, IBM SVC, or IBM

Storwize V7000 Storage

System

• Storage connected via vFC

(NPIV)

• Compliant Storage Layout

Figure 14 Overview of the FlashCopy Manager installation

The Tivoli FlashCopy Manager installation will follow the same steps as described in chapter

3.4.4.8 for the setup for a "Custom application environments". Additionally, a profile template

needs to be created on the management node. The following includes a sample profile for the

IBM XIV Storage:

>>> GLOBAL

# ACS_DIR /home/sapadm/acs

ACSD siccplvm 57328

# TRACE NO

<<<

>>> ACSD

ACS_REPOSITORY /home/sapadm/acs/acs_rep

# REPOSITORY_LABEL TSM

<<<

>>> DEVICE_CLASS STANDARD

COPYSERVICES_HARDWARE_TYPE XIV

# CLONE_DATABASE NO

#STORAGE_SYSTEM_ID 3984

PATH_TO_XCLI /opt/IBM/xiv/XIVGUI3.1

COPYSERVICES_SERVERNAME siccxiv

COPYSERVICES_USERNAME edmund

BACKUP_HOST_NAME siccps98

# RECON_INTERVAL 12

# GRACE_PERIOD 24

# USE_WRITABLE_SNAPSHOTS AUTO

# USE_CONSISTENCY_GROUPS YES

<<<

>>> CLIENT

LVM_FREEZE_THAW NO

ENHANCED_PARTITIONING NO

<<<



SAP LVM takes this as a template and generates “runtime” templates during the actual

execution. The “ACSD” entry in the GLOBAL section of the template needs to refer to the

hostname of the management system running the FCM acsd agent. SAP LVM may replace the

port number (2nd argument) during the actual run.

The ACS_REPOSITORY in the ACSD section will refer to the repository directory.

In the DEVICE_CLASS section, ensure that the following settings are included for IBM XIV

storage:

COPYSERVICES_HARDWARE_TYPE needs to be set to XIV

PATH_TO_XCLI needs to point to the installation directory of the XIV XCLI (see also

chapter 3.4.4.7)

COPYSERVICES_SERVERNAME needs to include the hostname / IP address of the XIV

system

COPYSERVICES_USERNAME needs to contain an administrative user on the IBM XIV

system. IBM Tivoli Storage FlashCopy Manager will use this user to communicate

with the IBM XIV System. This user needs to be created on the IBM XIV system first,

and the password needs to be entered later on during the FCM setup.

In the CLIENT section, the two entries LVM_FREEZE_THAW NO

ENHANCED_PARTITIONING NO

need to be specified.

3.3.3 Install IBM Systems Director

The IBM Systems Director installation package is available for download at the Systems

Director home page:

http://www-03.ibm.com/systems/software/director/

Select Resources Downloads Management Servers and choose the installation package

or DVD image for your intended target environment. Please use the current version 6.3 and do

not start with the older releases anymore.

The installation is dependent on the chosen platform and is fully documented at the Systems

Director Information Center:

http://pic.dhe.ibm.com/infocenter/director/pubs/index.jsp?topic=%2Fcom.ibm.director.main.h

elps.doc%2Ffqm0_main.html

Please follow the steps for “Installing on the management server” in the Planning and

Installing sections of the information center.

A number of short hands-on product demonstration movies for the Systems Director

installation and first use are available at:

http://www.ibm.com/developerworks/wikis/display/WikiPtype/Movies#Movies-sectiondir63

3.3.4 Install IBM Systems Director Plug-ins

Additional plug-ins are available on the Systems Director download page and might have to

be downloaded and installed separately. Please select the appropriate version that is supported

for Systems Director version 6.3.

3.3.4.1 IBM Systems Director VMControl

The current version of VMControl is already installed with IBM Systems Director

management server installation package. The further steps for discovery of the environment

and the base setup for OS deployment are described later in chapter 4.5.

http://www-03.ibm.com/systems/software/director/

http://pic.dhe.ibm.com/infocenter/director/pubs/index.jsp?topic=%2Fcom.ibm.director.main.helps.doc%2Ffqm0_main.html

http://pic.dhe.ibm.com/infocenter/director/pubs/index.jsp?topic=%2Fcom.ibm.director.main.helps.doc%2Ffqm0_main.html

http://www.ibm.com/developerworks/wikis/display/WikiPtype/Movies#Movies-sectiondir63



3.3.5 LDAP Server

3.3.5.1 User Management

A SAP LVM infrastructure requires a method to synchronize naming services on all enabled

servers. This includes information on users, groups, protocols, services, hosts, etc. Possible

solutions are NIS/NIS+ or the synchronization of local /etc files using tools like rdist.

However, development of NIS has stopped and the recommended solution for centralized user

management is LDAP (Lightweight Directory Access Protocol). LDAP is widely used and

has become the standard protocol for access to information registries, optimized for read

mostly access.

AIX clients can work with any LDAP server as long as it is V3 conform. In our sample

implementation, we implemented the IBM Tivoli Directory Server V6.3 as LDAP server.

More information is available at:

http://www-01.ibm.com/software/tivoli/products/directory-server/

The following introduces some of the LDAP terminology and discusses briefly some

important steps for planning and running LDAP on AIX. This is no replacement for the

product documentation. Especially an enterprise wide usage of LDAP sharing directory

information for other services requires careful planning and you need to consider issues like

namespaces, secure authentication using SSL, and implementing high availability using server

replication. These topics are fully covered in the product documentation and the available

Redbooks and whitepapers.

3.3.5.2 Schema

A schema defines which objects and attributes can be stored in a Directory Information Tree

(DIT). It defines the names of objects and attributes, syntax, and possible values as well as

whether attributes are required or optional for an object. For example, an object could be

people or server and attributes of the objects could be name, homedirectory, uid or ip-

address, hostname.

The following schemas are available for AIX user management and authentication:

AIX The AIX schema includes the aixAccount and aixAccessGroup object classes. This

schema offers the entire AIX user and group attributes. This schema is included to support

legacy LDAP installations prior to Version 5.2.

RFC2307 The RFC2307 schema includes the posixAccount, posixGroup and other NIS

related object classes. This experimental RFC defines a schema that allows NIS maps to be

imported into LDAP. RFC2307 only defines a subset of the AIX user and group attributes.

This schema supports any RFC2307 compliant platforms and AIX 5L Version 5.2.

RFC2307AIX The RFC2307AIX schema includes the RFC2307 schema plus the AIX

specific object classes, aixAuxAccount and aixAuxGroup. The AIX specific object classes

provide attributes to store additional attributes not defined by the RFC2307 standard.

The recommended schema for AIX5.2 and higher is RFC2307AIX and has been used for the

SAP LVM reference implementation. This has several advantages such as RFC2307 is

considered a standard for UNIX systems including Linux.

http://www-01.ibm.com/software/tivoli/products/directory-server/



3.3.5.3 Namespace

The LDAP naming model defines how entries are identified and organized. Entries are

organized in a tree-like structure called the Directory Information Tree (DIT) based on their

distinguished name (DN). A DN is a unique name that unambiguously identifies a single

entry. Distinguished Names are made up of a sequence of relative distinguished names

(RDN). Each RDN in a DN corresponds to a branch in the DIT leading from the root of the

DIT to the directory entry.

A RDN typically has the form <attribute name> = <value>. A DN is composed of a sequence

of relative distinguished names separated by commas.

The following DIT shows a sample DIT with a suffix o=isicc,c=de:

Figure 15 Sample LDAP Directory Information Tree for AIX user management

3.3.5.4 LDAP Server Installation and Configuration

For LDAP server implementations please review and follow the instructions in the mentioned

references.

3.3.5.5 LDAP client Installation

AIX LDAP client is available in a 32-bit and a 64-bit version. Their filesets are shipped with

the AIX CDs and should be included in the install images for the SAPL LVM client nodes.

3.3.5.6 LDAP client configuration

mksecldap is an AIX command for IBM Directory server and client setup. Please check the

man pages or the online documentation for the full syntax description.

The mksecldap command performs the required updates in various configuration files, starts

the secldapclntd daemon, and puts an entry in /etc/inittab so that the seclpdapclntd daemon

will be automatically started after reboot.

The following command is used in our demo environment to configure an AIX LDAP client:

/usr/sbin/mksecldap -c -h siccldap1,siccldap2 -a cn=root -p ldappw

There are a number of commands available to control the client daemon:

c=de

o=isicc

cn=user cn=group

uid=c01adm

uid=orac01

gid=sapsys

gid=dba



start-secldapclntd start ldap client daemon

stop-secldapclntd stop ldap client daemon

restart-secldapclntd restart the ldap client daemon

ls-secldapclntd show the status of the ldap client daemon

flush-secldapclntd clear the cache of the ldap client daemon

3.3.5.7 User Authentication

The last step required to setup user login through LDAP is to change a user‟s authentication

mechanism. User authentication is controlled by the SYSTEM attribute in the

/etc/security/user file. For example to enable LDAP user c01adm to login to the local system,

you could execute the following command;

chuser SYSTEM=LDAP registry=LDAP c01adm

To allow all LDAP users to login to the system, one can set each LDAP user's SYSTEM

attribute to LDAP, or simply set the default stanza's SYSTEM attribute to LDAP. This has to

be done manually in the /etc/security/user file, where you need to change the following two

attributes in the default stanza.

Default:

SYSTEM = "LDAP or compat"

registry = LDAP

We recommend using local authentication for the root user so that you can still login to the

server in case there is something wrong with the LDAP setup. With local authentication the

root user‟s stanza should look like:

root:

SYSTEM = "compat"

registry = files

In our sample environment the complete client setup is automated in a NIM script. The script

uses mksecldap to configure the client and then runs a Perl command to automatically change

the default and root user‟s stanza to the appropriate:

3.3.5.7.1 Sample customizing script for automated LDAP client setup

#Setup LDAP Client

#AIX users are managed by LDAP

/usr/sbin/mksecldap -c -h siccldap1,siccldap2 -a cn=root -p ldappw

perl -p -i -e 'if ( /default:/ .. /SYSTEM =/ ) {

s/"compat"/"LDAP or compat"\n\tregistry = LDAP/;

} elsif ( /root:/ .. /SYSTEM =/ ) {

s/"compat"/"compat"\n\tregistry = files/;

}' /etc/security/user

3.3.5.8 Create Users and Groups

The AIX mkuser command is able to create a user on the LDAP server using the load module

option “-R LDAP”. Unfortunately the “smit mkuser” panel doesn‟t allow for this option. You

should create a new user using the mkuser command. Once a user is created, you can use the

“smit chuser” panel to change user attributes. For example to create a user c01adm, use the

command

mkuser –R LDAP registry=LDAP c01adm



Then use the “smit chuser” panel to change the other required attributes (of course, you could

also set them directly using the appropriate attribute options with the mkuser command).

The problem also applies to the mkgroup command, so you have to create the required groups

manually as well: mkgroup –R LDAP sapsys

Consider using some naming conventions for the numerical user id, for example by reserving

certain ranges for the SAP and database administration users:

mkgroup -R LDAP registry=LDAP id=400 dbc01adm

mkgroup -R LDAP registry=LDAP id=401 dbc01ctl

mkgroup -R LDAP registry=LDAP id=402 dbc01mnt

mkgroup -R LDAP registry=LDAP id=403 dbc01mon

mkgroup -R LDAP registry=LDAP id=404 sapsys

mkgroup -R LDAP registry=LDAP id=405 sapinst

mkuser -R LDAP registry=LDAP id=400 pgrp=sapsys

groups='sapsys,dbc01ctl,dbc01mon,sapinst' home='/home/c01adm' gecos='SAP

System Administrator' shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1

core=2097151 c01adm

mkuser -R LDAP registry=LDAP id=401 pgrp=dbc01adm groups='dbc01adm'

home='/db2/db2c01' gecos='SAP Database Administrator'

shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 db2c01

mkuser -R LDAP registry=LDAP id=402 pgrp=dbc01mnt groups='dbc01mnt'

home='/home/sapc01' gecos='ABAP Database Connect User'

shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 sapc01

mkuser -R LDAP registry=LDAP id=403 pgrp=dbc01mon groups='dbc01mon'

home='/home/sapc01db' gecos='Java Database Connect User'

shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 sapc01db

3.3.6 Other Central Management Components

3.3.6.1 OS Management

We highly recommend a centralized OS software management concept. There are two

common solutions to manage the installation and maintenance of AIX images in a server

landscape. The first one is using the AIX built-in Network Installation Management (NIM)

features. The second one is to use the VMControl plugin of IBM Systems Director.

VMControl uses NIM under the cover as well, but will simplify some of the administration

tasks.

NIM allows the installation and maintenance of AIX base operating system and additional

software from a central repository. As images are sent across the network to the partition to be

installed, the installation steps can be performed fully remotely without needing physical

access to the server. During and after base installation, you can customize the machines by

running specific customizing scripts. Later on, you can deploy fixes and new software images

to the server partitions from a central administration point.

NIM supports several different machine types: diskless, dataless and standalone. A diskless

machine has no own disks, all file systems and other resources are installed remotely and

accessed via the network. A dataless machine has local paging and dump devices, the file

systems for / and /usr are located on a remote server, all the other file systems are located

either local or remote. On a standalone machine, all file systems and resources are installed

on local disks. Due to performance and availability reasons, we recommend using local (or



SAN attached disks) on the computing nodes and install them as standalone machines. It is

possible to use diskless or dataless machines for test and development systems. They boot of

the network and mount the OS from the NIM server.

The following two Redbooks describe in detail the concepts of centralized OS management

using NIM or VMcontrol:

NIM from A to Z in AIX 5L

http://www.redbooks.ibm.com/abstracts/sg247296.html?Open

IBM Systems Director VMControl Implementation Guide on IBM Power Systems


3.3.6.2 Network Infrastructure

There are no special network infrastructure requirements for an SAP LVM landscape.

Common standard today and recommended environment is a switched Gigabit Ethernet

network infrastructure.

The major difference compared to a traditional SAP implementation is the consequent usage

of virtual IP names and addresses assigned to a SAP service. These addresses need to be

implemented in the customer‟s name services infrastructure, typically based on DNS (please

refer to the section 3.1. for some recommendations about naming of virtual IP names). If the

decision is to pick virtual network addresses in new subnets, one might have to update router

tables, saprouter definitions, etc. to ensure that the end user can access the SAP service using

the virtual IP name. You also have to change SAP logon groups to reflect the virtual IP

names.

Note 962955 - Use of virtual TCP/IP host names

3.4 Install Managed environments

The following section provides an overview about the preparation steps required for the

environment. This includes considerations about

PowerVM / VIOS

IBM Systems Director

IBM HMC

OS setup for the resources

SAP Installation

3.4.1 VIOS/ PowerVM

Base assumption is that the volume(s) for the AIX operating system (rootvg) is/are located on

the same storage system like the SAP system also, and that the “rootvg” storage volumes are

also attached via virtual fibre channel (vFC). In this case, the “host” and “adapter” definition

in the storage system required for the access to the “rootvg” volumes will ensure the required

connectivity to the SAP application related volumes in the storage system also.

3.4.2 IBM Systems Director Setup

The following describes high-level configuration steps required for the integration of either

IBM Systems Director or IBM FlexSystem Manager with SAP LVM: Further details are






described in SAP Note 1728222 - Configuration of IBM Systems Director VMControl for

SAP LVM

In the text the term “virtualization manager” is applied for both of the integration scenarios:

Both “IBM Systems Director” and “IBM FlexSystem Manager” require the identical steps for

the setup. The tasks on the virtualization manager include:

Discover the IBM HMCs on the virtualization manager and configure access to them.

Perform a "collect inventory" task on the virtualization manager for IBM HMC, IBM

Power servers, and for all logical partitions (LPARs).

Discover operating systems running on LPARs, configure access and perform "collect

inventory" task on virtualization manager for all the discovered operating systems.

Define a user for of SAP LVM access on the virtualization manager

The default administrator user can be used as user for SAP LVM too: The

administrators in SAP LVM then have the same credentials and get data for all the

IBM Power Systems configured in the virtualization manager. To restrict the access to

the specific set of IBM Power Systems that are desired to be managed by SAP LVM

(and so from SAP LVM administrators) a specific user can be created in the

virtualization manager: This user doesn‟t have all the credentials for the whole set of

IBM Power Systems defined in the virtualization manager, but only to those in scope

of the SAP LVM environment. The configuration is described in chapter 4.5.

Create a security certificate on the virtualization manager

Per default, the virtualization manager uses Port 8422 for communication:

Communication via this port must be allowed in the network and firewall settings for

the environment.

o Open the URL “https://<virtualization manager host>:8422" in a browser

window. During the first connection attempt the browser displays a notification

window to accept the certificate. This virtualization manager certificate is

stored in the internet browser configuration locally, and can be exported as

certificate file.

o The virtualization manager certificate needs to be imported into the SAP

NetWeaver CE system where the SAP LVM software is deployed to.

The certificate store must be "TrustedCAs".

Check that the hostname in the certificate is valid.

For more information on this topic see also the SAP documentation "Using the

AS Java Key Storage".

Using the virtualization manager for AIX OS provisioning requires to discover, to

access and to perform "collect inventory" tasks for NIM, VIO servers, FC switches

and storage managers in addition.



http://help.sap.com/saphelp_nwce10/helpdata/en/e9/a1dd44d2c83c43afb5ec8a4292f3e0/content.htm

http://help.sap.com/saphelp_nwce10/helpdata/en/e9/a1dd44d2c83c43afb5ec8a4292f3e0/content.htm



It must be possible to

capture and deploy an AIX operating system via NIM (4.5).

select a particular IBM Power server as target for deployment

select a storage system for deployment

within the different layers of the virtualization manager (VMControl, Storage

Control). Then similar provisioning operations can be triggered out of SAP LVM.

Using the virtualization manager for LPAR relocation (Live Partition Mobility)

requires configuration of “virtual farms”. If it is intended to migrate LPAR "X" hosted

on IBM Power system "A" to IBM Power system "B", then both Power system "A"

and Power system "B" must be members of the same virtual farm.

If must be possible to relocate a LPAR using virtual farms out of the virtualization

manager. Then the same operation can be triggered out of SAP LVM.

3.4.3 HMC setup

The recommended setup for SAP LVM is to use IBM Systems Director (or IBM FlexSystem

Manager) as Virtualization Manager. However the management via IBM Power HMC

instead of IBM systems Director is possible too. This is described in detail in SAP Note

1728293 - Configuration of IBM Power HMC for SAP LVM.

3.4.4 Install compute nodes

This section describes the base configuration steps on an AIX operating system level for the

compute nodes. All the required steps are described, and need to be executed at least on the

first system image: This image then can be captured (NIM, mksysb) and can be used as

“golden image” template for the deployment of all the additional compute nodes. Main steps

include

Activate IPSec

Check/ Update DNS configuration

Configuration of the LDAP client

Adapt /etc/services base configuration

Check/adapt filesystem sizes

Install SAP Host Agent package and IBM libraries

Install XIVGUI (IBM XIV Storage systems)

Install IBM Tivoli Storage FlashCopy Manager

3.4.4.1 Activate IPSec

Each of the resource LPARs that may participate in a system provisioning use case for SAP

system cloning or SAP system copy needs to be prepared for the network isolation. In an AIX

environment, the network isolation is provided via IPSec: IP Security, known commonly as

IPSec, is a protocol developed by the Internet Engineering Task Force (IETF), designed to

provide "end-to-end" cryptographically-based security for IP network connections. IPSec

implementations include a method of restricting connections to various services, based on

their origin and destination. This feature is known as packet filtering. All packets on an IP

network originate from an IP address and a port, and are destined for another IP address and

port. A packet filter is a physical or virtual device that sits between the endpoints of a

connection and determines whether the packet should be permitted to continue to its

destination. The decision is made by comparing various attributes of the packet to rules that

are defined by the administrator of the packet filter. Those attributes include source address





and subnet, source port, destination address and subnet, destination port, protocol, direction of

the connection, and fragmentation of the packet.

Make sure that APAR IZ98741 "DYNAMIC FILTER RULE ID NOT MODIFIED BY

RMFILT -N ALL" is installed in the AIX OS image

IPSec is activated via the AIX SMIT tool using fastpath “ips4_start”:

# smitty ips4_start

And then switching to the next panel:

Figure 16 Start IPSec

Alternatively, IPSec can be started directly via the command line:

# /usr/sbin/mkdev -c ipsec -t 4

No manual definitions are required for packet filters: The packet filters for the system

isolation are defined/released automatically during the execution of the System Copy/ System

Clone workflows by SAP LVM.

Figure 17 illustrates an additionally required step during the resource definition for the

resource LPAR. The checkbox “Isolation Ready” must be enabled so that the LPAR can

participate as a target resource in a System Copy/System Clone scenario:



Figure 17 Define the Resource as “Isolation Ready” in SAP LVM

This is described in detail in Chapter 4.3

3.4.4.2 DNS Name resolution

Ensure that the search order for DNS name resolution is first based on the local /etc/hosts file,

and then on the external DNS nameserver.

#vi /etc/netsvc.conf

hosts = local,bind4

In our test environment, the content of the /etc/host file includes the IP label for the hostname

of the resource only. All external addresses are resolved solely via DNS. In this case,

communication to the DNS server needs to be allowed during the “isolated”, “network

fenced” state within the System Clone / System Copy workflow also: The required settings

for allowing DNS nameserver communication are is illustrated in chapter 4.4, Figure 41.

3.4.4.3 Configure LDAP client

As prerequisite for the use of LDAP the IBM Tivoli Directory Server Client needs to be

installed on each LPAR.

idsldap.clt32bit61

idsldap.clt64bit61

The LDAP client needs to be defined on the LPAR in the same way as described in chapter

3.3.5.5 and 3.3.5.6.

3.4.4.4 Adapt /etc/services entries

Table 8 illustrates the ports to be predefined for ABAP instances. For the sake of simplicity

all ports covering the full instance number range from 00 to 99 are added to /etc/services.

In total 400 ports for sapdpXX, sapgwXX, sapdpXXs and sapgwXXs are added. If only a

subset of the ports is required in the organization (e.g. due to SAP instance number standards)

then it is sufficient to add only those specific ports to the /etc/services file.



Port Description /etc/services content

sapdpXX

XX { 00, 01, 02, … 99 }

SAP System Dispatcher Ports sapdp00 3200/tcp

…

sapdp99 3299/tcp

sapgwXX

XX { 00, 01, 02, … 99 }

SAP System Gateway Ports sapgw00 3300/tcp

…

sapgw99 3399/tcp

sapdpXXs

XX { 00, 01, 02, … 99 }

SAP System Dispatcher Security Ports sapdp00s 4700/tcp

…

sapdp99s 4799/tcp

sapgwXXs

XX { 00, 01, 02, … 99 }

SAP System Gateway Security Port sapgw00s 4800/tcp

…

sapgw99s 4899/tcp

Table 8 Entries in /etc/services

3.4.4.5 Adapt filesystem sizes

Several software components need to be available on each of the LPARs of the managed

environment. If all the software components are installed locally on the LPAR, then sufficient

free space is required in certain filesystems for the management components. Table 9 gives an

example about required sizes for the components in the “local” filesystems/ directories:

Directory Size

(approx.)

MByte

VG Content

/usr/sap/hostctrl 200 rootvg or saplocalvg

SAP Host Agent Package, together with the IBM libraries

/opt/IBM/xiv/XIVGUI3.1 (example) 350 rootvg XIV XCLI Installation

/usr/tivoli/tsfcm 700 rootvg Installation directory for IBM Tivoli Storage

/home/sapadm/acs (example) 700 rootvg or saplocalvg

Work directory for IBM Tivoli Storage FlashCopy Manager (SAP LVM context)

Table 9 Local Filesystem layout

Increase the size of the /usr filesystem to allow to install the SAP Host Agent package

and the IBM Tivoli Storage FlashCopy Manager software (FCM)

Increase the size of the /home filesystem for the copy of the FCM package in

/home/sapadm/acs

Increase the size of the /opt filesystem for the XIVGUI package

# chfs -a size=+1500M /usr

# chfs –a size=+1G /home

# chfs -a size=+1G /opt



3.4.4.6 Install SAP Host Agent and IBM partner libraries

The runtime user for the SAP Host Agent is user “sapadm”: SAP LVM uses this user id to

authenticate to the SAP Host Agent on the managed system. Base assumption is that user

“sapadm” has been created in the LDAP system and is available on all the compute nodes

before the SAP Host Agent installation.

The SAP Host Agent package requires “root” user credentials for the installation.

The group “sapsys” has to exist on the host. If it doesn‟t exist, create the group, for

example with the command “mkgroup”.

The user “sapadm” has to exist on the host, and must be a member of the “sapsys”

group. User “sapadm” is used for authentication from SAP LVM to the host. If the

user does not yet exist, create the user, for example with the command “mkuser”.

Create home directory for sapadm on the resource LPAR. The FCM setup on the

managed LPAR for SAP LVM is located below the home directory of user sapadm

#mkdir /home/sapadm

#chown sapadm.sapsys /home/sapadm

Download the recent SAPHOSTAGENT.SAR archive and uncompress it using the

SAPCAR tool.

The archive contains the “saphostexec” program. The SAP Host Agent package is

installed/ upgraded by executing “saphostexec –install”, respectively “saphostexec –

upgrade” as root user. Install the SAP Host Agent package to directory

/usr/sap/hostctrl.

Some SAP Host Agent runtime executables have the S-User bit set. So privileged

commands can be executed during the prepare/ unprepare operations.

Copy the IBM specific libraries for the SAP Host Agent

# mkdir /usr/sap/hostctrl/exe

# mkdir /usr/sap/hostctrl/exe/operations.d

# mkdir /usr/sap/hostctrl/exe/operations.d/IBM

# mkdir /usr/sap/hostctrl/exe/operations.d/IBM/traces

SAP LVM calls the IBM storage library for mount/ unmount operations as hooks on top

of the ACOSPrepare and PostACOSUnprepare steps. During the execution of these hooks

the script /usr/sap/hostctrl/exe/operations.d/IBM/ibm_lvm_mgmt.ksh is called.

Verify that the settings for ACOSPrepare and PostACOSUnprepare hooks are

included in the profile /usr/sap/hostctrl/exe/host_profile of the SAP Host Agent.



Figure 18 illustrates the configuration for the host_profile:

Figure 18 Configuration of host_profile

o Ensure that the script ibm_lvm_mgmt.ksh is located in the

/usr/sap/hostctrl/exe/operations.d/IBM directory, that it is readable and has the

execute bit set.

SAP LVM calls to the IBM Tivoli Storage FlashCopy Manager (FCMCLI) are

managed as a “Host Agent Registered Script” (See SAP Note #1465491 for details).

Make sure that the configuration files FCMCLI.conf is available in the

/usr/sap/hostctrl/exe/operations.d directory on all the resource LPARs, and on the

management server hosting SAP LVM (Figure 19):

Figure 19 Content of .conf file FCMCLI.conf

Make sure that the file fcmcli.sh exists in directory

/usr/sap/hostctrl/exe/operations.d/IBM and is readable and executable.

SAP LVM calls to the XIV command line interface (XCLI) are managed as a “Host

Agent Registered Script” (See SAP Note #1465491 for details). Make sure that the

.conf files XCLI.conf is available in the /usr/sap/hostctrl/exe/operations.d on the

management server hosting SAP LVM (Figure 20)



Figure 20 Content of .conf file XCLI.conf

Make sure that the file xcli.sh exists in directory

/usr/sap/hostctrl/exe/operations.d/IBM on the management server hosting SAP LVM

and is readable and executable.

Start/ Restart the SAP Host Agent

If there were any changes to the profile or additional configuraton files, restart the

host agent on the affected LPAR.

Additional considerations for the SAP Hostagent are found in the SAP Note

Note 1292836 - Logfiles of the SAP Hostagent

3.4.4.7 Install XIVGUI

In the PoC environment XIVGUI 3.1 build 18 is installed. The XIVGUI image is deployed to

the directory /opt/IBM/xiv (XIVGUI is part of the OS image).

Parameter Value

XCLI Installation Directory /opt/IBM/xiv/XIVGUI3.1

Table 10 XIVGUI Install directory

At first create the target directories for the XIVGUI image

mkdir /opt/IBM/xiv

mkdir /opt/IBM/xiv/XIVGUI3.1

Install additional rpm packages for the XIVGUI image.

Although this is a prerequisite this is not explicitly mentioned in the XIV XCLI

installation guide: Required rpm packages include:

o libgcc

o libstdc

o readline-5

readline-5 additionally requires

o bash

o info

#rpm -i libgcc-4.2.0-3.aix6.1.ppc.rpm

#rpm -i libstdc++-4.2.0-3.aix6.1.ppc.rpm

#rpm -i gcc-4.2.0-3.aix6.1.ppc.rpm

#rpm -i bash-4.2-6.aix5.1.ppc.rpm

#rpm -i info-4.6-1.aix5.1.ppc.rpm

#rpm -i readline-5.2-3.aix5.1.ppc.rpm




Adjust the library settings and the LIBPATH to include these shared libraries

#cd /opt/freeware/lib

#ln -s /opt/freeware/lib/gcc/powerpc-ibm-aix6.1.0.0/4.2.0/libgcc_s.a

#ln -s /opt/freeware/lib/gcc/powerpc-ibm-aix6.1.0.0/4.2.0/libstdc++.a

#export LIBPATH=$LIBPATH:/opt/freeware/lib

Extract the XIVGUI software from the Software Image Repository

# ./xivgui-3.1-build18-aix.bin

Adapt the xcli script

There is a bug in the xcli script: xcli can be started only out of the installation path.

xcli needs to be patched to be able to start it from any location.

Edit the xcli script and add the line

prg="$0"

in the upper section of the xcli script

Validate the XCLI by a test connection to the IBM XIV storage system

# /opt/IBM/xiv/XIVGUI3.1/xcli

User Name: ...

Password: ...

Machine IP/Hostname: ...

connecting.

XIV 1300360>>

3.4.4.8 Install IBM Tivoli Storage FlashCopy Manager

SAP LVM exploits IBM Tivoli FlashCopy Manager for the backup of "Custom application

environments". On the managed systems, the “acsgen” agent is used to invoke the different

storage related activities on the LPAR during the System Copy/ System Cloning process.

Base Installation of IBM Tivoli Storage FlashCopy Manager

./3.1.0.1-TIV-TSFCMFTP-AIX.bin

Parameters:

- Choose Locale: 2- English

- PRESS <ENTER> TO CONTINUE

- Press Enter to continue viewing the license agreement, or enter "1" to

accept the agreement

- Choose the Install Set to be installed by this installer: 4 - IBM

Tivoli Storage FlashCopy (R) Manager (Custom Applications)



- ENTER AN ABSOLUTE PATH: /usr/tivoli/tsfcm/acs_3.1.0.1

- IS THIS CORRECT? (Y/N): Y

- PRESS <ENTER> TO CONTINUE

- PRESS <ENTER> TO EXIT THE INSTALLER

Check the license file (contained in IBM Tivoli FlashCopy Manager base installation).

If the FCM 3.1.0.1 package was downloaded as an update package, it will not include

the license file. The license file included in the FCM 3.1.0.0 base package needs to be

copied to the directory:

o Copy license file “tsmacs.lic” from the FCM 3.1.0.0 base package to the FCM

install directory /usr/tivoli/tsfcm/acs_3.1.0.1

Create Directory for FCM profile / environment:

mkdir /home/sapadm

chown sapadm:sapsys /home/sapadm

mkdir /home/sapadm/acs

chown sapadm:sapsys /home/sapadm/acs

o Run the setup for FCM custom application

cd /usr/tivoli/tsfcm/acs_3.1.0.1

./setup_gen.sh

Script not started from 'acs' sub-directory of the Home directory of a

custom application user. Trying to locate the correct Home directory or

install IBM Tivoli Storage FlashCopy(R) Manager in a new Home directory ...

Please enter the Home directory:

/home/sapadm

This is a new installation of IBM Tivoli Storage FlashCopy(R) Manager into

this Home directory.

Please enter the name of the user of this Home directory:

sapadm

Please enter the group name of the user:

sapsys

Installing/Updating IBM Tivoli Storage FlashCopy(R) Manager installation

for /home/sapadm/acs

Checking environment

- - - - - - - - - - -

+ OK: installation directory exists.



+ OK: setup_gen.sh runs in IBM Tivoli Storage FlashCopy(R) Manager

installation directory '/usr/tivoli/tsfcm/acs_3.1.0.1'

+ OK: IBM Tivoli Storage FlashCopy(R) Manager installation will be

installed to '/home/sapadm' for the first time.

Installation messages

- - - - - - - - - - -

+ OK: New installation !

+ OK: IBM Tivoli Storage FlashCopy(R) Manager directory '/home/sapadm/acs'

created successfully.

+ OK: Files successfully copied.

checking /home/sapadm/acs/acsgen ...

OK

Select one of these configurations:

(1) On-Site Production System configuration with optional remote Backup

System configuration

(2) On-Site Backup System configuration

Enter '1' to configure IBM Tivoli Storage FlashCopy(R) Manager on the

production server with the option to remotely synchronize the configuration

of one or multiple backup systems using Secure Shell.

Enter '2' to configure IBM Tivoli Storage FlashCopy(R) Manager on the

backup system as a 'separate installation'.

2

Going to install on backup server

Please enter the directory where the profile will be stored:

[/home/sapadm/acs]

Deactivating IBM Tivoli Storage FlashCopy(R) Manager management daemon.

Are you going to perform off-loaded backups to Tivoli Storage Manager?

[y|n]

n

Profile configuration

=====================

Special commands:



'?' ==> Show help for current parameter

'!d' ==> Delete current parameter instance (only applicable to

multi-instance parameters)

FMM8422I Creating new profile '/home/sapadm/acs/profile' for application

'GENERIC' ...

****** Profile parameters for section GLOBAL : ******

Path of the IBM Tivoli Storage FlashCopy(R) Manager directory {ACS_DIR} =

[/home/sapadm/acs]

Hostname and port of machine running Management Agent {ACSD} (<hostname>

<port>) = [] siccplvm 57328

Low level tracing {TRACE} (YES|NO) = [NO]

FMM8426I Saving profile '/home/sapadm/acs/profile' ...

FMM1555I Profile successfully created.

Please enter the password for authentication with the ACS daemon:

Please re-enter password for verification:

Creating password file at /home/sapadm/acs/shared/pwd.acsd.

A copy of this file needs to be available to all components that connect to

acsd.

Enter the device classes to use for this Backup System. Make sure each

device class is separated by a comma.

STANDARD

IBM Tivoli Storage FlashCopy(R) Manager requires at least two daemon

processes to be running at all times.

IBM Tivoli Storage FlashCopy(R) Manager can add the necessary daemon

processes to the inittab.

Alternatively you may choose to start and stop these processes yourself,

for instance if you want to include the processes in your HA policy.

Do you want IBM Tivoli Storage FlashCopy(R) Manager to create the inittab

entries for you? [y|n]

n

Please run the following daemon processes:

/home/sapadm/acs/acsgen -D -M -s STANDARD

Please make sure that the command '/home/sapadm/acs/tsm4acs' is started by

a scheduler or manually.



No special customization for the FCM profile is required on the resource LPAR.

SAP LVM will derive the required content for the profile out of the master profile on the

FCM management LPAR (“acsd”), and copy the profile to the source and target resource

during the execution of the SAP System Cloning/SAP System Copy workflow.

o Tivoli Storage FlashCopy Manager password file

The FlashCopy Manager configuration on the central node (acsd) generates a password file in

/home/sapadm/acs/shared/pwd.acsd. The password file needs to be copied to all the

FlashCopy Manager directories on the target resources:

It contains the connection password to the acsd, and also the connection password to the

Storage System.

3.4.5 SAP System

In the following section, base considerations for the installation of a new SAP system, as well

as the eventually required modifications of an existing SAP system are given. These

considerations need to be taken into account, before the SAP system then can be registered

with SAP LVM. The definition of the new system in the SAP LVM context is described in

chapter 4.4

3.4.5.1 Filesystem considerations for the SAP Systems (SAP System Copy)

The two tables below illustrate two different storage layouts for a SAP System based on DB2

UDB. The first example (Table 11) includes a simple layout, having a few different

filesystems only and just separating between SAP application and database filesystems. The

filesystem layout fullfills the requirements for the relocation and SAP system copy scenarios

with SAP LVM.

Filesystem Volume Group Content

/db2 vg01 Contains the instance directory /db2/db2<sid> and all other database related directories. All DB2 database specific data is stored in a directory structure below the filesystem /db2

/var/db2 vg01 Filesystem for DB2 global registry

/home/<sid>adm vg00 Filesystem for home directory of SAP admin user

/sapmnt/<SID> vg00 Filesystem for SAP profile, SAP work/interface data, and SAP Kernel source

/usr/sap/<SID> vg00 Filesystem for SAP Instance directories

/usr/sap/DAA/SMDA97 vg00 Filesystem for SMD agent

Table 11 Minimum filesystem layout (DB2 UDB)

In the second example (Table 12) the filesystem layout is more complex: The database layout

is split-up into different volume groups. The filesystem layout fulfils the requirements for the

relocation and SAP system copy scenarios with SAP LVM as well as the capability for

performing SnapShot/FlashCopy backups for the database with IBM Tivoli Storage

FlashCopy manager:



Filesystem Volume Group Content

/db2 vg01 Contains the instance directory /db2/db2<sid> and all other database related directories. The instance directory /db2/db2<sid> is *not* created as a filesystem, as SAP LVM V1.0 then is not able to handle that during a SAP System Copy

/db2/<SID>/db2dump vg01 Own filesystem for the db2diag.log for convenience

/db2/<SID>/log_archive vg01 Separate filesystem for DB2 inactive logs

/db2/<SID>/log_dir vg03 Separate filesystem for DB2 active logs: In case SnapShot backups are regularly taken and should be used for a fast restore, this filesystem *must* be separate from database data.

/db2/<SID>/db2<sid> vg02 Contains the database directory. In case that regular SnapShot backups are intended for the database, the content needs to be included in the SnapShot image together with the DB2 database containers/ storage pathes

/db2/<SID>/sapdataX vg02 Contains the database containers / storage pathes

/home/<sid>adm vg00 Filesystem for home directory of SAP admin user

/sapmnt/<SID> vg00 Filesystem for SAP profile, SAP work/interface data, and SAP Kernel source

/usr/sap/<SID> vg00 Filesystem for SAP Instance directories

/usr/sap/DAA/SMDA97 vg00 Filesystem for SMD agent

/var/db2 vg01 Filesystem for DB2 global registry

Table 12 Extended filesystem layout (DB2 UDB)

If such FlashCopy backups for the database are a requirement, then a layout similar to the

second example has to be implemented.

3.4.5.2 Install a new SAP system

For the SAP system installation, select one resource node of the SAP LVM environment that

has no active SAP services.

The filesystem layout needs to be compliant to chapter 3.4.5.1

For the SAP system installation the virtual IP addresses for the database service and

the central instance/ central services need to be present. Define the virtual IP addresses

to the network adapter using the “ifconfig” command. These addresses must not be

defined as persistent, as the virtual IP addresses for the SAP services later on will be

managed exclusively by SAP LVM during the prepare/unprepare workflows.



# ifconfig <nw interface> alias <virtual IP CI> netmask <netmask>

# ifconfig <nw interface> alias <virtual IP DB> netmask <netmask>

During the SAP system installation later on, use the SAPINST_USE_HOSTNAME

parameter to specify the virtual hostname for the service to be installed.

Users and Groups have to be defined centrally in the LDAP environment. As SAPinst

currently cannot handle LDAP users properly, create users and groups for SAP and

DB administration manually before the SAP system installation.

The detailled installation procedure needs to follow all the steps as described in the

SAP installation documentation, and the according SAP notes.

3.4.5.3 Adapt an already existing SAP system

File System Layout

The filesystem layout needs to be compliant to chapter 3.4.5.1. So it may be required

to copy or move the content of the standard installation directories into the SAP LVM

compliant storage layout.

Virtual IP Address and hostname

Within the SAP LVM environment the database server and the central instance/

central services can be operated separately on different nodes. Two different virtual IP

addresses are required for them. If one of the virtual IP addresses was defined as base

address of a network adapter, redefine the network adapter to a new address. Define

the virtual IP addresses as IP alias to the network adapter using the ifconfig command:

# ifconfig <nw interface> alias <virtual IP CI> netmask <netmask>

# ifconfig <nw interface> alias <virtual IP DB> netmask <netmask>

If the hostname for the SAP instance is changed to a new virtual hostname, make sure

to adapt all occurrences of it in the filename and contents of DEFAULT, START, and

instance profile. Create and update the following parameters in the Instance Profile to

the virtual hostname too.

SAPLOCALHOST = <virtual hostname CI>

SAPLOCALHOSTFULL = <full qualified virtual hostname CI>

icm/host_name_full = <full qualified virtual hostname CI>

For example, in the default profile DEFAULT.PFL, the SAPDBHOST has to reflect

the virtual IP label of the DB server, and the virtual IP label of the CI will be part of

the mshost, vbname, enqname, btcname, and gateway parameters:

SAPDBHOST = c01db01

rdisp/mshost = c01ci01

rdisp/sna_gateway = c01ci01

rdisp/sna_gw_service = sapgw10

rdisp/vbname = c01ci01_C01_10

rdisp/enqname = c01ci01_C01_10

rdisp/btcname = c01ci01_C01_10



Dependent on your individual environment, you may have to adapt settings in further

interfaces / SAP systems too.

(Re-) create the Users in LDAP

The administrative users for Database and SAP system have to be created with the

same numeric UID in the LDAP directory. If the UID for the users violate uniqueness

within the SAP LVM environment, the ownership of all files needs to be changed to

an unique UID.

Adapt user environment files

During Shell startup, several files are sourced to get the proper environment for SAP

admin and DB admin users. During installation time, these files are created containing

the hostname within their name. Rename them in the home directory of both SAP

admin and DB admin user.

# mv .sapenv_<virtual hostname>.sh .sapenv.sh

# mv .apoenv_<virtual hostname>.sh .apoenv.sh

# mv .dbenv_<virtual hostname>.sh .dbenv.sh

# mv .sapenv_<virtual hostname>.csh .sapenv.csh

# mv .apoenv_<virtual hostname>.csh .apoenv.csh

# mv .dbenv_<virtual hostname>.csh .dbenv.csh

You also need to check the shell start profiles (.cshrc) that it sources all required

environment files correctly. Each entry should look like

# RDBMS environment

if ( -e $HOME/.dbenv_`hostname`.csh ) then

source $HOME/.dbenv_`hostname`.csh

else if ( -e $HOME/.dbenv.csh ) then

source $HOME/.dbenv.csh

endif

This means the shell first searches for a file containing the hostname, will not find it

and then uses the corresponding file without the hostname. Sometimes the “else if”

clause is missing and you need to create it manually.

Adapt database configuration files to virtual IP address

o ORACLE: Adapt configuration files tnsnames.ora and listener.ora (ADDRESS =

(COMMUNITY = SAP.WORLD)

(PROTOCOL = TCP)

(HOST = c01db01)

(PORT = 1537)

)

The stanzas in tnsnames.ora and listener.ora have to reflect the virtual

hostname of the DB instance. The communication port should be unique within

the environment.

o DB2 UDB: Update db2cli.ini to the virtual DB server address

; Comment lines start with a semi-colon.

[C01]



Database=C01

Protocol=tcpip

Hostname=c01db01

Servicename=5912

[COMMON]

Diagpath=/usr/sap/C01/SYS/global/db6/db2dump

The “Hostname” in the db2cli.ini file needs to reflect the virtual hostname of

the DB instance. The service name reflects the communication port and should

be unique within the environment.

Perform a test connection to the database: Start the database (and, in case of DB

ORACLE, the listener process). As user <sid>adm, perform a test connection to the

database using R3trans:

c01adm> R3trans –d –v

This is R3trans version 6.22 (release 720 - 03.06.11 - 17:57:00).

unicode enabled version

R3trans finished (0000).

R3trans should return 0 for a successful DB connection.

3.4.5.4 Additional SAP profile parameters

See also SAP Note 1438774 - New profile parameter system/uuid and system/description:

SAP LVM will verify the system/uuid profile parameter against its configuration prior to

any operation (start, stop, relocate) on any instance of a (cloned) system, to ensured that the

operation is triggered on the correct instances. The system/description profile parameter

contains a verbal description of the system.

The parameter can be added to the SAP system later on also during the LVM configuration

phase. However, the SAP system then needs to be restarted to activate the parameter.

3.4.5.5 SAP license considerations

The license key for an SAP system is checked on the resource on which the message server is

running. If the message server is moved during a relocation event to another resource, a

license key for the new environment may be required. Request and import in advance license

keys for all the systems on which the message server may run. The license key depends on the

hardware key: The hardware key is different for different physical servers. (All the LPARs

within one physical IBM Power server have the same hardware key).

The license key is checked during the (re-) start of the message server:

If the relocation is initialized virtualization-based (Live-Partition Mobility), then the

message server keeps running in the same state as before.

o Even if there is no license key for the target server available the SAP system

continuous its operation. The attempt to restart the SAP system on the target

server while having no valid license for the hardware key of the target server

would then fail.

If the relocation is initialized adaptive-based (stop and unprepare on the source

resource/ prepare and start on the target resource), then the message server is restarted

on the target resource, and a valid license key for the hardware key of the target server

is required.




SAP has introduced a new license key feature called Flexible License Mechanism to simplify

the administration of SAP license keys in system landscapes where the message server can

move between several physical servers. With that method the license key is no longer tied to

the hardware key of the message server. The flexible license mechanism uses a separate ID

generator, which creates a unique network ID for each message server. This network ID is

hardware independent and therefore it is possible to move the message server to a different

host and retain the unique ID. It is possible to configure multiple ID generators to eliminate

any potential single point of failure.

Refer to the current SAP NetWeaver Help documentation for a detailed description and

instructions of how to set up this new Flexible License Mechanism:

SAP NetWeaver 7.0 EHP2 and SAP NetWeaver 7.3

3.4.6 Install SAP LVM Prerequisites on the management node

3.4.6.1 XIV XCLI

The XIV CLI is used for the communication of LPARs with the storage system. It needs to

run each of the management as well as on the managed LPARs.

It is a software which needs to be installed separately

http://help.sap.com/saphelp_nw70ehp2/helpdata/en/20/384ef6bd054d5aae1ddd6f67e07a9e/content.htm

http://help.sap.com/saphelp_nw73/helpdata/en/20/384ef6bd054d5aae1ddd6f67e07a9e/content.htm



4 Initial LVM Configuration

4.1 Engine Settings

The SAP LVM engine settings are normally configured when running the Initial

Configuration Wizard. If you need to change some of the settings as for example the default

user name or password for the Host Agents, you can do this under Setup General Settings

LVM Engine Settings.

Figure 21 LVM Engine Settings

4.2 Storage Manager Configuration

The IBM Storage Manager settings (IBM XIV Storage System, IBM Tivoli Storage

FlashCopy Manager) are defined via the Configuration Infrastructure Storage Manager

Panel. The start panel is illustrated in Figure 22: Navigate to this panel, and press “Add” to

define a Storage Manager in SAP LVM.



Figure 22 Storage Manager Configuration entry panel

In the next screen (Figure 23) the Storage Manager Type can be selected. Select Vendor

“IBM”, Product “IBM FCM Adapter” and click “Next”.

Figure 23 Storage Manager Configuration: Select Storage Manager type

In the next panel (Figure 24) the IBM Tivoli Storage FlashCopy Manager related settings are

specified (See also chapter 3.3.2, as a lot of these parameters need to be provided from the

FCM installation):

The “FCM/ XIV CLI Host Name” field specifies the node on which the IBM Tivoli Storage

FlashCopy Manager Management Agent is installed: The Management Agent (acsd) controls

the backup flow and mediates between the other agents running on the “resource” LPARs.

The Management Agent also provides access to the snapshot backup repository, which

contains information about the valid snapshot backups and their relationships to snapshot

capable storage devices. During the SAP system cloning / SAP system copy process SAP

LVM starts the Management Agent on the “FCM CLI Host Name” node.

The “FCM CLI Path” specifies the working directory for the IBM Tivoli Storage FlashCopy

Manager on all of the nodes. This directory reflects the path which was specified during the

installation customization of the IBM Tivoli Storage FlashCopy Manager (FlashCopy backup

for generic applications in “setup_gen.sh”)



The “IBM FCM template” reflects the filename of the profile template located in the “FCM

CLI Path”.

Figure 24 Storage Manager Configuration: Define FlashCopy Manager settings

When all required entries are defined, press “Test Configuration”. SAP LVM will initiate a

communication test to FlashCopy Manager and the storage subsystem CLI. The configuration

test should return “Connection successful: SAP LVM successfully gets data from storage

managers through IBM Tivoli Storage FlashCopy Manager”.

After a successful connection test, select “Next” to get to a user and password configuration

screen. Here you can enter a user and password for the configured storage subsystem. SAP

LVM stores the storage subsystem password internally: if the user or password is changed in

the storage subsystem, you have to update the LVM Storage Manager configuration too.

Figure 25 Storage Manager Configuration: user and password

Click “Next”. The summary screen appears; the SAP LVM Storage Manager definitions for

IBM XIV and IBM Tivoli Storage FlashCopy Manager are complete.



Figure 26 Storage Manager Configuration Summary

4.3 Discovery of resources (LPARs)

To discover a new resource in SAP LVM, the SAP Host Agent must be installed and running

on the target partition. Select Configuration Resources Add Resources to get to the “Find

Resources” panel. Here you have various options to perform a resource detection, in this

example we select “Detect Using Host and Instance Agent” as Source and “Resources only”

under Find.

Figure 27 Add resource – find Resources

Enter the host name of the partition you want to discover. You can use the default credentials

for the SAP Host Agent user id as configured in the engine settings or choose other

authentication types like specific user/password settings or a X.509 client certificate. Click the

Detect button and you should get a message that new resources are found.

On the next panel select the resource pool you want to assign this partition to and click the

next button.



Figure 28 Add resource - assign pool

Click the Save button on the summary screen to add this newly discovered resource.

The next step is to change a number of properties for this new resource. Select the new host

on the Resources Overview panel and click “Edit”. The “Basic Configuration” step should

already have the correct settings, click Next to get to the Resource Properties panel.

Figure 29 Add resource – define properties

You have to select the AC-Enabled flag and add a network interface that is used to bind the

virtual IP address of a SAP service. You also have to provide a capacity figure (SAPS

provided) for this resource. If the partition should be used as a target resource for cloning and

system copy operations, then you need to select the “Isolation Ready” flag too. Press the

“Save” button to store the changes.

4.4 Service Configuration/ Discovery in SAP LVM

The discovery for a new system is done in several steps. In the “base discovery” the new SAP

system is discovered and added to the SAP LVM configuration. Then the configuration of the

newly discovered services is completed. This includes the detailed storage configuration for

database and for the central instance.

4.4.1.1 Discovery for a new SAP system

The base discovery of new SAP systems is started out of the in the tab “Services” in the

“Configuration” view. A new SAP System is discovered by using the “Add Services” button

(Figure 30):



Figure 30 Add new SAP services

The new system needs to meet the following prerequisites:

the resource hosting the LPAR is already discovered in SAP LVM

the SAP Host Agent is installed and active on the resource

the SAP system is installed in an “adaptive” configuration:

o virtual hostnames are defined for both database and central instance

o the storage layout supports separation of database and central instance

the SAP system is active

all the user credentials (e.g. <sid>adm) and their passwords are known to the SAP

LVM admin

For older SAP releases (without having enhanced security measures implemented) the SAP

Host Agent is able to discover both database and SAP instances running on the LPAR. For

higher SAP releases the SAP Host Agent may not be able to detect the SAP instance and all

the details. Due to enhanced security settings, the “sapadm” user is not authorized any more to

retrieve all the instance parameters during the discovery. Discovery of the SAP instance

requires to contact the instance agent using the <sid>adm credentials to retrieve all relevant

instance data. The “Expert Mode” in the configuration panel has to be used.

Specify the connection data to the SAP Host Agent (IP address of the resource LPAR), as

well as the connection details to the Instance agent (Figure 31):



Figure 31 Expert Mode: Detect new services using host agent and instance agent

The discovery is then started by pressing the “Detect Button”. The Database and SAP Central

Service services are identified, and the data is integrated to the “Add Services” view. In this

view, the services are assigned to the appropriate resource pool and the SAP system is

described with additional details (Figure 32).

Figure 32 Add Services View

Select a target resource pool using the selection menu in the “Pool for New Systems”. When

the selection dialog screen appears (Figure 33), select the appropriate pool for the new

system.



Figure 33 Selection of the Resource Pool

After the “Resource Pool” was selected, the detailed description for the SAP System needs to

be specified (Figure 34):

Figure 34 Enter SAP System Description

Base configuration for the new system is done,

Figure 35 Add Services sub-menu

and finally the new system data is saved (Figure 36).



Figure 36 Summary view for the new system

Base system discovery is complete, and now configuration details for the SAP system, and

specifics for the database and the SAP central instance are started. These additional

configuration steps are required to enable the system for SAP LVM.

4.4.1.2 Base configuration

Select the “System Details” view for the new SAP System, and press the “Edit” button to

define all the detailed settings (Figure 37):

Figure 37 SAP System configuration

The customization is started (as indicated in Figure 38) and will run through the steps

Basic Configuration

ACM Settings

Network Isolation



Figure 38 SAP System configuration: general settings

Basic Configuration

The Provisioning Use-Cases allowed for the SAP System are selected in the “Basic

Configuration” (Figure 39). These include:

Cloning

Copying

Renaming

Application Server (Un-) Installation

If either copying or renaming are check-marked, then a RFC destination (target client(s) for

post copy automation) needs to be defined. The customization panel for a RFC destination

will open automatically by checking the “Configure RFC Destination” flag.

Figure 39 SAP System configuration. Overall system details



ACM Settings

ACM was not evaluated in the test installation (Figure 40). Click “Next” to go to the next

step.

Figure 40 SAP system configuration: ACM

Network Isolation

In the network isolation step, all network connections which are still allowed during the

“Network fencing” step are configured. The settings can still be adapted during the SAP

System Copy/System cloning definition steps. The settings defined here will appear as default

allowed connections during the System Copy / System Cloning step (Figure 41).

Figure 41 SAP System configuration: Network fencing defaults

As examples, the network connections to “siccserv2” (DNS server for the test environment,

Figure 42), and the “ssh” protocol (Figure 43) were explicitly added as allowed per default to

the network fencing configuration.



Figure 42 SAP System configuration: Add allowed communications

Figure 43 SAP System configuration: Add allowed communications

Complete with “Save” afterwards.

4.4.1.3 DB Configuration

After the base configuration for the SAP system is done, the next step will configure details

for the SAP database service. In the “Services” tab in the “Configuration” view expand the

new SAP System, and select the database. Press the “Edit” button (Figure 44).



Figure 44 DB Configuration: Entry screen

Configuration of the database will be done in the order

Basic Configuration

Service Properties

Mount Points

Basic Configuration

Keep (or change) the settings in the basic configuration view (Figure 45), afterwards click

“Next”:

Figure 45 DB Configuration: Basic configuration



Service Properties

Check-mark the box “AC-Enabled” in the Adaptive Enablement session of the screen (Figure

46). Select the appropriate Network. Additional values will be shown in the “Requirements”

section of the screen. Enter the required SAPS and required Memory values for the service.

These attributes will be used during Start- and Relocate actions to determine which target

resource is a candidate for the service. The definition of the target resource needs to satisfy

the conditions defined here in this dialog.

Select the allowed OS levels for the service, and press “Next”

Figure 46 DB Configuration: Add resource details

Mount Points

In this step, the storage configuration for the system is defined. This step will depend on the

storage provider. For the IBM storage environment, some data needs to be gathered on the

resource LPAR. In detail the required steps are:

Create a file containing all the filesystems related to the database of the SAP System

to be discovered on the LPAR

Use the script ibm_lvm_mgmt.ksh from the IBM Storage library to create a XML

structure referencing all required storage information for the database

Copy this information and paste it into the LVM configuration screen

Un-check the flag “OS Managed Mounts /Automounter)”. The screen changes its

appearance. Select “Extract Mount Points” to enter the XML data gathered on the resource

(Figure 47).



Figure 47 SAP LVM Storage Definition Screen

Logon to the LPAR hosting the SAP system and create a list for the database filesystems.

Figure 48 illustrates a sample list of database filesystems stored in a config file.

Figure 48 Create a list of the DB filesystems

This list is then used as input data for the ibm_lvm_mgmt.ksh script. Additional arguments

for the script are the name of the XIV Storage System and the “mountconfig” keyword

(Figure 49). Specify this as arguments and execute the ibm_lvm_mgmt.ksh script:

/usr/sap/hostctrl/exe/operations.d/IBM/ibm_lvm_mgmt.ksh mountconfig [filename] [XIV

name]

The execution of the script generates a XML structure for the mount data to standard out:

Each filesystem from the list will get an own mount stanza in the XML output.



Figure 49 Create XML-File for mountconfig

Copy the XML structure from the screen of the LPAR (Figure 49) and paste it into the SAP

LVM “Input for Extraction Window” (Figure 50):

Figure 50 Paste Mountconfig in LVM storage config

Then press the “Extract Mount” Button in the SAP LVM screen: SAP LVM transfers the

XML data into its filesystem data model (Figure 51):



Figure 51 Extract Mount Points

All the mount points for the database are integrated into the SAP LVM configuration, and the

database service configuration can be saved (Button “Save”, Figure 52).

Figure 52 Final LVM Mount point configuration

Database configuration is complete.

4.4.1.4 Configuration of SAP Central Instance

After the configuration of the database instance is done, the next step will configure details

for the SAP Central Instance. This is similar to the “database configuration”.

In the “Services” tab in the “Configuration” view expand the new SAP System, and select the

Central Instance. Press the “Edit” button (Figure 53):



Figure 53 Service Details Central Instance

Configuration of the database will be done in the order

Basic Configuration

Service Properties

Mount Points

Basic Configuration

Keep (or change) the settings in the basic configuration view (Figure 54), afterwards click

“Next”:

Figure 54 Basic Configuration for Central Instance



Service Properties

Check-mark the box “AC-Enabled” in the Adaptive Enablement session of the screen (Figure

55). Select the appropriate Network. Additional values will be shown in the “Requirements”

section of the screen: Enter the required SAPS and required Memory values for the service.

These attributes will be used during Start- and Relocate operations to determine which target

resource is a candidate for the service. The definition of the target resource needs to satisfy

the conditions defined here in this dialog.

Select the allowed OS levels for the service, and press “Next”.

Figure 55 Service properties for Central Instance

Mount Points

In this step, the storage configuration for the system is defined. This step will depend on the

storage provider. For the IBM storage environment, some data needs to be gathered on the

resource LPAR. In detail the required steps are:

Create a file containing all the filesystems related to the central instance of the SAP

System to be discovered on the LPAR

Use the script ibm_lvm_mgmt.ksh from the IBM Storage library to create a XML

structure referencing all required storage information for the central instance

Copy this information and paste it into the LVM configuration screen

Un-check the flag “OS Managed Mounts /Automounter)”. The screen changes its appearance.

Select “Extract Mount Points” to enter the XML data gathered on the resource (Figure 56):



Figure 56 Mount Config for Central Instance

Logon to the LPAR hosting the SAP system and create a list for the central instance

filesystems. Figure 57 illustrates a sample list of database filesystems stored in a config file.

Figure 57 Create List of filesystems for central instance

This list is then used as input data for the ibm_lvm_mgmt.ksh script. Additional arguments

for the script are the name of the XIV Storage System and the “mountconfig” keyword

(Figure 58). The execution of the script generates a XML structure for the mount data to

standard out.



Figure 58 Extract Mount Config for Central Instance

Copy the XML structure from the screen of the LPAR (Figure 58) and paste it into the SAP

LVM “Input for Extraction Window” (Figure 59):

Figure 59 Transfer Mount Config to SAP LVM

All the mount points for the database are integrated into the SAP LVM configuration, and the

database service configuration can be saved (Button “Save”, Figure 60).



Figure 60 Apply Mount Config to SAP LVM

SAP Central Instance configuration is complete

4.5 Setup for the environment in IBM Systems Director

4.5.1 Discover the OS deployment components in IBM Systems Director

Execution of image deployment requires activities on the Hardware Management Console

(HMC), the virtual I/O server(s), and the NIM server. These components need to be

discovered in IBM Systems Director first. A OS mksysb image then needs to be captured

from the NIM server. New deployments using this image then can be started.

All the screenshots shown here are based on IBM Systems Director 6.2.1 and VMControl

2.3.1: So the screens with IBM Systems Director 6.3. / VMControl 2.4 may look slightly

different. The overall procedure however is still valid.

4.5.1.1 Discover the HMC

Figure 61 illustrates the HMC discovery steps: In the Systems Director GUI go to:

Inventory System Discovery Enter the IP address of the HMC. Then choose

“Discover Now”



Figure 61 HMC Discovery in IBM Systems Director

Request access to the HMC (Figure 62)

Choose the entry reflecting the operating system Enter the HMC‟s user id and

password. Then choose “Request Access”

Figure 62 Request access to the HMC

Collect HMC inventory (Figure 63)

In the Systems Director GUI go to:

Inventory View and Collect Inventory choose „Browse‟ in the target section

select the two HMC entries choose “Add” choose “OK”.



Figure 63 HMC inventory collection

An inventory collection is started in the background. When the collection job is

completed, you can view the discovered inventory (Figure 64):

Figure 64 View HMC inventory

4.5.1.2 Discover the VIO server(s)

Discovery of the VIO Server(s) follows a similar sequence as illustrated before for

discovery of the HMC. In the Systems Director GUI go to:



Inventory System Discovery Enter the IP address of the VIO server choose

“Discover Now”

Request access to the VIO server

Choose the entry reflecting the operating system Enter the VIO server‟s user id and

password choose “Request Access”

Collect VIO server inventory


Inventory View and Collect Inventory choose „Browse‟ in the target section

select the VIO server entry choose “Add” choose “OK”.



Figure 65 View VIO server inventory

4.5.1.3 Discover the NIM server

Discovery of the NIM Server LPAR follows a similar sequence as illustrated before

for discovery of the HMC. In the Systems Director GUI go to:

Inventory System Discovery Enter the IP address of the NIM server choose

“Discover Now”

Request access to the NIM server

Choose the entry reflecting the operating system Enter the NIM server‟s user id

and password choose „Request Access‟

Collect NIM server inventory


Inventory View and Collect Inventory choose “Browse” in the target section

select the NIM server entry choose “Add” choose “OK”.





Figure 66 View NIM server inventory

4.5.2 Setup VMControl

VMControl can be customized when HMC, VIOS and NIM Server are discovered.

VMControl interfaces with the NIM server via an agent: This agent needs to be installed on

the NIM server LPAR first.

4.5.2.1 Install NIM server agent on NIM server

Important:

The following agents have to be installed on the NIM server to ensure a successful

connection to Systems Director. The agent version correlates to the VMControl

version in place:

root> lslpp -l | grep -i agent

DirectorCommonAgent 6.2.1.3 COMMITTED All required files of

Director Common Agent, including JRE

DirectorPlatformAgent 6.2.1.2 COMMITTED Director Platform Agent for

Tivoli_Management_Agent.client.rte

cas.agent 1.4.2.32 COMMITTED Common Agent Services Agent

If you find other versions installed on the LPAR the agents may need to be removed

completely before a new version can be installed: Update of the agents doesn‟t work in

all cases. To ensure a clean discovery, remove the existing NIM server settings from

the existing Systems Director Database first.


Navigate Resources Choose “All Systems” Search and right click the NIM

server Choose “Remove”



Figure 67 Remove NIM server

(Re-) Discover the NIM server

In the Systems Director GUI go to (Figure 68):

Inventory Discover System Enter the IP Address of the NIM server Click

“Search”

Figure 68 Discover NIM server



Request Access to the NIM server


Navigate Resources Choose “All Systems” Search and right click the NIM

server Choose “Request Access”

Collect inventory


Inventory View and Collect Inventory Select or Browse the target system

Click “Collect Inventory”

Figure 69 Collect Inventory of NIM server

Install the NIM subagent


Welcome VMControl Enterprise Edition In the “Common tasks” table, click on

“Install agents” (Figure 70) Click “Next” Choose “Common Agent Subagent

Packages” Choose “CommonAgentSubagent_VMControl_NIM-2.x.x.x”

(according to the current VMControl version) and click “Add” Choose the NIM

server and click “Add” Click “Finish”



Figure 70 Installation of NIM subagent on the NIM server with VMControl

Collect the inventory once again


Inventory View and Collect Inventory Select or Browse the target system

Click “Collect Inventory”

4.5.2.2 Prepare NFS filesystem for the OS images on the NIM server

On the NIM server a NFS filesystem is created for the OS images. During the capture

activities, this filesystem is mounted via NFS on the NIM client: It will be the target directory

for the “mksysb” image. In smitty, export the related directory, in our case it is

/export/mksysb (Figure 71):

Change Attributes of an Exported Directory

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[TOP] [Entry Fields]

* Pathname of directory to export /export/mksysb

* Version of exported directory to be changed 3

Anonymous UID [-2]

Public filesystem? [no] +

* Change export now, system restart or both both +

Pathname of alternate exports file []

Allow access by NFS versions [3]

External name of directory (NFS V4 access only) []

Referral locations (NFS V4 access only) []

Replica locations []

Ensure primary hostname in replica list yes +

Allow delegation? []

Scatter none +

Security method 1 [sys,krb5p,krb5i,krb5,dh] +

Mode to export directory [read-mostly] +

Hostname list. If exported read-mostly [siccps99]

Hosts & netgroups allowed client access []

Hosts allowed root access [siccps99]

Security method 2 [] +

Figure 71 Export mksysb directory



4.5.2.3 Create mksysb backup on the target LPAR

On the LPAR to be backed up, mount the NFS share mount siccnim2:/export/mksysb /mnt

On the LPAR to be backed up, enter following command: mksysb -i /mnt/sap_lvm_generic.mksysb

4.5.2.4 Import “mksysb” as Virtual Appliance into VMControl

In the Systems Director CLI, enter: root> smcli captureva -r <resource number> -F repos:<location of the

mksysb> -n <image name> -A "cpushare=<number of physical

CPUs>,memsize=<memory size>,disksize=<disk size>"

Example: root> smcli captureva -r 74091 -F

repos://export/mksysb/sap_lvm_generic.mksysb -n "SAP LVM generic" -A

"cpushare=0.1,memsize=4096,disksize=20000000000"

Tip: to get the resource number, enter: root> smcli lsrepos –o

Tip: to delete an image from systems director, enter root> smcli rmva -V <ApplianceID>

where the value of the ApplianceID can be obtained using: root> smcli lsva –l

4.5.2.5 Deployment test for the Virtual Appliance

When the image is successfully captured in VMControl, it can be deployed now:


Welcome VMControl Enterprise Edition Change to the „Virtual Appliances‟ tab

Select the virtual appliance you want to deploy Click „Deploy Virtual Appliance‟

Follow the wizard Click „Finish‟



Figure 72 Trigger of Virtual Appliance Deployment

4.5.3 Setup of IBM Systems Director Security

You can use IBM Systems Director administrator users for integration with SAP LVM. Some

virtualization use cases (OS provisioning and LPAR relocation using Live Partition Mobility)

require administration credentials. For virtualization monitoring and management, you may

use restricted credentials. IBM Systems Director administrators may allow monitoring and

management of particular elements only (e.g. of certain physical servers, LPARs, operating

systems, etc.) and thus restrict a view of SAP LVM users. Below we describe how to extend

IBM Systems Director predefined users so that you may use them for integration with SAP

LVM.

Log in to operating system where IBM Systems Director is running on and create a group

(e.g. using mkgroup) and a user (e.g. using mkuser) that you want to use for integration with

SAP LVM. The user must belong to the group you created previously.

Log in to IBM Systems Director console as administrator and navigate to „Users and

Groups“. Press „Authorize Groups“ and authorize the group created in previous step. All the

users that belong to this group will appear as valid IBM Systems Director users with minimal

access rights.



Figure 73 Setup of IBM Systems Director Security – Authorize Groups

Navigate to „Roles“ and create a copy of predefined roles „SMManager“ and „SMUser“ by

clicking on „Create Like...“ The copy of „SMManager“ (named „LVM_Manager“ in our

example) is a role for SAP LVM management authorization. The copy of „SMUser“ (named

„LVM_User“ in our example) is a role for SAP LVM monitoring authorization.

Figure 74 Setup of IBM Systems Director Security – Copy Predefined Roles

Edit both of the copied roles and extend them by assignment of additional permissions.

Extend the copied SMUser Role with following permissions:

General Create Group, General Edit (Description: Provides an editor for

changing an existing group)

System Status and Health Monitors



Extend the copied SMManagement Role with following permissions:

Power On/Off Power Off Now, Power On, Shutdown and Power Off

Figure 75 Setup of IBM Systems Director Security – Modify Copied Roles

Navigate to “Resource Explorer” and create two static groups that include any types of

resources. The first group (monitoring) should include physical machines, virtual appliances,

storage pools and virtual servers that will be visible in SAP LVM. The second group

(management) should include only the virtual servers that will be manageable through the

SAP LVM. If possible, use IBM Systems Director administration credentials for first

integration with SAP LVM. In that case SAP LVM virtualization adapter for IBM Systems

Director would create a list of dynamic groups containing all the resources that may be shown

in SAP LVM. You can then navigate to these dynamic groups and add the resources you are

interested into the static groups.

Figure 76 Setup of IBM Systems Director Security – Define Resource Groups



Navigate back to „Users and Groups“ and choose the user that you would like to configure.

Click „Assign Role...“ in order to step into the assignment dialog.

Figure 77 Setup of IBM Systems Director Security – Assign Roles to User

Associate the created groups with created roles, whereby the copy of SMUser role becomes

associated with monitoring group and the copy of SMManagement role becomes associated

with management group. Add these groups / roles combinations to the list of “Assigned

Roles”.

Figure 78 Setup of IBM Systems Director Security – Assign Groups and Roles to User

After finishing this configuration you may use the configured user for integration of IBM

Systems Director with SAP LVM.

4.6 Virtualization Manager Configuration in SAP LVM

The IBM Virtualization Manager settings (IBM Systems Director/ VMControl) are defined

via the Configuration Virtualization Managers Panel. SAP Note 1728222 – Configuration

of IBM Systems Director VMControl for SAP LVM describes further background

information about IBM Systems Director VMControl integration with SAP LVM for this

step. The start panel is illustrated in Figure 79. Navigate to this panel, and press “Add” to

define a new virtualization manager in SAP LVM:



Figure 79 Virtualization Manager Configuration

Select “IBM Systems Director VMControl” as platform (Figure 80):

Figure 80 Virtualization Manager: IBM Systems Director VMControl

Then enter all the relevant values for the virtualization manager.

As Basic properties provide a label in SAP LVM, the user name and password of the

“administration” user that was created for the IBM Systems Director in chapter 4.5, and the

URL pointing to the IBM Systems Director Management server.

In the Additional Properties provide the Systems Director Port (Default is 8422), the

monitoring interval (how often SAP LVM tries to retrieve data from IBM Systems Director),

and the Storage Connection “NPIV” for LPAR deployments1

Press “Test Connection” to validate all the settings.

1 The deployment of LPARs end-to-end (LPAR creation, storage assignment, and OS installation) requires the

Storage Control plug-in and customizing for all data sources (SAN switches, Storage Systems).



Figure 81 Virtualization Manager properties

Press “Next” to proceed to the next screen (Figure 82 ): Press “Save” to confirm all settings.

Figure 82 Virtualization manager summary

4.7 Additional Configuration Steps

4.7.1 Network definition in SAP LVM

As a prerequisite for SAP System Cloning / Copy / Refresh, the network definitions need to

be completed in SAP LVM. The configuration dialog is started via

Configuration Infrastructure Network

Press “Add” to define a new network (Figure 83):



1

2

3

4

5

1

2

3

4

5

Figure 83 Configure Network

In the first panel “Basic Configuration” settings are specified. They includes Name, Subnet

Mask, and the Broadcast Address of the network.

In the next step, “Advanced Configuration” settings are specified. They include Gateway

Server, DNS Zone, Reverse DNS Zone and Update IP Range.

The “Update IP Range” can restrict the range for new IP addresses that are provisioned out of

SAP LVM if automatic DNS update is configured. If the "Auto Assign IP Address" option is

selected in the "Host Names" step during a clone or copy, a random free IP address is selected

from within this range. If an IP address is specified manually, it must still be within this

range.

IP addresses are expected in dotted quad notation.

Multiple IP ranges can be separated by ";"

The lower and upper parts of a range must be present in each range and are separated

by ";" or ":"

Ranges are calculated by using the IPs as numbers.

Example: The range 12.48.2.3:12.48.4.2 includes 12.48.2.4, 12.48.2.255, 12.48.3.0,

12.48.3.255, 12.48.4.0, 12.48.4.1, and 12.48.4.2.

An IP range must not be a single IP. However, a single IP can be represented as a

range. Example: 12.48.2.38 as 12.48.2.38:12.48.2.38



4.7.2 Configuration for SAP System Copy

During a SAP System Copy and a SAP System Rename activity SAP LVM invokes the

SAPInst rename tool on the target LPAR to change the SAP system ID of the system. SAP

LVM expects that the SAPInst tool is available in an installation folder which can be accessed

on the target LPAR: In the test environment a central NFS server is used as repository, and

the directories are mounted to all the (target) LPARs. This path is declared in Configuration

Extended Settings System and Provisioning for each different target OS type:

Figure 84 illustrates the configuration settings:

Figure 84 Customizing SAPInst rename path



5 Detailed Description / Execution of Use Cases

5.1 LVM Standard Edition Use Cases

This chapter describes the use cases which are included in the standard version of SAP LVM.

Basically the SAP LVM Standard Edition allows to Start, Stop, and Relocate SAP Systems.

5.1.1 Description of the LVM Standard Edition Use Cases

This chapter introduces the use cases of LVM Standard edition. Following use cases are

possible:

Prepare / unprepare SAP instances and systems

Start / stop SAP instances and systems

Relocate SAP instances and systems

Start / stop resources

Monitoring

5.1.1.1 Prepare / unprepare SAP instances and systems

The Standard Edition of SAP LVM is able to prepare and unprepare SAP systems and

instances. That means, in case of a prepare, the SAP instance or system is made ready to be

started on a resource like an LPAR. During the prepare phase, the storage volumes belonging

to an SAP system are mapped to the LPAR, the volume groups are imported, the file system

are mounted, and the SAP instance service is started.

In case of an unprepare, the SAP instance service is stopped, file systems volumes belonging

to an SAP system are unmounted, the volume groups are exported, and the storage are

unmapped. This allows to map and start the SAP system to/on the same LPAR, or to/on

another LPAR.

The prepare / unprepare can be run on SAP instance or on a complete SAP system including

all instance belonging to an SAP system. If the complete SAP system is affected, the function

is called mass operation.

Prepare / unprepare triggers following calls:

Storage CLI commands in order to attach or detach storage volumes containing SAP

systems to / from certain OS hosts

OS specific storage libraries for handling the storage mapping

OS specific libraries for handling the SAP instance service

5.1.1.2 Start / stop SAP systems and instances

In order to start an SAP instance or system, it needs to be successfully prepared. After an

SAP instance or system has been stopped, it can be unprepared.

The prepare+start and stop+unprepare can be executed as a single workflow each.

As for prepare / unprepare mass operations allow to start / stop complete SAP systems versus

single SAP instances.

Start / stop triggers following calls:

OS specific libraries for handling the SAP tasks start / stop

5.1.1.3 Relocate SAP instances and systems

The relocation of SAP instances and systems allows to move SAP instances or systems to

another LPAR.

There are two flavors of SAP instance or system relocation:



IBM PowerVM based relocation

This means that the complete LPAR including the SAP and operation system is moved

(also called migrated) from one physical host to another physical host. In that case

LVM uses IBM PowerVM Live Partition Mobility (LPM). The LPAR and SAP

system stay running during the execution of LPM. LPM is used e.g. for hardware

maintenance activities.

Hypervisor based relocation triggers following calls:

o IBM PowerVM specific libraries for handling Live Partition mobilty in the

HMC or Systems Director

SAP adaptive based relocation

This means that the SAP system is moved from one operating system or LPAR to

another operating system or LPAR. This causes a short SAP downtime as the SAP

system needs to be stopped on the previous operating system and started on the „new‟

operating system. The process automates following functions:

o Stop SAP instance or system

o Unprepare SAP instance or system

o Prepare SAP instance or system on the „new‟ LPAR

o Start SAP instance or system on the „new‟ LPAR

As for prepare / unprepare or start / stop mass operations allow to relocate complete

SAP systems versus single SAP instances.

SAP based relocation is used e.g. for operating system based maintenance.

SAP based relocation triggers following calls:

o Storage CLI commands in order to attach or detach storage volumes containing

SAP systems to / from certain OS hosts

o OS specific storage libraries for handling the storage mapping

o OS specific libraries for handling the SAP instance service

Figure 85 and Figure 86 illustrate the two relocation scenarios.



Figure 85 IBM PowerVM based relocation

Figure 86 SAP adaptive based relocation



5.1.1.4 Start / stop resources

With SAP LVM LPARs can be started or stopped.

Caution: Powering off LPARs without stopping and unpreparing the SAP systems running

inside the LPAR may cause data loss or data corruption.

Starting or stopping LPARs triggers following calls:

IBM PowerVM specific libraries for handling Live Partition mobilty in the HMC or

Systems Director

5.1.1.5 Monitoring

SAP LVM provides several types and levels of monitoring, e.g. CPU consumption, memory

consumption, dashboard, history data etc.

SAP LVM communicates with the virtualization manager in order to retrieve monitoring data

about IBM Power physical servers and logical partitions. Optionally SAP LVM tries to gather

CPU utilization data for discovered elements (physical servers and LPARs). SAP Notes

1728222 (for IBM Systems Director VMControl adapter) and 1728293 (for IBM HMC

adapter) provide configuration details of components involved in this scenario.

Following kinds of setup are supported:

Monitoring through IBM Systems Director VMControl adapter

Monitoring through HMC adapter

5.1.2 Executing the LVM Standard Edition Use Cases

5.1.2.1 Prepare / unprepare SAP instances and systems

The following section describes the required steps to prepare a SAP instance. As the

unprepare process is very similar, it will not be shown separately.

As a prerequisite, the SAP system needs to be installed and discovered in SAP LVM with

correct storage volumes and mount points.

To prepare a SAP instance:

Log in to SAP LVM Chose „Operations‟ Chose tab „Services‟ Enhance the SAP

system you want you work with Select the instance to be prepared Select or enter a

resource Chose the small arrow in the right lower corner of the button „Prepare and Stop‟

Chose ‚Prepare„

Figure 87 shows an example on how to prepare a SAP instance.



Figure 87 Prepare SAP instance

You can monitor the progress in the „Logs‟ tab, see Figure 89.

5.1.2.2 Start / stop SAP systems and instances

The following section describes the required steps to start a complete SAP system including

the prepare step. As the complete SAP system is started, this is a mass operation.

The stopping process is very similar, it will not be shown separately.


correct storage volumes and mount points.

To start a complete SAP instance including preparation:

Log in to SAP LVM Chose „Operations‟ Chose tab „Services‟ Select the SAP

system be started Select or enter the resource for each instance to be started Select

„Mass Start (including prepare if possible) Chose ‚Execute„

Figure 88 shows the steps to start a complete SAP system as mass start operation.



Figure 88 Start SAP system as mass operation

Figure shows the logs.

Figure 89 Logs of the 'Start SAP system' operation



5.1.2.3 Relocate SAP instances and systems

The following section describes the required steps to relocate a complete SAP system using

the adaptive relocation. That is, only the SAP system (not the operating system) is moved to

another LPAR. As the complete SAP system is relocated, this is a mass operation.


correct storage volumes and mount points, and it needs to be running.

To start the relocation of a complete SAP system:

Log in to SAP LVM Chose „Operations‟ Chose tab „Services‟ Select the SAP

system be relocated Select both instances to be relocated Select or enter the target

resource for each instance Select „Mass Relocate All in State „Running‟‟ Chose

‚Execute„

Figure 90 shows how to trigger the relocation of SAP system as a mass operation.

Figure 90 Relocation of SAP system as mass operation



5.1.2.4 Start / stop resources

The following section describes the required steps to stop a LPAR. As the unprepare process

is very similar, it will not be shown separately.

Make sure that all SAP systems and instances are stopped and unprepared. Otherwise there

might be data loss, or the database might get corrupted.

To stop a LPAR:

Log in to SAP LVM Chose „Operations‟ Chose tab „Resources‟ Select the LPAR to

be stopped Chose tab Virtual Resource Chose „Deactivate‟ and select „Power Off‟

Figure 91 shows how to stop an LPAR.

Figure 91 Stop LPAR



5.1.2.5 Monitoring

The following section describes how to access the different monitoring functions.

Dashboard

The Dashboard is the first screen that is opened per default when you log in to LVM. It

provides information e.g. regarding services (SAP systems), physical resources, and virtual

resources. By selecting „Add Pool‟ you can open a „window‟ showing an additional subset of

information. Figure 92 shows an example of the Dashboard.

Figure 92 Dashboard in SAP LVM



Visualization

The „Visualization‟ tab provides a visual view on the infrastructure and SAP systems and

instances running on the infrastructure.

Figure 93 shows an example of the Visualization functionality.

Figure 93 Infrastructure Visualization in SAP LVM

Performance monitoring

Performance monitoring enables you to display metrics for specific entities (such as SAP

systems, specific instances, or virtual machines) for a specific time period. The historical

monitoring data is fed to automatic capacity management (ACM) to provide response time

data.

To open the performance monitor:

Log in to SAP LVM Choose Monitoring Chose tab „Performance‟ select one or

more entities on the left (for example, application server) click the metric you want to

display (for example, CPU usage) and the period (for example, hour)

To add entities to the display of a metric, specify the entity in the Entity field (or choose the

entity using the value selector) and choose Add.

To add the metric to the dashboard as a pod, choose Add to Dashboard. If you have added

entities to the display of the metric, the metric is displayed on the dashboard only for the first

entity.



Figure 94 shows an example of the Performance Monitor.

Figure 94 Performance monitoring in SAP LVM

5.1.3 OS Provisioning

SAP LVM provides an interface for provisioning of AIX partitions through IBM Systems

Director VMControl. SAP LVM and VMControl coordinate and automatically execute all the

provisioning workflow steps that are:

Optional creation of „mksysb“ OS image on NIM server

Definition of LPAR profile on IBM Power HMC

Allocation of storage volumes required for rootvg

Connection of storage volumes with the LPAR incl. FC zoning definition, VIOS

mapping or NPIV tunneling and host bus adapter masking on storage subsystem

Installing of „mksysb“ image on a newly created partition

Registration of the provisioned operating system in IBM Systems Director and

gathering of the OS inventory data

SAP LVM interacts in this scenario as a user interface and workflow coordinator. All the

steps are explicitly or implicitly executed on IBM Systems Director VMControl itself. SAP

LVM provides three scenarios where OS provisioning is involved:

Provisioning of a new SAP LVM virtual resource. By that scenario SAP LVM

user chooses a source operating system or a predefined “mksysb” image as

template and provisions an additional SAP LVM virtual resource based on this

template. In case a template is a running partition – SAP LVM creates a temporary

“mksysb” image first. That image is deleted at the end of the provisioning process.

This scenario is available in SAP LVM standard and enterprise editions. We

describe this scenario below in this chapter.

Provision a new SAP system clone / copy target. This scenario is a subtask of SAP

clone / copy scenarios described in chapters 5.2.35.2.5 and requires SAP LVM



enterprise edition. Instead of (re-) using an existing resource a new resource is

created during the process and is used as target for the SAP system clone/ copy.

Create an SAP clone / copy by provisioning of a new operating system. By this

scenario SAP LVM takes a partition where a SAP source system is active as

provisioning template: The provisioned operating system includes a clone of SAP

source system. SAP LVM prepares this clone for later usage and integrates it into

SAP LVM landscape. This scenario is not described in this cookbook because it

requires that SAP system is installed in the “rootvg” which is typically not desired

in customer environments. SAP LVM enterprise edition is required for the

execution of this use case.

All these scenarios require configuration of SAP LVM virtualization manager for IBM

Systems Director VMControl as described in chapter 4.6. SAP Note 1728222 – Configuration

of IBM Systems Director VMControl for SAP LVM describes what should be done on IBM

Systems Director VMControl itself in order to integrate it with SAP LVM. If you are able to

capture and deploy an operating system via IBM Systems Director VMControl using NIM

and selecting a particular IBM Power server and storage system then you can perform similar

provisioning operation over SAP LVM.

5.1.3.1 Provisioning of a new SAP LVM virtual resource

This scenario includes following subtasks:

Gathering user input for the provisioning process.

Optional creation of an “mksysb” image through IBM Systems Director

VMControl in case a running partition is taken as provisioning template. This

subtask corresponds to IBM Systems Director VMControl “Capture Image” task.

Provisioning of a new operating system. This subtask corresponds to IBM Systems

Director VMControl “Deploy Workload” task.

Register a provisioned operating system in IBM Systems Director. This subtask

corresponds to IBM Systems Director “System Discovery”, “Configure Access”

and “Collect Inventory” tasks.

Optional deletion of “mksysb” image in case it was implicitly created for the

provisioning process.

Below we provide screenshots (Figure 95) that show execution of OS provisioning process in

SAP LVM.

Navigate to Provisioning Virtual Resource Provisioning and choose SAP LVM

virtualization manager where the provisioning process should be triggered. You see a list of

preconfigured “mksysb” images and running partition which may be taken as provisioning

templates. An “mksysb” image must occur in IBM Systems Director VMControl as a valid

deployment image. Otherwise it will be not shown in this dialog. Only running LPARs with

AIX operating system that is discovered, unlocked and for which inventory is collected by

IBM Systems Director may be taken as templates for provisioning. Partitions that are stopped

or where corresponding operating system is not known on IBM Systems Director are not

shown in this dialog. Choose an image or a partition and press “Provision”.

https://websmp230.sap-ag.de/sap%28bD1lbiZjPTAwMQ==%29/bc/bsp/spn/sapnotes/index2.htm?numm=1728222

https://websmp230.sap-ag.de/sap%28bD1lbiZjPTAwMQ==%29/bc/bsp/spn/sapnotes/index2.htm?numm=1728222



Figure 95 OS Provisioning – Template Selection

Define a name for LPAR (Figure 96). We strongly recommend that you use not fully qualified

hostname of the operating system as LPAR name.

Figure 96 OS Provisioning – LPAR Name Definition

Choose IBM Power server that should host target LPAR (Figure 97). You may provision

LPARs which run in dedicated or in shared CPU usage mode. If you choose IBM Power

server or “Default Pool” – the provisioned partition will run in dedicated CPU usage mode. If

you provision LPAR into “Physical Shared CPU Pool” – the partition will share its CPU

cycles with other partitions running in that pool.



Figure 97 OS Provisioning – Target Server Selection

Define storage pool where storage volumes for rootvg of the provisioned system should be

allocated (Figure 98). Storage pools shown in the dialog correspond to the pools provided in

IBM Systems Director VMControl dialog for deployment of a new workload. These may be

external storage subsystems and storage pools available on VIO servers.

Figure 98 OS Provisioning – Storage Pool Selection

Define parameters required for execution of provisioning process (Figure 99). The number of

configuration parameters varies depending on what is chosen as provisioning template.



Figure 99 Provisioning – Definition of Provisioning Parameters

In case you use an “mksysb” image as provisioning template you may / must specify

following parameters:

Hostname for the operating system

CPU Entitlement. The guaranteed physical CPU capacity for the virtual

system. This parameter is relevant in case you provision into “Physical Shared

CPU Pool” and is not relevant in case you partition gets dedicated CPUs. The

shown value is a value that was configured as default in the LPAR profile of

the captured partition. You may specify values within the certain range that is

shown in description of this parameter. The value configured here must fit to

the number of virtual CPUs specified by another parameter.

Virtual CPUs. The number of online virtual CPUs. The shown value is a value

that was configured as default in the LPAR profile of the captured partition.

You may specify values within the certain range that is shown in description of

this parameter. The value configured here must fit to the CPU entitlement

specified by another parameter.

Partition Memory. Online partition memory. The shown value is a value that

was configured as default in the LPAR profile of the captured partition. You

may specify values within the certain range that is shown in description of this

parameter.

OS User / OS Password. The OS credentials are required for gathering

inventory on the deployed operating system. The inventory gathering is

required in case you plan to use the new partition as source for further



deployments. These parameters are optional but we strongly recommend to

define them.

Virtual Network. Virtual network adapter (VLAN) of the primary network.

Use F4/OVS to see possible input values.

Additional network specific parameters (Short host name for the system, Static

IP address and netwrk mask for the network adapter specified by parameter

„Virtual Network“, Default IPv4 gateway, DNS domain name for the system,

IP addresses of DNS servers for system). Specify parameters required for

configuration of the OS network adapter.

NIM Resource or Resource Group. Specify the name of an existing NIM

Resource or NIM Resource Group to allocate during the deployment. Any

defined NIM Resource Group, or Resource of class "resources" can be

specified, except: mksysb, spot, lpp_source, ovf_vm, master.

In case you use a running LPAR as provisioning template you may / must specify following

parameters:

Hostname for the operating system.

OS User / OS Password. The OS credentials are required for gathering inventory

on the deployed operating system. The inventory gathering is required in case you

plan to use the new partition as source for further deployments. These parameters

are optional but we strongly recommend to define them.

Virtual Network. Virtual network adapter (VLAN) of the primary network. Use

F4/OVS to see possible input values.

Image Repository. Define repository that is capable to store images of the source

partition. The image created trough this process will be deleted automatically. Use

F4/OVS to see possible input values.

In the last configuration step (Figure 100) you may validate your configuration and choose if

you would like to register the operating system that will be provisioned as SAP LVM virtual

resource. The checkbox „Power On Virtual Resource After Provision“ is not relevant for

provisioning using NIM through IBM Systems Director VMControl. A provisioned system

will be started automatically at the end of the provisioning process.



Figure 100 Provisioning – Final Parameter Confirmation

After you click on provisioning SAP LVM starts the process. You may navigate to

monitoring and check the status (Figure 101). After some minutes (23 minutes in our

environment) you may use the new additional SAP LVM virtual resource.

Figure 101 OS Provisioning – Process Monitoring



5.2 SAP LVM Enterprise Edition Use Cases

The execution of the SAP LVM Enterprise edition use-case require that

LVM Enterprise Edition (VCM_ENT.SCA) is deployed to the managing SAP

NetWeaver 7.3 Java Web AS system.

5.2.1 SAP System Copy Use Cases

The SAP LVM Enterprise Edition Use-Cases include the SAP System Copy Use-Cases

Figure 102 illustrates the three different scenarios for SAP System copies in SAP LVM:

SAP System Clone

Initial SAP System Copy

SAP System Copy Refresh

Figure 102 SAP System Copy scenarios supported by SAP LVM (Source SAP)

These are discussed in more detail in the following chapter.

5.2.1.1 SAP System Clone

In the SAP System Clone scenario, an exact duplicate of an existing SAP system is created.

Content and system configuration are 100% identical in all aspects to the original system. To

avoid conflicts e.g. with interfaces dealing with the original system, the clone system is fully

isolated from the rest of the environment before the SAP target system can be started:

Therefore, the clone system is established in an isolated networking environment. AIX ipsec

allows to restrict the outbound TCP/IP network communication to hosts/ ports which don‟t

participate in any interface communication of the SAP system.

SAP System cloning is realized in two different versions:



Storage based System Clone

For storage based cloning the system to be cloned must be “adaptive enabled”: This

includes

o virtual IP addresses for both database and central instance

o Storage Setup separating all the SAP and database entities from the operating

system image. The SAP and database entities must be deployed to storage

supported by the SAP LVM storage adapter.

The target resource must match all prerequisites for a relocation target of adaptively

installed systems, e.g. the users, services, etc. must exist on it.

Virtualization based

For virtualization based System Cloning the system to be cloned must be installed into

the rootvg of a LPAR that is managed the SAP LVM virtualization manager, and

which is registered with SAP LVM. In addition virtualization based cloning is

supported with an “offline database copy” only. During virtualization based System

Cloning, the SAP System is stopped, the LPAR is captured via IBM Systems Director/

VMControl, and is deployed to the new target LPAR afterwards.

5.2.1.2 Initial SAP System Copy

During a SAP System Copy a duplicate of an existing SAP system is created with a different

system name and a unique system ID. The target system is updated with the new system

information, and can run in the same network environment as other SAP systems.

As a first step the System Copy involves the creation of a system clone to create the initial

target system. Then this initial target system is modified during subsequent execution steps to

update the settings according to the new SAP System ID. A special version of SAPInst,

“SAPInst Rename” is used to change the SAP system ID during this process. Additional post

configuration tasks in the SAP System are achieved via the Post Copy Automation (PCA)

tool. PCA provides task lists, with a predefined sequence of configuration tasks to configure

the extensive technical scenarios automatically. In the SAP LVM environment the execution

of a PCA task list is embedded in the overall System Copy workflow.

5.2.1.3 SAP System Refresh

Given are two systems 'SAP source' and 'SAP target' which exist and are configured in a

copy-relationship (target is copied from source) SAP LVM allows to 'refresh' the 'target'

system by:

exporting configuration data from the 'target' system

SAP System Copy for copying the source system to the 'target' system

(same target SID, same hostnames as previous system copy)

importing the formerly exported configuration data into the new 'target' system

The System Refresh feature has the same requirements as the System Copy.

5.2.1.4 SAP System Rename

The SAP System Copy can be executed in two steps also:

Create a SAP System Clone in a first step

Then use this clone later on, and rename it to a new SAP System ID. Like in scenario

5.2.1.2, the SAPINST rename tool is invoked for the SAP System ID change.



At first, the cloned SAP system needs to be enabled for the SAP System Rename.

In Configuration Services, select the newly cloned System, and press “Edit”.

In the configuration screen, the activate the checkbox “Renaming” and press “Save”.

1

2

1

2

Figure 103 Enable the cloned System for SAP System Rename

Now, the SAP System Rename can be invoked. (See chapter 5.2.7)

5.2.2 SAP ACM (Automatic Capacity Management) Use-Cases

The SAP ACM Use-Cases are SAP LVM enterprise use cases also: They will however not be

discussed in further detail here. Just note that the following considerations need to be taken

into account for the ACM Use-Cases:

Both adaptively installed SAP systems and traditionally installed SAP systems are

supported for ACM

One dialog instance per resource LPAR is supported

All ACM managed dialog instances must be in the same set of logon groups reserved

for use by ACM.

SAP CCMS must be installed and operational for each managed SAP system.

For ACM-managed VMs in the same LVM pool, all the OS users must be configured

identical and must be defined in the central user store (LDAP).



5.2.3 Required preparation steps

Table 13includes an overview about the required initial setup step before a system can be

cloned, copied, refreshed or renamed with SAP LVM. For each of the required steps it

references to the section in this document where the step is described in more detail.

Setup Procedure in SAP LVM

Clone System Copy System Refresh System/ Database

Rename System

Configuring Networks

Required (See chapter 4.7.1)

Configuring Storage Providers

Required, if the source system is adaptively installed. (See chapter 4.2)

Not Required

Configuring User Management

Not part of the test environment: All users for the target systems had been pre-defined in LDAP

Configuring Name Servers

Not part of the test environment: All virtual hostnames and IP addresses had been pre-defined in DNS.

(See chapter 5.2.4.1 for additional information)

Configuring Remote Function Call Destinations

Not Required Required

(See chapter 5.2.5.2)

Setting the Infrastructure Assignment

Required (See chapter 4.4.1.1 and 4.4.1.2)

Configuring System Rename Configurations

Not Required Required

(See chapter 4.7.2)

Configuring Mount Points

Required, if the source system is adaptively installed (See chapter 4.4.1.3 and 4.4.1.4)

Configuring Java Post Copy Automation

Not Required Not part of the test environment:

SAP ABAP Stack tests only

Configuring a universally unique identifier (UUID)

Required (See chapter 3.4.5.4)

Not Required

Enabling the system for (See chapter 5.2.1.4, Figure 103)

cloning copying refreshing renaming

Table 13 Required setup procedures

5.2.4 Execution of a SAP System Clone

The following section describes the required steps for starting the SAP system copy in SAP

LVM. As a prerequisite, a SAP source system need to be installed and discovered in SAP

LVM, and need to be enabled for SAP System cloning in the SAP LVM configuration

A SAP System Copy is started in the “System and AS Provisioning” tab in the “Provisioning”

view. A SAP System is selected, and the “Clone System” Button is entered. See Figure 104:



11

Figure 104 Invoke SAP System Clone

SAP LVM starts the System Cloning dialog: In the first screen, the Storage Pool is shown,

and a short name and a description for the target system are specified (Figure 105).

Accept the default settings, and press “Next” to get to the next screen.

11

Figure 105 SAP System Clone: Basic data for the target system

In the next screen (Figure 106) the target resource for the SAP system clone is specified:

1. In this example the default value “Use Existing Resource” was choosen. SAP LVM

takes one existing LPAR out or the pool of resources. The SAP System will be cloned

and attached to this already existing LPAR.

Alternatively “Provision New Resource” could be selected. Then a new LPAR will be



created as the target resource for the SAP System Clone. SAP LVM calls the

virtualization manager (IBM system Director/VMControl) to deploy the new target

resource.

2. Press “Next” to continue with the next selection screen

1

2

1

2

Figure 106 SAP System Clone: Resource selection

In the next screen, the virtual host names and the network for the SAP system clone are

defined (Figure 107). The virtual hostname per default is derived from naming conventions.

The short name specified in Figure 105 is inserted into the virtual hostname.

5.2.4.1 Excursus: DNS Name resolution considerations

DNS name server considerations

All target IP addresses for the “clone” IP labels are predefined in DNS by an appropriate

address range. Assuming the pre-defined IP addresses (see the example in Table 14),

Role IP Label (DNS server) IP Address

SAP Source cl6ci01 192.168.165.116

cl6db01 192.168.165.126

SAP Target cl6cloneci01 192.168.165.136

cl6clonedb01 192.168.165.146

Table 14 Examples for IP addresses

SAP Note 1572841 - Setup of name server update of SAP LVM describes an alternative

solution where SAP LVM automatically updates/creates virtual hostnames in the DNS server.

SAP LVM then creates the respective A (forward) and PTR (reverse) lookup records for IPv4

addresses in the DNS server at the runtime during the SAP System Clone operation. During a

SAP System Destroy operation these records previously added are removed. This mechanism

ensures that in the DNS server the new IP addresses are assigned to the new IP label. In the

test environment however all virtual hostnames and IP addresses had been pre-defined in

DNS: The automatic name server update was not tested here.




DNS behaviour on the target resource

On the target resource, things may look slightly different: SAP LVM acquires the new IP

Adresses on the target resource during the “prepare” operation, e.g.

On the target resource:

root> netstat -in

Name Mtu Network Address

en0 1500 link#2 86.1f.b8.54.6b.2

en0 1500 9.153.164 9.153.164.137

en0 1500 192.168.164 192.168.165.146

en0 1500 192.168.164 192.168.165.136

However, the Name resolution on the target resource still states the “source” IP labels. This is

required for SAP system operation of the clone. The clone system is a 100% identical clone of

the source system, and so the virtual IP labels need to be unchanged on the target resource (as

e.g. the virtual IP labels appear in the instance profiles, and start scripts, etc.).


root> netstat -in

Name Mtu Network Address

en0 1500 link#2 86.1f.b8.54.6b.2

en0 1500 9.153.164 siccp137

en0 1500 192.168.164 cl6db01

en0 1500 192.168.164 cl6ci01

To achieve that behaviour, SAP LVM adds new entries to the local /etc/hosts file:


root> cat /etc/hosts

…

192.168.165.136 cl6ci01.isicc.de.ibm.com cl6ci01 #SAP-LM: Added to

modify local hostname resolution for cloning DO NOT MODIFY

192.168.165.146 cl6db01.isicc.de.ibm.com cl6db01 #SAP-LM: Added to

modify local hostname resolution for cloning DO NOT MODIFY

So, with the “local” name resolution the new “target” IP addresses are mapped to the “source”

IP labels.To ensure that the mechanism works it is mandatory that the DNS search order

evaluates the local /etc/hosts first. This ensures that the local host file supersedes the

(external) nameserver. The nameserver gets contacted only in case the record to be resolved is

not found in the local hosts file.

root> cat /etc/netsvc.conf

# order of search: 1. /etc/hosts 2. nameserver

hosts = local,bind4

However, back to the selection screen of the SAP system clone (Figure 107):

1. The network name needs to be selected

2. Click “Next” to continue with the next screen



1

2

1

2

Figure 107 SAP System Clone: Network definition

In the next screen, all the storage definitions are made for the SAP system clone (Figure 108).

1. Select the Consistency Method: By selecting “Online Clone”, SAP LVM will place

the DB2 UDB database into “write suspend” state before the FlashCopy/ SnapShot is

triggered in the storage system. When the FlashCopy / SnapShot is finished, SAP

LVM will initiate a “write resume” to bring back the database into operational state.

2. Selection of “Full Clone” allows to create a new, independent set of target volumes

(which no longer depend on the source volume: So theoretically the source volume

could be removed later on)

3. Press next to get to the next selection screen.



1

2

3

1

2

3

Figure 108 SAP System Clone: Storage Definitions

In the next screen, all the port definitions for the network fencing of the SAP system clone are

specified (Figure 109).

The highlighted item in the allowed connections is the DNS name server: In the environment,

the standard /etc/hosts file didn‟t include entries for the virtual IP labels. All the name

resolution is done via DNS to the DNS server. So it is important that the LPAR can

communicate with the DNS server in fenced state also. Therefore the DNS server (here:

siccserv2) was added as allowed communication already during the system setup in SAP

LVM 4.4, Figure 41).

1. Select “Next” to get to the final confirmation screen



11

Figure 109 SAP System Clone: Network Fencing

The final confirmation screen appears (Figure 110): It is possible to review all the parameters

defined in all the steps before.

1. Press “Start System Cloning” to finally start SAP System Clone execution

11

Figure 110 SAP System Clone: Final parameter confirmation



After starting the SAP System Cloning process, the Monitoring screen is launched and the

progress of the SAP System Clone can be tracked.

5.2.5 Execution of a SAP System Copy

The following section describes the required steps for starting the SAP system copy in SAP

LVM. As a prerequisite, a SAP source system need to be installed and discovered in SAP

LVM, and need to be enabled for SAP System copies in the SAP LVM configuration.

5.2.5.1 Preparations for Post Copy Automation (PCA)

Post Copy Automation (PCA) is part of SAP NetWeaver Landscape Virtualization

Management, enterprise edition solution. This section illustrates how to set-up PCA initially.

Several preparation steps before set-up of SAP Post Copy Automation need to be executed in

the source system first: Details are described in the document "ABAP Post Copy Automation

Guide for Enterprise Edition" . See SAP NetWeaver Landscape Virtualization Management -

Installation & Upgrade Guides SAP NetWeaver Landscape Virtualization Management

1.0 and in SAP Note 1614266 - System Copy: Post Copy Automation (PCA) / LVM 1.0.

The file "Note_Overview.pdf" (which is attached to SAP Note 1614266) list a set of

additional SAP notes which need to be implemented as a prerequisite for PCA in the source

system. Dependent on the actual support package stack of the SAP system they may (or may

not be) already included in the system, and corrections may need to be applied to the SAP

system (via TA SNOTES). The ABAP program Z_SAP_NOTE_ANALYZER (which is also

attached to SAP Note 1614266) automates this procedure. The SAP system is checked, and

according to its actual support package stack the required SAP notes content is downloaded

directly during execution of the from Z_SAP_NOTE_ANALYZER report, the content is

transferred into the SNOTES notes assistant transaction and added to the queue. If all the pre-

requisites are met, then PCA is installed via the “Post Copy Automation Installer” (PCAI)

automatically into the managed source system. See also

Note 1589145 - Task Manager for Technical Configuration and

Note 1589175 - System Copy: Task Content for Task Manager

Further details are described in chapter 3 and chapter 4 of the “ABAP Post Copy Automation

Guide for Enterprise Edition“.

PCA is part of the enterprise edition of the SAP NetWeaver Landscape Virtualization

Management solution. PCA is shipped within SAP NetWeaver packages, but is deactivated

per default. The License Enabler add-on PCAI_ENT 1.0 is activates PCA and is so required in

the system to enable the execution of PCA task lists. Install the add-on PCAI_ENT 1.0 before

the first system copy of the system using transaction SAINT. Details are described in chapter

5 of the “ABAP Post Copy Automation Guide for Enterprise Edition“.

To grant the privileges to execute PCA activities for the administrator (dialog user) the

corresponding roles SAP_BC_STC need to be assigned to all the relevant users. This includes

the system administrator user responsible for PCA activities, and the technical users that will

be used via remote access by SAP LVM. Details are described in chapter 7 of the “ABAP

Post Copy Automation Guide for Enterprise Edition“.

PCA provides task lists, with a predefined sequence of configuration tasks to configure

extensive technical scenarios automatically. To prepare the run of these task lists the ABAP-

based “task manager for technical configuration” is used. In the SAP LVM environment the

http://service.sap.com/instlvm

http://service.sap.com/instlvm






execution of such a task list is induced without calling the task manager for technical

configuration directly. Further details are described in chapter 8 and chapter 9 of the “ABAP

Post Copy Automation Guide for Enterprise Edition“.

5.2.5.2 Add the RFC destinations for PCA to the SAP System configuration

In the services configuration, define the RFC destinations that will be used for the PCA

execution for the SAP system. Define the logon parameters for the main destinations, and

press “Test Connection”.: An attempt to connect to the SAP system is made.

Press “Retrieve Clients” and the list will be populated with all the clients of the existing

system. Different PCA activities may be executed specific for each client of the SAP system.

Adjust the Users accordingly, and define the passwords.

Figure 111 Define RFC destination for PCA execution

5.2.5.3 Start the SAP System Copy

A SAP System Copy is started in the “System and AS Provisioning” tab in the “Provisioning”

view. A SAP System is selected, and the “System Copy” Button is entered. Figure 112 then

shows an example for the screen of the first selection step. First configuration data for the

target system needs to be entered.

The SAP System ID for the target system is specified, and the master password needs to be

entered. SAP System ID and master password will be used later on during the

“SAPINST_RENAME” step to adapt the configuration. The additional description may be

adjusted additionally, or left on the default value. This will be the description for the new

system in the SAP LVM configuration.



Figure 112 SAP System Copy – Basic data

Finally, data entry for Step 1 is confirmed by clicking the “Next” button.

In the second step the target resource(s) for the system copy is defined. Figure 113 shows the

screen for the second “Resource Selection” step. By selecting type “Use Existing Resource”

SAP LVM suggests a potential target, which fulfils the boundary conditions (capacity

requirements in SAPS and memory) for the target system. LPARs satisfying these conditions

can be selected in a drop-down list.

Alternatively “Provision New Resource” will trigger IBM Systems Director/ VMControl

respectively IBM Flex Systems Manager to provision a new LPAR as target for the system.

The services for the target system (Central Instance, Database) can be provisioned either

together to one target LPAR, or separated to different target LPARs.

Figure 113 SAP System Copy - Resources

Data entry for Step 2 is confirmed by clicking the “Next” button.



In the next steps the network configuration for the target system is specified (Figure 114). The

IP addresses for the target system are proposed according to the naming conventions for the

virtual IP labels (<sid>ciXX, <sid>dbXX). The addresses and labels are either predefined in

the DNS (respectively in the local host files of the SAP LVM system and the involved

resources) already.

Otherwise, SAP LVM may directly update the DNS domain name server for the new entries,

however that need to be defined and configured according to SAP Note 1572841 - Setup of

name server update of SAP LVM first.

A network name for the Network needs to be selected by using the drop-down list (The

network name was defined before, see chapter 4.7.1).

Figure 114 SAP System Copy – Network settings


In Step 4 the Instance number(s) for the Instances of the target system are defined (Figure

115). Per default, the target system will be installed using the same instance numbers like the

source system.

Figure 115 SAP System Copy – Instance Number






In step 5 the storage configuration for the target system is defined (Figure 116). The storage

image for the target system is created as a SnapShot/ FlashCopy of all source volumes to the

target volumes. The main structure like the overall number of AIX volume groups and their

content cannot be changed. Filesystem names can be adjusted however. Per default, all SAP

System IDs in the filesystem names are updated to the new SAP System ID of the target

system.

The consistency settings need to be specified: In the IBM Tivoli Storage FlashCopy Manager

scenario, specify “Database Consistency (Online Clone)”.

Figure 116 SAP System Copy – Storage Definition


In Step 6 the OS administration users are specified for the target SAP system (Figure 117). If

LDAP is used for central user management, then the users need to be created manually on the

target resource before the SAP system Copy is started. “sapinst” currently is not able to

handle LDAP users correctly, and also the SAP LVM “Create User” step doesn‟t work with

AIX.



Figure 117 SAP System Copy – User definitions


In Step 7 further additional settings for DB2 schema user and DB2 instance are specified for

the new SAP System ID (Figure 118). For DB2 UDB, the schema user cannot be changed

during the process, and must be set identical to the schema user of the source database. Name

of the database instance for the target system is derived from the SAP System ID according to

the SAP naming conventions.

Figure 118 SAP System Copy – Rename SAP system ID


In Step 8, the allowed ports/ connections for the network fencing step are defined (Figure

119). Additional ports/ hosts allowed for communication can be selected out of the set of

“Current connections on host” from the source system, and then added to the “allowed

outgoing connections on the target hosts”. To avoid additional manual steps it is best-practise



to include all the necessary hosts/ ports required for outbound connections of the system

already in the SAP system definition.

Figure 119 SAP System Copy – Network fencing


In Step 9 the SAP clients for executing the Post-Copy Automation (PCA) steps can be

defined. In the test environments the SAP system copy was run without subsequent execution

of the PCA for the first test runs in any of the clients. This can be achieved by removing all

RFC destinations in this PCA section. (See Figure 120). Although the RFC destinations are

removed from the configuration here, the RFC definitions need to be present at first in the

base configuration of the SAP system in SAP LVM to enable it for the SAP system copy.

Figure 120 SAP System Copy - Select RFC destination for PCA




Before actually starting the SAP System Copy a summary screen is displayed: All the

parameter settings can be reviewed by expanding the different subsections, and eventually

adjust them once again (Figure 121).

Figure 121 SAP System Copy – Summary view

The SAP System Copy process is finally started by clicking the “Start System Copy” button.

After starting the SAP System Copy, the Monitoring screen is launched and the progress of

the system copy can be tracked.

5.2.6 Execution of SAP System Refresh

Prerequisite for the SAP System Refresh is a SAP System Copy run that was previously

completed successfully. The execution of the SAP System Rename is very similar to the steps

before. Start with the Provisioning view. Press “Refresh System” to start the parameter dialog

(Figure 122):

11

Figure 122 SAP System Refresh: Start system refresh

In the first screen base configuration parameters can be adjusted (Figure 123):



1. Specify and confirm the Master Password

2. Press “Next” to continue with the next selection screen

1

2

1

2

Figure 123 SAP System Refresh: Basic Data

Acknowledge target resource description in the following screen and press “Next” (Figure

124):

11

Figure 124 SAP System Refresh: Resource Data



Also, acknowledge the virtual hostnames and network settings (Figure 125). Press Next:

11

Figure 125 SAP System Refresh: Host Names

In the next screen, the storage settings are defined (Figure 126):

1. Decide if a full clone (target volumes are fully independent from the source volumes)

shall be created

2. Select the appropriate consistency method (typically “Online”)

3. Click “Next” to proceed with the next screen

1

2

3

1

2

3

Figure 126 SAP System Refresh: Storage Settings

Acknowledge the User settings. As it is a SAP system refresh, the Users should already exist.



Press Next (Figure 127).

11

Figure 127 SAP System Refresh: Users

Enter the password for the schema user (Figure 128). Press “Next” to proceed.

1

2

1

2

Figure 128 SAP System Refresh: DB Schema



Acknowledge or eventually adjust the isolation settings during the network fencing (Figure

129).

11

Figure 129 SAP System Refresh: Network Isolation

Adjust or confirm the PCA settings (Figure 130). Press “Next” to enter the summary screen.

11

Figure 130 SAP System Refresh: PCA settings

Eventually review the settings. Finally invoke the SAP System Refresh by pressing “Start

System Refresh” (Figure 131).



11

Figure 131 SAP System Refresh: Parameter summary

5.2.7 Execution of SAP System Rename

The execution of the SAP System Rename is very similar to the steps before. Start with the

Provisioning view. The system needs to be enabled for SAP System Rename: Only then the

appropriate button is shown. Press “Rename system” to start the parameter dialog (Figure

132).

11

Figure 132 Start SAP System Rename



In the following screen (Figure 133) enter the basic data for the SAP System Rename.

1. Enter and confirm the Master-Password

2. Press Next

1

2

1

2

Figure 133 SAP system Rename: Basic Data

The following screen appears (Figure 134). For convenience, the virtual hostname can be

changed here, as the one from the system clone will not fit to the naming conventions.

1. Then, assign a name to the network

2. Press “Next” to proceed to the next screen



1

2

1

2

Figure 134 SAP System Rename: Virtual Hostnames

The SAP Instance number for the renamed system can be specified or, per default, use the

same as the source system (Figure 135). Press “Next to proceed to the next screen.

11

Figure 135 SAP System Rename: Instance numbers

In the following screen, the mount points are adjusted to the new SAP System ID (Figure

136). In most cases you can accept the default names, except if you specifically want to

update certain mount points. Afterwards, proceed with “Next” to the following screen.



11

Figure 136 SAP System Rename: Mount points

The administrative users for the new SAP System are selected (Figure 137). As SAPinst

currently is not able to create LDAP users correctly, and also the SAP LVM user method

doesn‟t work correctly on AIX, create users and groups manually in LDAP before.

If the users are already existing (in LDAP), then the checkbox “Already exists” is

automatically flagged. Select “Next” to proceed to the following screen.

11

Figure 137 SAP System Rename: Specify users for target system



In the next step, the settings for the connect user are specified. For DB2, the connect user

cannot be changed during the System Rename.

1. Specify the password for the connect user

2. Press “Next” to proceed to the following screen

2

1

2

1

Figure 138 SAP System Rename: Specify database parameters

The next screen describes the allowed communications during the network fencing phase

(Figure 139). Ensure that all necessary connections (e.g. to the DNS nameserver) are allowed,

but that all potential dangerous communication is not explicitly allowed (and so prohibited).

Press “Next” to proceed with the following screen.



11

Figure 139 SAP System Rename: Configure network isolation

In this screen, the SAP target clients for Post Copy Automation execution are specified

(Figure 140). In this example, the Post Copy Automation was explicitly disabled. To disable

PCA, remove all the potential clients from the list.

11

Figure 140 SAP System Rename: Disable Post Copy Automation (I)

If the table is empty (Figure 141), then the PCA won‟t be invoked for any client. Click “Next”

to proceed to the final confirmation screen.



22

Figure 141 SAP System Rename: Disable Post Copy Automation (II)

In the final selection screen, the parameters can be checked before the actual start (Figure

142). Then press “Start System Rename” to start the execution.

11

Figure 142 SAP System Rename: Summary Screen



5.2.8 Additional Considerations for SAPInst Rename

See also SAP Note 1619720 - System Rename for SAP Systems based on SAP NetWeaver

for additional information.

Currently, there are some issues in the transfer of the parameterization for the SAPInst rename

execution from SAP LVM (at least for DB2 UDB/ AIX): The SAP LVM team is currently

working on that.

Additionally, the test environment identified the following issues in context of SAP system

copies:

1. The SAP System Copy in SAP LVM procedure doesn‟t handle DB2 UDB with

Automatic Storage Management (ASM) correctly.

o For the final relocation of the DB2 UDB database on the target system a

parameter file “relocate.template” is generated by the brdb6brt tool on the

source system. These parameter file doesn‟t include the required ASM

information for STORAGE_PATH and CONT_PATH information.

o The ASM directory structure follows the conventions

/db2/<SID>/sapdataX/db2<sid>/NODE0000/<SID>

Not all the <SID> and <sid> entries are adapted by SAP LVM to

the “new” target <SID> / <sid> values of the target system

Resolution: Handled in Event script, see below

2. SAPinst and SAP LVM don‟t handle user creation in LDAP correctly

Make sure that the ABAP Connect User on the target resource has the

following group sets assigned groups=db<src-sid>mon,db<src-

sid>mnt,db<tgt-sid>mon,db<tgt-sid>mnt


3. SAPINST_RENAME cannot handle the update to db2nodes.cfg if a “full qualified

hostname” is set as hostname on the target resource

If hostname and netname are specified in db2nodes.cfg, only the hostname

is changed. If only the hostname is specified, then it is translated

errornously


4. An issue with “db2chgpath” was identified in the test environment (DB2 UDB

V9.7 FP5SAP)

Resolution: Fixed in DB2 UDB V9.7 FP6 (APAR IC80849).

DB2 UDB V9.7 FP6 is available, however at the moment not yet certified by SAP.

5. Update to AIX 6.1 TL7 SP4 (6100-07-04-1216) introduced a new issue IV22062

on some of the nodes.


http://www-1.ibm.com/support/docview.wss?uid=swg1IC80849

http://www-304.ibm.com/support/docview.wss?uid=isg1IV22062



Resolution: Install eFIX on top of AIX 6.1 TL7 SP4

root> emgr -l

ID STATE LABEL INSTALL TIME UPDATED BY ABSTRACT

======================================================

1 S IV22062s04 06/11/12 12:42:43 Ifix [email protected]

Item 1.), 2.) and 3.) are addressed by a “Pre-Event” script, which is invoked before SAPInst

rename is started: This is realized via a “Pre-Hook” in SAP LVM. The following outlines how

to implement such a hook. See also SAP Note 1465491 - Provider Implementation Definition

for further details.

5.2.8.1 Create the script and register it with the Host Agent

The script is invoked for both SAP Central Instance and Database. It checks, if it is running

for the database, and then identifies the Source SID based on the environment. Target SID is

specified as argument. The script provides the following functionality

Check and eventually create/ correct the LDAP Users and Groups required for the

target system

Handle DB2 UDB ASM

o Update relocate.template to include storage and container pathes in the

template

o Correct directory structure below the /db2/<SID>/sapdataX filesystems

Update db2nodes.cfg (in the directory of the source instance)

When the script is created, then a .conf file needs to be created in the directory

/usr/sap/hostctrl/exe/operations.d .so that it can be invoked from the Hostagent.

cat PreSAPinst.conf

Name: PreSAPinst

Command: /usr/sap/hostctrl/exe/operations.d/IBM/PreSAPinst.ksh

$[SYSTEM_HOST:#required] $[system_sid:#required] $[SERVICE_CLASS:#required]

$[service_name:#required] $[service_type:#required]

Workdir: /usr/sap/hostctrl/exe/operations.d/IBM/traces

ResultConverter: flat

Platform: Unix

In the .conf file, the Script and its parameters are defined in the „Command:“ section.

The “Workdir:” contains the actual work directory for the script: Make sure that the directory

exists.

Restart the SAP Host Agent to register it.

5.2.8.2 Define the Provider Implementation Definition

For the Provider Implementation Definition, goto Setup Extended Settings Custom

Operations and Custom Hooks Provider Implementation Definition (Figure 143) in SAP

LVM. Choose “Add” to define a new definition.

mailto:[email protected]




Figure 143 Provider Implementation Definition (I)

In the following screen (Figure 144), enter

1. Name and select Type “Script Registered with Host Agent”

2. The hostname running a SAP Host Agent where the script was registered

3. Press “Retrieve Registered Scripts”

2

3

1

2

3

1

Figure 144 Provider Implementation Definition (II)



Now, select the Registered Script from the Pick List (Figure 145):

Figure 145 Provider Implementation Definition (III)

The registered script is a Custom Hook, and it shall be invoked for a Service (Database, SAP

Central Instance): Checkmark “Hook” in the row for “Service”. Afterwards, save the new

Definition (Figure 146)

1

2

1

2

Figure 146 Provider Implementation Definition (IV)



5.2.8.3 Define the Custom Hook Definition

The “Custom Hook Definition” then maps the script defined before to the “Pre-SAPInst”

Event. Change to the Custom Hook Definition View (Setup Extended Settings Custom

Operations and Custom Hooks Custom Hooks), and press “Add” (Figure 147):

Figure 147 Custom Hook Definition (I)

Figure 148 illustrates the next panel:

1. Enter a name, and select the provider implementation definition defined before

2. Then select the Hook type from the list

1

2

1

2

Figure 148 Custom Hook Definition (II)

1. Select “pre_SAPInst” to link the definition as Pre-Event to SAPInst Rename: During a

System Copy/ System Rename/ System Refresh activity the script is executed on the



target resource before the SAPInst step (Figure 149).

2. The picklist disappears, and the screen turns back. Press “Save” to confirm the

changes.

1

2

1

2

Figure 149 Custom Hook Definition (III)

5.2.9 Destroy of a SAP System

During the SAP system destroy action

the storage configuration and all the actual volumes for the systems will be deleted in

the storage system

IP labels, which were created in the DNS server during SAP system creation by SAP

LVM, will be removed from the DNS server once again

The SAP system will be deleted in the SAP LVM repository.

All services for the system needs to be in “unprepared” state before the “Destroy” action can

be started in SAP LVM. If the services are still running, then the system needs to be stopped,

and an uprepare operation needs to be started for it first.

5.2.9.1 Stop and Unprepare a SAP System

Change to the „Operations“ view and check the status of the system: As a prerequisite to be

able to start the “Destroy” operation all the services of the system need to be in state

“Unprepared”. In the example as illustrated in Figure 150 the system is still active: A mass

operation to “Stop and Unprepare” is selected for both the database and the SAP Central

Instance.



Figure 150 System State in Operations view

Press „Execute“ to start the mass operation: The service gets locked for other operations, and

the “Stop and Unprepare” operation is started (Figure 151):

Figure 151 Mass operation „Stop and Unprepare” started

The progress of the current action can be monitored via the “Monitoring” view. On the upper

section of the screen a status and progress overview is given. By selecting an operation from

the list, and choosing the “Steps” sheet the individual steps of the workflow can be monitored

(Figure 152):



Figure 152 Monitoring view

Wait until the „Stop and Unprepare“ operation is completed for all the services (Figure 153):

Figure 153 Mass Stop completed

5.2.9.2 Destroy the SAP System

Change to the „Provisioning“ view and select the system to be destroyed. The System needs

to be stopped and all the services need to be in state “Unprepared” as a prerequisite (Figure

154).



Figure 154 Destroy System

Press the button „Destroy“ System: Three additional selection/ information steps are

processed before the final destroy operation takes place.

During the destroy operation all the storage volumes which were deployed for the system

during the Install/ Copy/ Clone process are removed on the storage system. In the first step,

all these storage volumes to be deleted later on are shown (Figure 155):

Figure 155 Destroy operation – delete storage volumes

Press „Next“ to get to the next screen. If SAP LVM is configured for DNS updates and has

generated DNS entries for the system during a SAP LVM deployment process then these

entries will get removed during the destroy operation. This feature was not configured in the

test environment. Instead all IP addresses and corresponding DNS entries were preconfigured

in the DNS before. So in the example case (Figure 156) the list is empty.



Figure 156 Destroy operation – delete host names

Click next to proceed to the summary screen. The parameters are shown, and the actual

destroy operation is started by pressing the “Start System Destroy” button (Figure 157).

Figure 157 Destroy Operation – summary screen



6 Troubleshooting hints and tricks

6.1 Phase “Prepare Clone Volumes”

Error Message “Error while writing FCM profile <profile> on FCM target host <resource>”

Check .conf file FCMCLI.conf in /usr/sap/hostctrl/exe/operations.d on the host

<resource”. Does the “workdir” exists, and is user “root” allowed to access it?

6.2 Cleanup activities on a resource LPAR

Stop SAP service and all agents for it

Stop the SAP Hostagent on the LPAR

Un-mount all the filesystems belonging to the SAP Service

lsvg |grep -v rootvg |while read i ; do

lsvgfs $i

done | sort -r | while read fs; do

mount | grep -w ${fs}

if [[ $? -eq 0 ]]

then

fuser -cuk $fs

umount $fs

fi

done

Vary-off and export all the volume groups

lsvg |grep -v rootvg |while read i ; do

varyoffvg $i && exportvg $i

done

Delete the disk devices corresponding to those items

Delete virtual hostnames from the network adapter

Remove all IP filters in IPSec

rmfilt -v 4 -n all && mkfilt -v 4 –u

Clean-up /usr/sap/sapservices

Remove all entries set by SAP LVM on the target resource

o /etc/hosts

o /etc/services

Restart SAP Host Agent



6.3 Clean-up orphaned volumes in the IBM XIV

Search for volumes which belong to the cloned system and are mapped to the target

resource:

o Un-map all these volumes (“unmap_vol”)

o Delete the volumes (“vol_delete”)

Search for temporary volumes created during the cloning process (Prefix “TSM”,

respectively the prefix defined in the FCM manager profile)

o Delete the volumes

6.4 Delete the System in SAP LVM

Goto Configuration Services

Then select the appropriate system, and press “Remove Service and System” (Figure 158).

Confirm deletion then in the following conformation dialog.

Figure 158 Remove System in SAP LVM



7 Resources

7.1 IBM Documentation

7.1.1 Whitepapers

Flyer “Cloud-enabled SAP application management”

IBM Cloud Solution for SAP – Integrating IBM Infrastructure with SAP Landscape

Virtualization Management

Introduction to an Integral IBM / SAP Cloud Solution for SAP Landscape Management

IBM entry cloud configuration for SAP solutions on Power Basic Concepts and High Level

Design

7.1.2 Redbooks / Technotes

Deploying Cloud Components on POWER

IBM Flex System Manager

IBM PureFlex System and IBM Flex System Products and Technology

IBM Flex System p260 and p460 Compute Node

7.1.3 Product Documentation

Requirements and support for AIX using Network Installation Manager (NIM)

Requirements and support for AIX, IBM i, and Linux using storage copy services (SCS)

7.1.4 iRAM

IBM Systems Director VMControl Implementation Perform Guide

7.2 SAP Documentation and SAP Notes

7.2.1 SAP Documentation

SAP NetWeaver Landscape Virtualization Management

SAP Community Network (SCN) Virtualization and Cloud

SAP Help Portal (SAP LVM)

SAP NetWeaver Landscape Virtualization Management, Enterprise

SAP NetWeaver Landscape Virtualization Management at a Glance

http://public.dhe.ibm.com/common/ssi/ecm/en/sps03044deen/SPS03044DEEN.PDF

https://scn.sap.com/docs/DOC-24822



http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4803

ftp://submit.boulder.ibm.com/sales/ssi/ecm/en/pol03119usen/POL03119USEN.PDF

ftp://submit.boulder.ibm.com/sales/ssi/ecm/en/pol03119usen/POL03119USEN.PDF

http://www.redbooks.ibm.com/redbooks/pdfs/sg248009.pdf

http://www.redbooks.ibm.com/technotes/tips0862.pdf

http://www.redbooks.ibm.com/redbooks/pdfs/sg247984.pdf

http://www.redbooks.ibm.com/technotes/tips0880.pdf

http://publib.boulder.ibm.com/infocenter/director/pubs/topic/com.ibm.director.vim.helps.doc/fsd0_vim_r_power_virtualization_using_NIM.html

http://publib.boulder.ibm.com/infocenter/director/pubs/topic/com.ibm.director.vim.helps.doc/fsd0_vim_r_sb_aix_on_power.html

https://w3-03.ibm.com/tools/cm/iram/artifact/CFAA94C6-4CF7-6FC3-9D66-C0ACB1AAAA81/1.0/Additional%20Delivery%20Assets/IBM%20Systems%20Director%20VMControl%20Implementation%20Perform%20Guide_v02.rar

http://www.sap.com/solutions/technology/cloud/virtualization/index.epx

http://scn.sap.com/community/it-management/virtualization

http://help.sap.com/nwlvm

http://help.sap.com/saphelp_lvm10enterprise/helpdata/en/4d/f88a8f418c5059e10000000a42189c/frameset.htm

http://scn.sap.com/docs/DOC-25771



7.2.2 SAP Notes

SAP Note Title / Description

Note 962955 Use of virtual TCP/IP host names

Note 1292836 Logfiles of the SAP Hostagent

Note 1438774 New profile parameter system/uuid and system/description

Note 1465491 Provider Implementation Definition

Note 1527538 SAP NetWeaver Landscape Virtualization Management 1.0

Note 1572841 Setup of name server update of SAP LVM

Note 1589145 Task Manager for Technical Configuration

Note 1589175 System Copy: Task Content for Task Manager

Note 1614266 System Copy: Post Copy Automation (PCA) / LVM 1.0

Note 1619720 System Rename for SAP Systems based on SAP NetWeaver

Note 1644520 Integration of IBM Components into SAP LVM

Note 1728222 Configuration of IBM Systems Director VMControl for SAP LVM

Note 1728293 Configuration of IBM Power HMC for SAP LVM

Table 15 List of SAP Notes
















COPYRIGHT LICENSE

If this information contains sample application programs in source language, which illustrate

programming techniques on various operating platforms, you may copy, modify, and

distribute these sample programs in any form without payment to IBM, for the purposes of

developing, using, marketing or distributing application programs conforming to the

application programming interface for the operating platform for which the sample programs

are written. These examples have not been thoroughly tested under all conditions. IBM,

therefore, cannot guarantee or imply reliability, serviceability, or function of these programs.

The information contained in this document has not been submitted to any formal IBM test

and is provided "AS IS" with no warranties or guarantees either expressed or implied.

All examples cited or described in this document are presented as illustrations of the manner

in which some IBM products can be used and the results that may be achieved. Actual

environmental costs and performance characteristics will vary depending on individual client

configurations and conditions.

IBM is not responsible for printing errors in this document that result in pricing or

information inaccuracies.

A full list of U.S. trademarks owned by IBM may be found at:

http://www.ibm.com/legal/copytrade.shtml

A full list of trademarks owned by SAP may be found at:

http://www.sap.com/company/legal/copyright/trademark.epx

SAP and other SAP products and services mentioned herein, as well as their respective logos,

are trademarks or registered trademarks of SAP AG in Germany and in several other countries

all over the world.

UNIX is a registered trademark in the United States, other countries or both.

Linux is a registered trademark of Linus Torvalds in the United States, other countries or

both.

Microsoft, Windows, Windows NT and the Windows logo are trademarks of Microsoft

Corporation in the United States, other countries or both.

http://www.ibm.com/legal/copytrade.shtml

http://www.sap.com/company/legal/copyright/trademark.epx

Documents

Landscape Virtualization Management