View
59
Download
1
Category
Tags:
Preview:
Citation preview
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 1 of 169
IBM SAP Technical Brief
Implementing and Using
SAP NetWeaver Landscape Virtualization Management
on IBM Power Systems and IBM PureFlex Power-nodes
IBM SAP International Competence Center
Walldorf, Germany
Version: 1.0
Status: July 2012
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 2 of 169
Preface Edition Notice (July 2012)
SAP NetWeaver Landscape Virtualization Management software (SAP LVM) is the
successor and extension of the SAP Adaptive Computing Controller. This document is
complement to the ISICC Implementation Guide “SAP‟s Adaptive Computing on Power
Systems”, Version 3.0, August 2011. The paper focuses on the new features and uses-cases
provided with SAP LVM in conjunction with IBM Systems Director managed Power
Systems. It describes the IBM infrastructure specifics required to implement and operate an
on-premise Cloud Solution for SAP landscapes.
The cookbook covers a solution stack tested during a Proof-of-Concept at the ISICC. This
includes SAP Business Suite 7 with IBM DB2 UDB LUW 9.7 database on IBM Power
Systems using AIX 6 and AIX 7 operating systems, IBM Systems Director, IBM Tivoli
Storage Flashcopy Manager and IBM XIV Storage System. At the time of writing, SAP LVM
V1.0 SP3 is generally available.
Scope and Audience
People reading this document should understand core components and pre-requisites of the
SAP and Systems Software solution stack in an IBM POWER environment including their
installation and handling. They should understand the typical use-cases and their execution
and effects on a POWER/AIX based SAP infrastructure.
The document is intended for IBM (Practitioners from Pre- & Post-Sales and Services,
GBS/GTS, ATS, eTS,) and customer personnel involved in the implementation of SAP
Landscape Virtualization Management software on IBM Power Systems and IBM Systems
Director. Since we do not repeat existing documentation and technical information from both
IBM and SAP the readers are expected to have a solid knowledge in AIX, PowerVM
virtualization, IBM Systems Director and its plug-ins, FlexSystems Manager, IBM Tivoli
Storage FlashCopy Manager and SAP Basis.
Authors
Dr. Edmund Haefele, IBM Technical Sales for SAP
Dr. Thomas Hebert, IBM Technical Sales for SAP
Maik Gasterstaedt, IBM SAP International Competence Center
Walter Orb, IBM SAP International Competence Center
Matthias Koechl, IBM SAP International Competence Center
Arnold Beilmann, IBM R&D
Feedback
We are interested in any feedback you have. Please send your comments to
isicc@de.ibm.com.
Disclaimer
This document is subject to change without notification and will not comprehensively cover
the issues encountered in every customer situation. It should be used only in conjunction with
the product literature accompanying the products listed above. The information contained in
this document has not been submitted to any formal IBM test and is distributed AS IS.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 3 of 169
1 INTRODUCTION ....................................................................................................................................... 9
1.1 BACKGROUND AND BUSINESS CONTEXT .............................................................................................. 9 1.2 SCOPE OF THIS COOKBOOK ................................................................................................................... 9 1.3 SAP NETWEAVER LANDSCAPE VIRTUALIZATION MANAGEMENT SOFTWARE (SAP LVM) ............... 10
2 OVERVIEW AND PLANNING ............................................................................................................... 13
2.1 ARCHITECTURE OVERVIEW ................................................................................................................. 13 2.2 COMBINING THE IBM AND SAP LAYERS ............................................................................................ 15 2.3 PLANNING PROJECT RESOURCES AND TIMING .................................................................................... 20 2.4 TESTED CONFIGURATION .................................................................................................................... 22 2.5 SAP LVM ENVIRONMENT .................................................................................................................. 24
3 INSTALLATION ...................................................................................................................................... 32
3.1 NAMING CONVENTIONS ...................................................................................................................... 33 3.2 INSTALL STORAGE INFRASTRUCTURE ................................................................................................. 34 3.3 INSTALL MANAGEMENT ENVIRONMENT ............................................................................................. 44 3.4 INSTALL MANAGED ENVIRONMENTS .................................................................................................. 52
4 INITIAL LVM CONFIGURATION ....................................................................................................... 71
4.1 ENGINE SETTINGS ............................................................................................................................... 71 4.2 STORAGE MANAGER CONFIGURATION ............................................................................................... 71 4.3 DISCOVERY OF RESOURCES (LPARS) ................................................................................................. 74 4.4 SERVICE CONFIGURATION/ DISCOVERY IN SAP LVM ........................................................................ 75 4.5 SETUP FOR THE ENVIRONMENT IN IBM SYSTEMS DIRECTOR .............................................................. 92 4.6 VIRTUALIZATION MANAGER CONFIGURATION IN SAP LVM ........................................................... 104 4.7 ADDITIONAL CONFIGURATION STEPS ............................................................................................... 106
5 DETAILED DESCRIPTION / EXECUTION OF USE CASES ......................................................... 109
5.1 LVM STANDARD EDITION USE CASES ............................................................................................. 109 5.2 SAP LVM ENTERPRISE EDITION USE CASES .................................................................................... 126
6 TROUBLESHOOTING HINTS AND TRICKS ................................................................................... 165
6.1 PHASE “PREPARE CLONE VOLUMES” ................................................................................................ 165 6.2 CLEANUP ACTIVITIES ON A RESOURCE LPAR ................................................................................... 165 6.3 CLEAN-UP ORPHANED VOLUMES IN THE IBM XIV ........................................................................... 166 6.4 DELETE THE SYSTEM IN SAP LVM .................................................................................................. 166
7 RESOURCES ........................................................................................................................................... 167
7.1 IBM DOCUMENTATION ..................................................................................................................... 167 7.2 SAP DOCUMENTATION AND SAP NOTES.......................................................................................... 167
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 4 of 169
List of figures
Figure 1 Logical connections of IBM and SAP components ..................................................... 9
Figure 2 Generic SAP NetWeaver LVM capabilities .............................................................. 11
Figure 3 SAP LVM Block Diagram (Source: SAP) ................................................................ 13
Figure 4 IBM Systems Director and Plug-Ins .......................................................................... 14
Figure 5 Software structure to integrate IBM IT layers ........................................................... 16
Figure 6 Generic SAP System Copy Workflow ....................................................................... 17
Figure 7 Architecture overview diagram ................................................................................. 23
Figure 8 redundant virtual fibre channel attachment................................................................ 28
Figure 9 Service configuration in SAP LVM with distributed file system layout ................... 34
Figure 10 Example configuration of an SAP system using GPFS ........................................... 39
Figure 11 Volume Group and Filesystem layout ..................................................................... 41
Figure 12 Volumes of SAP system CL6 .................................................................................. 43
Figure 13 SAP System CL6 in LVM and mount points .......................................................... 44
Figure 14 Overview of the FlashCopy Manager installation ................................................... 46
Figure 15 Sample LDAP Directory Information Tree for AIX user management .................. 49
Figure 16 Start IPSec ................................................................................................................ 55
Figure 17 Define the Resource as “Isolation Ready” in SAP LVM ........................................ 56
Figure 18 Configuration of host_profile .................................................................................. 59
Figure 19 Content of .conf file FCMCLI.conf ......................................................................... 59
Figure 20 Content of .conf file XCLI.conf ............................................................................... 60
Figure 21 LVM Engine Settings .............................................................................................. 71
Figure 22 Storage Manager Configuration entry panel ............................................................ 72
Figure 23 Storage Manager Configuration: Select Storage Manager type .............................. 72
Figure 24 Storage Manager Configuration: Define FlashCopy Manager settings ................... 73
Figure 25 Storage Manager Configuration: user and password ............................................... 73
Figure 26 Storage Manager Configuration Summary .............................................................. 74
Figure 27 Add resource – find Resources ................................................................................ 74
Figure 28 Add resource - assign pool ....................................................................................... 75
Figure 29 Add resource – define properties ............................................................................. 75
Figure 30 Add new SAP services ............................................................................................. 76
Figure 31 Expert Mode: Detect new services using host agent and instance agent ................. 77
Figure 32 Add Services View .................................................................................................. 77
Figure 33 Selection of the Resource Pool ................................................................................ 78
Figure 34 Enter SAP System Description ................................................................................ 78
Figure 35 Add Services sub-menu ........................................................................................... 78
Figure 36 Summary view for the new system .......................................................................... 79
Figure 37 SAP System configuration ....................................................................................... 79
Figure 38 SAP System configuration: general settings ............................................................ 80
Figure 39 SAP System configuration. Overall system details ................................................. 80
Figure 40 SAP system configuration: ACM ............................................................................ 81
Figure 41 SAP System configuration: Network fencing defaults ............................................ 81
Figure 42 SAP System configuration: Add allowed communications ..................................... 82
Figure 43 SAP System configuration: Add allowed communications ..................................... 82
Figure 44 DB Configuration: Entry screen .............................................................................. 83
Figure 45 DB Configuration: Basic configuration ................................................................... 83
Figure 46 DB Configuration: Add resource details ................................................................. 84
Figure 47 SAP LVM Storage Definition Screen ...................................................................... 85
Figure 48 Create a list of the DB filesystems ........................................................................... 85
Figure 49 Create XML-File for mountconfig .......................................................................... 86
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 5 of 169
Figure 50 Paste Mountconfig in LVM storage config ............................................................. 86
Figure 51 Extract Mount Points ............................................................................................... 87
Figure 52 Final LVM Mount point configuration .................................................................... 87
Figure 53 Service Details Central Instance .............................................................................. 88
Figure 54 Basic Configuration for Central Instance ................................................................ 88
Figure 55 Service properties for Central Instance .................................................................... 89
Figure 56 Mount Config for Central Instance .......................................................................... 90
Figure 57 Create List of filesystems for central instance ......................................................... 90
Figure 58 Extract Mount Config for Central Instance ............................................................. 91
Figure 59 Transfer Mount Config to SAP LVM ...................................................................... 91
Figure 60 Apply Mount Config to SAP LVM ......................................................................... 92
Figure 61 HMC Discovery in IBM Systems Director .............................................................. 93
Figure 62 Request access to the HMC ..................................................................................... 93
Figure 63 HMC inventory collection ....................................................................................... 94
Figure 64 View HMC inventory .............................................................................................. 94
Figure 65 View VIO server inventory ...................................................................................... 95
Figure 66 View NIM server inventory ..................................................................................... 96
Figure 67 Remove NIM server ................................................................................................. 97
Figure 68 Discover NIM server ............................................................................................... 97
Figure 69 Collect Inventory of NIM server.............................................................................. 98
Figure 70 Installation of NIM subagent on the NIM server with VMControl ......................... 99
Figure 71 Export mksysb directory .......................................................................................... 99
Figure 72 Trigger of Virtual Appliance Deployment ............................................................. 101
Figure 73 Setup of IBM Systems Director Security – Authorize Groups .............................. 102
Figure 74 Setup of IBM Systems Director Security – Copy Predefined Roles ..................... 102
Figure 75 Setup of IBM Systems Director Security – Modify Copied Roles ........................ 103
Figure 76 Setup of IBM Systems Director Security – Define Resource Groups ................... 103
Figure 77 Setup of IBM Systems Director Security – Assign Roles to User ......................... 104
Figure 78 Setup of IBM Systems Director Security – Assign Groups and Roles to User ..... 104
Figure 79 Virtualization Manager Configuration ................................................................... 105
Figure 80 Virtualization Manager: IBM Systems Director VMControl ................................ 105
Figure 81 Virtualization Manager properties ......................................................................... 106
Figure 82 Virtualization manager summary ........................................................................... 106
Figure 83 Configure Network ................................................................................................ 107
Figure 84 Customizing SAPInst rename path ........................................................................ 108
Figure 85 IBM PowerVM based relocation ........................................................................... 111
Figure 86 SAP adaptive based relocation .............................................................................. 111
Figure 87 Prepare SAP instance ............................................................................................. 113
Figure 88 Start SAP system as mass operation ...................................................................... 114
Figure 89 Logs of the 'Start SAP system' operation ............................................................... 114
Figure 90 Relocation of SAP system as mass operation ........................................................ 115
Figure 91 Stop LPAR ............................................................................................................. 116
Figure 92 Dashboard in SAP LVM ........................................................................................ 117
Figure 93 Infrastructure Visualization in SAP LVM ............................................................. 118
Figure 94 Performance monitoring in SAP LVM .................................................................. 119
Figure 95 OS Provisioning – Template Selection .................................................................. 121
Figure 96 OS Provisioning – LPAR Name Definition ........................................................... 121
Figure 97 OS Provisioning – Target Server Selection ........................................................... 122
Figure 98 OS Provisioning – Storage Pool Selection ............................................................ 122
Figure 99 Provisioning – Definition of Provisioning Parameters .......................................... 123
Figure 100 Provisioning – Final Parameter Confirmation ..................................................... 125
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 6 of 169
Figure 101 OS Provisioning – Process Monitoring ............................................................... 125
Figure 102 SAP System Copy scenarios supported by SAP LVM (Source SAP) ................. 126
Figure 103 Enable the cloned System for SAP System Rename ........................................... 128
Figure 104 Invoke SAP System Clone ................................................................................... 130
Figure 105 SAP System Clone: Basic data for the target system .......................................... 130
Figure 106 SAP System Clone: Resource selection ............................................................... 131
Figure 107 SAP System Clone: Network definition .............................................................. 133
Figure 108 SAP System Clone: Storage Definitions ............................................................. 134
Figure 109 SAP System Clone: Network Fencing ................................................................. 135
Figure 110 SAP System Clone: Final parameter confirmation .............................................. 135
Figure 111 Define RFC destination for PCA execution ........................................................ 137
Figure 112 SAP System Copy – Basic data ........................................................................... 138
Figure 113 SAP System Copy - Resources ............................................................................ 138
Figure 114 SAP System Copy – Network settings ................................................................ 139
Figure 115 SAP System Copy – Instance Number ................................................................ 139
Figure 116 SAP System Copy – Storage Definition .............................................................. 140
Figure 117 SAP System Copy – User definitions .................................................................. 141
Figure 118 SAP System Copy – Rename SAP system ID ..................................................... 141
Figure 119 SAP System Copy – Network fencing ................................................................. 142
Figure 120 SAP System Copy - Select RFC destination for PCA ......................................... 142
Figure 121 SAP System Copy – Summary view ................................................................... 143
Figure 122 SAP System Refresh: Start system refresh .......................................................... 143
Figure 123 SAP System Refresh: Basic Data ........................................................................ 144
Figure 124 SAP System Refresh: Resource Data .................................................................. 144
Figure 125 SAP System Refresh: Host Names ...................................................................... 145
Figure 126 SAP System Refresh: Storage Settings ................................................................ 145
Figure 127 SAP System Refresh: Users ................................................................................. 146
Figure 128 SAP System Refresh: DB Schema ....................................................................... 146
Figure 129 SAP System Refresh: Network Isolation ............................................................. 147
Figure 130 SAP System Refresh: PCA settings ..................................................................... 147
Figure 131 SAP System Refresh: Parameter summary .......................................................... 148
Figure 132 Start SAP System Rename ................................................................................... 148
Figure 133 SAP system Rename: Basic Data ........................................................................ 149
Figure 134 SAP System Rename: Virtual Hostnames ........................................................... 150
Figure 135 SAP System Rename: Instance numbers ............................................................. 150
Figure 136 SAP System Rename: Mount points .................................................................... 151
Figure 137 SAP System Rename: Specify users for target system ........................................ 151
Figure 138 SAP System Rename: Specify database parameters ............................................ 152
Figure 139 SAP System Rename: Configure network isolation ............................................ 153
Figure 140 SAP System Rename: Disable Post Copy Automation (I) .................................. 153
Figure 141 SAP System Rename: Disable Post Copy Automation (II) ................................. 154
Figure 142 SAP System Rename: Summary Screen .............................................................. 154
Figure 143 Provider Implementation Definition (I) ............................................................... 157
Figure 144 Provider Implementation Definition (II) .............................................................. 157
Figure 145 Provider Implementation Definition (III) ............................................................ 158
Figure 146 Provider Implementation Definition (IV) ............................................................ 158
Figure 147 Custom Hook Definition (I) ................................................................................. 159
Figure 148 Custom Hook Definition (II) ............................................................................... 159
Figure 149 Custom Hook Definition (III) .............................................................................. 160
Figure 150 System State in Operations view ......................................................................... 161
Figure 151 Mass operation „Stop and Unprepare” started ..................................................... 161
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 7 of 169
Figure 152 Monitoring view .................................................................................................. 162
Figure 153 Mass Stop completed ........................................................................................... 162
Figure 154 Destroy System .................................................................................................... 163
Figure 155 Destroy operation – delete storage volumes ........................................................ 163
Figure 156 Destroy operation – delete host names ................................................................ 164
Figure 157 Destroy Operation – summary screen .................................................................. 164
Figure 158 Remove System in SAP LVM ............................................................................. 166
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 8 of 169
List of tables
Table 1 Functionalities in SAP LVM V1.0 .............................................................................. 12
Table 2 Supported SAP releases for the SAP LVM scenario .................................................. 13
Table 3 Overview of the Scenarios .......................................................................................... 20
Table 4 Project Phases and estimated time .............................................................................. 22
Table 5 Software Stack ............................................................................................................ 23
Table 6 Operational Model ...................................................................................................... 32
Table 7 Sample directory structure for a distributed file system implementation ................... 33
Table 8 Entries in /etc/services ................................................................................................. 57
Table 9 Local Filesystem layout .............................................................................................. 57
Table 10 XIVGUI Install directory .......................................................................................... 60
Table 11 Minimum filesystem layout (DB2 UDB) .................................................................. 65
Table 12 Extended filesystem layout (DB2 UDB) ................................................................... 66
Table 13 Required setup procedures ...................................................................................... 129
Table 14 Examples for IP addresses ...................................................................................... 131
Table 15 List of SAP Notes .................................................................................................... 168
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 9 of 169
1 Introduction
1.1 Background and Business Context
In the era of worldwide business relations and processes, the flexible and non-disruptive
operation of business applications and their underlying middleware and hardware becomes an
essential factor for success. Even unforeseen fluctuations in transaction volumes and changing
business processes must be reflected immediately by the supporting system capacities. On the
other hand, planned interventions such as hardware or application maintenance and testing
must not disrupt SAP Business Suite availability to worldwide partners and consumers. In
order to accelerate innovation and fast adoption of new business processes, non-production
stages of SAP systems need to be deployed on demand, without latency for procurement and
installation.
Consequently, manually operating and administrating a growing number of individual
systems or landscape components is no longer an option. Here, the concepts of cloud
computing not only promise to save costs but also to increase flexibility, elasticity, and
automation of system operations to efficiently serve the needs of the business.
This paper describes how to design and implement an on-premise cloud environment by
combining IBM platform technologies, namely IBM Systems Director and Tivoli Flash Copy
Manager, with the SAP NetWeaver® Landscape Virtualization Management (SAP LVM)
environment. Finally the paper shows, how all the use-cases, including cloning, copying, and
refreshing of SAP systems, are managed via the SAP LVM Administrator interface.
1.2 Scope of this Cookbook
The technical integration of SAP LVM and IBM Systems and Systems Management Software
components is basis for the solution. Figure 1 illustrates all the major components.
The integration includes IBM Power Systems in combination with PowerVM and
VMControl, selected IBM Storage Systems respectively SAN Volume Controller in
combination with IBM Tivoli Storage FlashCopy Manager, and finally the IBM Systems
Director as common base for plug-ins and interfaces.
Figure 1 Logical connections of IBM and SAP components
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 10 of 169
This cookbook describes the specifics and installation steps for implementing the SAP
NetWeaver Landscape Virtualization Management software V1.0 on an IBM PowerVM /
AIX based infrastructure. It also provides how-to‟s for using the resulting administration
environment for launching and monitoring the most popular SAP “cloud use cases”.
The installation sequence has been tested in a Proof-of-Concept including
SAP Business Suite 7
DB2 UDB LUW
IBM POWER
AIX 6 and AIX 7
IBM Systems Director,
IBM Tivoli FlashCopy Manager
SAN-attached IBM XIV storage subsystem.
Because of the technical affinity of FlexSystem Manager (FSM) and IBM Systems Director,
we assume that the content applies to a PureFlex system containing virtualized POWER
nodes, too.
At the time of writing SAP LVM V1.0 has been made GA (June 2012) and is available with
SP3. Relevant to IBM infrastructure, this new support package added support for SAN
Volume Controller and V7000 in addition to the already supported IBM XIV Storage.
The fact that we describe a POWER based implementation does in no way mean other
hardware and virtualization platforms (including IBM System x) do not offer a comparable
degree of SAP LVM integration. However, the described scenario and its functionality are
most advanced and thus can provide an important differentiator for the IBM POWER and
storage platforms.
The cookbook is not intended to replace a solid technical knowledge of any of the involved
layers and is no substitute for existing literature by IBM and SAP. We will point to applicable
literature where needed.
1.3 SAP NetWeaver Landscape Virtualization Management software (SAP LVM)
SAP NetWeaver® Landscape Virtualization Management software lets customers monitor
and manage entire SAP software landscapes and their associated heterogeneous IT
infrastructures. SAP LVM provides a single point of control, which allows visualizing,
monitoring, and managing data center tasks. The managing aspect includes resource
allocation and de-allocation on IT, but also on SAP Business Suite level.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 11 of 169
Figure 2 Generic SAP NetWeaver LVM capabilities
SAP LVM is an evolution of the previously established SAP Adaptive Computing Controller
(ACC 7.3). As such SAP LVM inherits functions like “Landscape-wide Visualization and
Monitoring” and “Automated Capacity Management” from the former ACC.
In addition, SAP LVM provides new and more powerful functions like “SAP System
Cloning” and “SAP System Copy / Refresh”, which are frequently performed operations at
SAP sites. They combine numerous steps on infrastructure and SAP application level
including Post Copy Automation (PCA) sequences. SAP PCA is only available with the SAP
LVM “Enterprise” Edition.
SAP LVM Standard is available for download from SAP Service Market Place as a default
component of any NetWeaver 7.3ff stack with no additional fee; the SAP LVM Enterprise
Edition is associated with additional SAP license fees.
SAP LVM Version 1.0 provides all the administrative functionalities as listed in Table 1.
Functions that require SAP LVM Enterprise Edition are highlighted in blue and italic font.
The Standard Edition establishes the basic framework and includes administrative functions
that have been available with SAP Adaptive Compute Controller (SAP ACC) in the past.
These are related to landscape monitoring and automated startup, stop, and relocate of SAP
instances.
The Enterprise Edition extends these core capabilities with the advanced functions including/
involving SAP Post Copy Automation sequences. One can mostly automate complex and
frequently applied SAP system management tasks. This includes generation of isolated clones
of a SAP Production system for testing purposes, but also replication of a Production system
into a QA- or Pre-Production-System that stay fully integrated within the existing transport
stages. For latter purpose, business data and system settings need to be replicated, while at
any time avoiding conflicts with the online production environment.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 12 of 169
SAP LVM 1.0 Functionalities, GA June 2012
End-to-End SAP System Clone / Copy / Refresh for ABAP
End-to-End SAP System Clone / Copy / Refresh for JAVA
Post-Copy Automation (PCA) for ABAP and JAVA
Automatic Capacity Management (ACM)
Application Server Installation / Un-installation
Mass Operations (start / stop / relocate)
User Configurable Dashboards
Reporting
Landscape Visualization
Custom Operations and Custom Hooks (Extensibility)
Custom Services (Management of Non-SAP Components)
Table 1 Functionalities in SAP LVM V1.0
Although SAP LVM today supports a high degree of automation for individual systems, it
does not reflect interdependencies among logically linked systems on an application level.
Future versions should improve the way in which management of application topologies can
be automated also. SAP LVM will also become aware of complementary systems attached to
a backend system - like TREX, liveCache etc. - and consider those systems when performing
landscape change actions.
The next section will provide a closer look into the involved components, overall solution
architecture, and details about the technical integration of SAP LVM with IBM cloud
enabling technology.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 13 of 169
2 Overview and Planning
2.1 Architecture overview
This section provides an introduction where and how individual functions are implemented.
2.1.1 SAP NetWeaver Landscape Virtualization Management
2.1.1.1 Technical Implementation
SAP LVM is an add-on to SAP NetWeaver 7.30/7.31 Application Server Java. The SAP
LVM software is available for any platform that supports the SAP NetWeaver releases
mentioned above.
The SAP systems managed by SAP LVM can run far older SAP releases. Table 2 lists the
supported releases for the managed SAP systems in a SAP LVM scenario. SAP note 1527538
describes the details on restrictions that may apply to some of the older SAP releases.
Supported Managed SAP systems (See SAP Note 1527538)
NetWeaver 7.00ff Web AS 6.40 Web AS 6.20 Web AS 6.10 R/3 4.6D R/3 4.6C
Table 2 Supported SAP releases for the SAP LVM scenario
Figure 3 illustrates the communication between all the different components. SAP LVM
software (both the SAP NetWeaver AS Java and the SAP LVM Add-on) is installed on a
single system designated as “Management Server. SAP LVM communicates with the
managed nodes via the SAP Host Agent (a process running on each managed node) for
landscape-wide monitoring and execution of tasks on the systems. Additionally there are
direct interfaces (black boxes in the diagram) to the IBM PowerVM virtualization layer and
the IBM Storage Layer. The IBM solution described here uses of all those paths.
Figure 3 SAP LVM Block Diagram (Source: SAP)
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 14 of 169
SAP LVM provides a set of APIs as interface. In order to support specific hardware,
virtualization, OS and DB layers, each SAP partner needs to supply a set of libraries that
translate the generic SAP LVM operations to platform specific commands and executes them
as integrated solution on the appropriate component. SAP Note 1527538 describes all the
supported platforms for SAP LVM: Not all platform providers have implemented specific
libraries for all of their platforms, or partly they support just a subset of the functionality.
Thus, the supported use case for the SAP LVM solution may vary by platform.
2.1.2 IBM Systems Management Components
IBM Systems Director® is the platform management backbone, providing all the building
blocks for integrated services management. Systems Director provides the IT administrator
with the features:
Unified management of physical and virtual resources for IBM servers, storage, and
networks.
Automated data center operations by implementing cloud-ready virtual infrastructures.
Figure 4 IBM Systems Director and Plug-Ins
Key value of IBM Systems Director is its ability to work in diverse IT environments, so it
reduces the number of required management tools and interfaces. This is one reason, why it
was selected as programmatic SAP LVM counterpart: Instead of creating and maintaining
multiple interfaces for several IBM platforms, one generic interface to SAP LVM is
sufficient.
The components highlighted by a red frame are the base for this document and were explicitly
tested in the environment. This does not mean that other combinations and stacks are not
supported by SAP or IBM. For example, IBM SAN Volume Controller (SVC) has become
available with the GA of SAP LVM and can support a large variety of IBM and non-IBM
“backend storage” subsystems attached.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 15 of 169
In case customers plan to implement the documented scenario on top of an IBM PureFlex
platform with POWER nodes and V7000 storage, this document should apply too. The Flex
System Manager (FSM) then substitutes IBM Systems Director. In respect to functions and
technical interfaces, both management layers are consistent.
Optional plug-ins can extend IBM Systems Director. The SAP LVM integration exploits:
IBM® Systems Director VMControl™ is the multi-platform virtualization management
solution. This application provides the following features that are also relevant in any SAP
landscape:
Create and manage virtual machines
Relocate virtual machines
Import, edit, create and delete virtual images
Deploy virtual images
The VMControl plug-in is used to actually perform the LPAR and AIX related steps along the
overall processing sequence.
IBM® Systems Director Storage Control extends the management of systems to include the
storage layer. It is based on IBM Tivoli Storage Productivity and enables functions as:
Extends storage management of IBM Systems Director to cover most IBM storage
systems
Storage device discovery and coverage in integrated physical and logical topology views
Show relationships between storage and server resources
Ability to configure logical and physical configuration
Ability to view controller and volume status and to set notification alerts
Integration with IBM Systems Director VMControl storage provisioning for image
creation, deployment, and cloning
Storage Control is used in the end-to-end provisioning sequence to allow “bare-metal”
LPAR creation and AIX installation: During the provisioning process, Storage Control
takes care that the storage for the OS is provisioned and attached to the LPAR.
IBM Tivoli® Storage FlashCopy® Manager. This software provides fast application-aware
backups and restores by exploiting snapshot technologies in IBM storage systems.
In the context of SAP landscape management, IBM Tivoli FlashCopy Manager is used to
create instantaneously consistent copies of online SAP systems. Instead of using these copies
as source for backups, SAP LVM uses the flashes as source for further processing system
replica by subsequent post-processing steps.
2.2 Combining the IBM and SAP Layers
2.2.1 Integration by Platform Libraries
Figure 3shows how the SAP LVM architecture is implemented for IBM POWER and IBM
storage platforms. The integration consists of a set of libraries, which transfer generic SAP
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 16 of 169
LVM commands into the specific syntax for the IBM environment. IBM provides these
libraries as part of the SAP LVM software stack.
Different communications paths exist between the managing server and the managed nodes
One communication path is via the SAP Host Agent
The SAP Host agent can execute OS commands on each managed SAP node. SAP
LVM communicates with this process, which calls the vendor libraries in order to map
the LVM commands to an OS, DB, or storage specific language and syntax.
The second communication path uses the SAP LVM virtualization adapters.
The managing system interfaces directly with the VMControl plug-in of IBM Systems
Director to control PowerVM and OS layers,
Another communication path uses the SAP LVM storage adapter (IBM Tivoli Storage
FlashCopy Manager). For non-disruptive copies of running SAP instances, SAP LVM
interacts with the Tivoli Storage Flash Copy Manager, which interfaces to the storage
system for the cloning, and handles then handles all storage activities required on the
managed nodes also. This implementation guarantees a consistent replication of SAP
instances by synchronizing the required steps among the involved infrastructure
layers (storage, AIX, DB).
For any supported IBM component own libraries need to be created: The introduction of IBM
Systems Director and SAN Volume Controller as common interface and abstraction layers
towards various underlying virtualization and hardware layers makes this step easier.
Figure 5 Software structure to integrate IBM IT layers
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 17 of 169
2.2.2 Example Workflow “SAP System Copy”
In this section, a walk-through through a sample “SAP System Copy” use case will
demonstrate how the integration works together. The workflow is composed of two phases
(See Figure 6):
System Cloning Phase
The system cloning phase begins by creating host name mappings on a name server
and is finished when the clones SAP systems is ready to be started in an isolated
network segment. These steps are the same for the system copy and system cloning
scenarios. At the end of this phase, a 1:1 clone of the source SAP system exists which
per default cannot be accessed from other system systems from outside that fenced
segment.
System Copy Phase
After the system-cloning phase is completed, SAP LVM triggers additional automated
tasks that make up the system copy phase. This includes renaming the SAP system
clone. The Post Copy Automation (PCA-) sequence adjusts the complete SAP
configuration so that finally a unique SAP system (SID) is established.
Figure 6 Generic SAP System Copy Workflow
For a detailed description of the workflow, the following landscape example is used:
The source system is an SAP WebAS (Application Server) ABAP system
It is deployed in an AIX LPAR.
The SAP system is configured with virtual IP addresses.
The operating system, SAP system, and database are installed on separate AIX logical
volume groups.
The physical disks for these volume groups are allocated on SAN storage systems.
The source system is configured in SAP NetWeaver Landscape Virtualization
Management.
The administrator navigates to the “Provision view” in the SAP LVM console and selects the
source SAP system (All the detailed steps are shown later in chapter 5.2). After pressing the
Copy button, SAP LVM guides the administrator through dialogue steps defining the end-to-
end process. SAP LVM knows all the configuration settings of the registered source system;
however, the administrator has to specify configuration parameters for the target system.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 18 of 169
Reasonable values for some parameters are calculated automatically and are pre-set as default
during the configuration dialogue. Following parameters can be changed/ overridden (this is a
non-exhaustive list):
Target resource: SAP NetWeaver Landscape Virtualization Management
provides the option to choose an already existing AIX LPAR or to create a new
AIX partition as target for the system. (For the latter the virtualization manager
configured in SAP NetWeaver Landscape Virtualization Management needs to
be set-up for creating and configuring such an operating system partition).
Host names: The copied SAP system is given a new virtual IP address and
host name. The host name can be configured manually. Alternatively, SAP
NetWeaver Landscape Virtualization Management is able to create new
hostnames and to update the domain name server automatically.
Storage volumes: SAP NetWeaver Landscape Virtualization Management
uses the registered storage adapter to gather the data about the volumes
attached to the source system. The information is gathered directly from the
storage management system. The administrator can specify target volume
names and mount points.
Database consistency: SAP NetWeaver Landscape Virtualization
Management asks whether the SAP source system is to be stopped during the
cloning process or whether the system should stay up and running (for
example, in case of a production system). With the online mode, the
administrator decides whether SAP NetWeaver Landscape Virtualization
Management is responsible for database consistency. Otherwise, there is no
guarantee that the cloned database can be recovered and brought online. When
selecting database consistency, SAP NetWeaver Landscape Virtualization
Management switches the source database into online backup mode (Oracle) or
into suspend I/O mode (DB2). This prevents write operations to the storage
volumes during the actual cloning step.
Target isolation: On the target system network, network fencing has to be
established to ensure that the cloned system can be started without interfering
with other systems in the data center. This is achieved by using the IPsec
feature of AIX. SAP NetWeaver Landscape Virtualization Management allows
the definition of permitted outgoing network connections. All other outbound
connections are blocked while incoming connections are permitted.
SAP copy parameters: SAP NetWeaver Landscape Virtualization
Management queries information about the copied SAP system, such as the
new SAP system identifier (SAP SID) and master password.
SAP copy users: SAP NetWeaver Landscape Virtualization Management
allows new users to be created and defined for the copied SAP system on a
central user repository.
SAP post copy automation: SAP NetWeaver Landscape Virtualization
Management allows the selection of task lists and task list variants to be used
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 19 of 169
for post copy automation. This is possible for multiple clients in the SAP
system (such as basis tasks in „000‟ and BDLS in „100‟).
After all the configuration parameters have been defined, the administrator reviews the input
and then starts the SAP system copy workflow.
There is a narrated ScreenCam video available at TechDocs, in case you want get a real
impression about the look and feel of the SAP LVM administrator console.
2.2.3 Supported Use-Cases and involved components
The use-cases for
Landscape-wide Overview and System monitoring
VM activate, deactivate and relocate
Provision new Resources (AIX LPARs)
are included in the standard edition of SAP LVM and will leverage one of the virtualization
adapters during execution. These use-cases are not dependent on special storage, as long as
the storage satisfies the requirements of the virtual environment. IBM Tivoli Storage
FlashCopy is not required for execution of these use-cases. Enhanced functionalities (e.g.
customized dashboards, generation of reports) will require the enterprise edition.
The use-cases for
SAP Storage volumes: Attach, Detach, Relocate
SAP Clone/ Copy/ Refresh
have some dependencies on the storage infrastructure.
The implementation of Attach/ Detach/ Relocate depends on the actual storage type: For NAS
and GPFS the functionality is included in the platform library. For SAN-based IBM storage,
the Storage Adapter is invoked during execution of the use-cases.
Table 3 summarizes the different use case families in SAP LVM and illustrates which Edition
for SAP LVM is required, which SAP LVM adapters needs to be invoked, and which
boundary conditions need to be fulfilled for the execution.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 20 of 169
Use Case Landscape-
wide Overview
and System monitoring
VM activate, deactivate
and relocate
Provision new Resources
(AIX LPARs)
SAP Storage volumes: Attach, Detach, relocate
SAP Clone/ Copy/
Refresh
SAP LVM Adapter
Virtualization Adapter
Storage Adapter or Platform Library
Storage Adapter
SAP Landscape Virtualization Management Edition
Standard Edition
(advanced
scenarios require Enterprise
Edition)
Standard Edition
Standard Edition
Standard Edition (NFS and GPFS), Enterprise Edition for SAN based
storage.
Enterprise Edition
Virtu
aliz
ation
Mana
ge
ment
IBM Power Systems with IBM Systems Director
Yes, requires VMControl Express Edition
Yes, requires VMControl Express Edition
Yes, requires VMControl Express Edition, requires Storage Control
not required not required IBM PureSystems with IBM Flex System Manager
HMC Yes Yes No
Sto
rage
Mana
ge
ment
IBM Tivoli Storage FlashCopy Manager
not required not required not required not required required
Supported Storage
Any Any Any
NAS (NFS), GPFS, or IBM
XIV, IBM SVC, IBM
V7000
IBM XIV, IBM SVC, IBM
V7000
Table 3 Overview of the Scenarios
2.3 Planning Project Resources and Timing
Table 4 summarizes the activities in order to implement a SAP LVM managed POWER
landscape comparable to our PoC setup described in the following section 2.4. The estimated
duration per phase provides a rough guideline only. In detail, the times are dependent on the
complexity of the SAP landscape (IT and SAP systems) and its operation as well as on the
availability of all the required skills for all involved disciplines.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 21 of 169
Project Phase Included Tasks / Accomplishments Estimated time
Planning workshop, Verification of landscape prerequisites in client environment
Network and disk design requirements, SW prerequisites
Validate that customer has obtained the necessary SAP and AIX software, validate that dedicated IBM hardware infrastructure is available
SAP Landscape needs to support virtual host names, and a proper range of IP addresses needs to be available
2-3 Day workshop with customer
Plan and Prepare setup of storage infrastructure
Storage LUNs
Network, SAN zoning
Eventually map SVC to existing storage
Plan and create filesystems
3-4 days working with customer
Installation and configuration steps for the management environment (IBM Systems Director, SAP NetWeaver Landscape Virtualization Management)
Define LPARs on the managed systems, and install AIX OS on the LPARs
Set-up virtualization environment o Install and configure IBM Systems
Director including VMControl and Storage Control
o Integrate NIM server into System Director VMControl environment
o Set-up Storage Control for all storage entities (SAN, Storage System)
Install and configure storage management software (FlashCopy Manager, Storage CLI) on the management system
Plan and configure shared infrastructure services
o Install and configure LDAP (if not to be integrated with existing LDAP environment
o Plan for DNS
Install SAP LVM system
Two weeks working with customer
Installation and configuration steps of managed environment incl. basic testing Installation from scratch for one SAP System
Define AIX “golden image” content
Install first AIX LPAR with golden image
Install and configure storage management software (FlashCopy Manager, SVC CLI)
Install first “source” SAP system
LVM base configuration (Virtualization Manager, Storage Manager configuration, discover resource LPARs and SAP systems, users)
Capture image in Systems Director VMControl for further deployment
Additional LVM use case specific configuration (storage, post processing, steps per use case) Note: configuration of PCA post processing requires skilled SAP specialist with both SAP Basis and Application/Business Process knowledge)
Two weeks with customer
Installation and configuration steps of managed environment incl. basic testing Migrate/ Customize one already existing
Adapt existing SAP system to virtualised environment
Eventually Upgrade SAP Hostagent
Install and configure agents and storage management
Install and configure storage management
At least 2 weeks with customer
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 22 of 169
Project Phase Included Tasks / Accomplishments Estimated time
SAP system
software (FlashCopy Manager, SVC CLI)
Discovery and setup of LPARs and SAP systems in IBM Systems Director and SAP LVM
Additional LVM use case specific configuration (storage, post processing, steps per use case) Note: configuration of PCA post processing requires skilled SAP specialist with both SAP Basis and Application/Business Process knowledge)
Verification Steps using the customer’s SAP system
Perform functional verification
Execute and document each use case o Start / Stop / Relocate of LPARs and SAP
systems o Clone SAP Systems (target LPAR available) o Copy SAP Systems (target LPAR available) o Refresh SAP system o Clone SAP Systems (create target LPAR
from scratch) o Copy SAP Systems (create target LPAR
from scratch)
At least 2 weeks working with customer
Acceptance Tests
Support customer in LVM acceptance testing from basis or application level
At least 1 week
Document Solution or Handover
at least 2 days for handover, documentation at least 1 week
Table 4 Project Phases and estimated time
2.4 Tested configuration
The installation hints in this document are based on a PoC test installation in the ISICC demo
landscape using IBM Power Systems and IBM XIV Storage Systems.
Figure 7 illustrates the environment in the PoC.
In the management environment:
Virtualization Manager is IBM Systems Director 6.3.1 which is installed on one
LPAR. Plug-in VMControl 2.4.1 is deployed to that node also. The Storage Control
plug-in was not part of the PoC environment: All the “end-to-end” deployment use-
cases for provisioning of a new resource LPAR (as shown in section 5.1.3) were
executed within a different environment.
SAP NetWeaver Landscape Virtualization Management software V1.0 SP3 is
deployed on top of a SAP NetWeaver Java AS 7.3 engine in a second LPAR
Additional infrastructure servers like DNS, LDAP server, and NIM server are
available and configured.
In the managed environment
at least one SAP source system is existing
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 23 of 169
all resources (AIX LPARs) for both the SAP source system(s) and SAP target
system(s) are registered within SAP LVM. The resources are attached via virtual fibre
channel to an IBM XIV Storage system.
On each resource LPARs the software components
SAP Host Agent package
IBM Tivoli Storage FlashCopy Manager
IBM XIV XCLI
are installed and configured.
Systems DirectorAIX
NIM ServerAIX
Management Environment
IBM XIV
SVC / Storwize
SAP LVMAIX
LDAP
IBM Tivoli Directory Server
Source
Target
VIOS
vFC (NPIV)
IBM Systems Director 6.3.1
VM Control 2.4.1
DNS
SAP NetWeaver Landscape
Virtualization Management V1.0 SP3
SAP NetWeaver Java AS 7.3
SAP SourceAIXSAP Source
AIXSAP SourceAIXSAP Source
AIXSAP SourceAIXSAP Source
AIX
SAP Component - SAP NetWeaver
Managed Environment
VIOS
vFC (NPIV)
SAP TargetAIX
• Storage connected
via vFC (NPIV)
• Compliant Storage
Layout
IBM Tivoli FlashCopy Manager 3.1.0.1
Storage Manager native Interfacee.g. IBM XIV XCLI 3.1 build 18
IBM Tivoli FlashCopy Manager 3.1.0.1
Storage Manager native Interfacee.g. IBM XIV XCLI 3.1 build 18
IBM Tivoli FlashCopy Manager 3.1.0.1
Storage Manager native Interfacee.g. IBM XIV XCLI 3.1 build 18
Figure 7 Architecture overview diagram
Table 5 reflects the detailed software stack tested in the scenario:
Software Version
SAP Landscape Virtualization Management Software
1.0 SP3
IBM Systems Director / VMControl 6.3.1 / 2.4.1
IBM Script Package
SAP Host Agent 7.20 PatchLevel 103
SAPINST (Rename)
abi version : 722 make variant: 720_REL
build: 1287791 compile time: Dec 19 2011 22:27:22
AIX 6.1 TL7 SP2 AIX 7.1 TL1 SP3
6100-07-02-1150 7100-01-03-1207
IBM Tivoli Storage FlashCopy Manager 3.1.0.1
IBM XIV 10.2.4.a
XIVGUI XCLI 3.1 build 18
DB2 UDB DB2 v9.7.0.5 Special Build 28492 (FP5SAP)
Table 5 Software Stack
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 24 of 169
2.5 SAP LVM Environment
2.5.1 Storage Virtualization
A SAP LVM infrastructure requires a centralized storage solution. The storage infrastructure
in the ISICC sample environment is based on IBM System Storage™ solutions using a
Storage Area Network (SAN).
IBM System Storage virtualization products achieve the abstraction from physical volumes of
data storage to a logical level. It addresses the increasing complexity of managing storage,
while reducing the associated costs. Its main purpose is the full exploitation of the benefits
promised by a SAN. Virtualization enables data sharing, ensuring higher availability,
providing disaster tolerance, improving performance, allowing for consolidation of resources,
providing policy-based automation, and much more besides, which do not automatically result
from the implementation of today‟s SAN hardware components. Storage virtualization is
possible on several levels of the storage network components, meaning that it is not limited to
the disk subsystem. Virtualization separates the representation of storage to the operating
system and its users, from the actual physical components.
Storage virtualization accumulates the storage into storage pools, which are independent of
the actual layout of the storage (that is, the overall file system structure). Because of this
independence, new disk systems can be added to a storage network, and data migrated to
them, without causing disruption to applications. Since the storage is no longer controlled by
individual servers, it can be used by any server as needed. In addition, it can allow capacity to
be added or removed on demand without affecting the application servers. Storage
virtualization will simplify storage management, which has been an escalating expense in the
traditional SAN environment.
IBM SAN Volume Controller (SVC), IBM Storwize V7000, or IBM XIV System Storage is
not mandatory for all the use-cases in an adaptive computing environment, but they help in
the overall management of the storage landscape. For the SAP System Copy Use-Cases
however the Storage Adapter and Storage Library require one of those systems.
2.5.1.1 SAN Volume Controller System Storage
The IBM (SVC) is an in-band, block-based virtualization product that minimizes the
dependency on unique hardware and software, decoupling the storage functions expected in a
SAN environment from the storage subsystems and managing storage resources. SVC
combines software and hardware into a comprehensive, modular appliance that uses
symmetric virtualization.
Symmetric virtualization is achieved by creating a pool of managed disks (MDisks) from the
attached backend storage systems. Those storage systems are then mapped to a set of volumes
for use by attached host systems. System administrators can view and access a common pool
of storage on the storage area network (SAN). This functionality helps administrators to use
storage resources more efficiently and provides a common base for advanced functions.
Key feature exploited for the SAP System Copy scenarios is the ability to create and manage
FlashCopies of existing (SAP application) volumes. IBM FlashCopy is a part of the SVC
Copy Services and provides a point-in-time copy function. This includes thin-provisioned
FlashCopy to make multiple targets affordable.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 25 of 169
2.5.1.2 IBM Storwize V7000 Storage System
The IBM Storwize V7000 is a storage server with internal disks, providing the same storage
virtualization functions and features like SAN Volume Controller.
2.5.1.3 IBM XIV System Storage
The IBM® XIV® Storage System is a high-end disk storage series designed to address
storage challenges across the application spectrum, including for virtualization, email,
database, and analytics and data protection solutions. The IBM XIV Storage System is a fully
virtualized system designed to eliminate the need for performance tuning and numerous other
storage management activities. IBM XIV provides consistent, hotspot-free enterprise
performance and exceptional ease of use. As virtualized storage that meshes tightly with
hypervisors, XIV offers optimal agility for cloud and virtualized environments.
Key feature exploited for the SAP System Copy scenarios is the ability to create and manage
snapshots of existing (SAP application) volumes. The IBM XIV Storage System has taken
this concept one step further, offering a completely innovative approach to snapshot creation
and management. The XIV system offers clear advantages, including the following:
Unlimited number of snapshots in the system.
Snapshot creation in virtually zero time, regardless of the size of replicated volumes.
Unaffected performance levels in a system that supports snapshots, regardless of the
number of snapshots currently defined in the system.
2.5.1.4 General Parallel File System (GPFS)
Exploiting GPFS is the easiest and most convenient way to implement the „classic“ adaptive
scenarios for relocating SAP services between different resources.
IBM's General Parallel File System (GPFS) provides file system services to parallel and serial
applications. GPFS allows parallel applications simultaneous access to the same files, or
different files, from any node, which has the GPFS file system mounted, while managing a
high level of control over all file system operations. GPFS is particularly appropriate in an
environment where the aggregate peak need for data bandwidth exceeds the capability of a
distributed file system server.
GPFS allows users shared file access within a single GPFS cluster and across multiple GPFS
clusters. A GPFS cluster consists of:
AIX nodes, a node may be:
o An individual operating system image on a single computer within a cluster.
o A system partition containing an operating system.
Network shared disks (NSDs) created and maintained by the NSD component of
GPFS
A shared network for GPFS communications allowing a single network view of the
configuration. A single network is used for GPFS communication, including the NSD
communication.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 26 of 169
All disks utilized by GPFS must first be given a globally accessible NSD name. On AIX
server running GPFS, an existing virtual shared disk or physical disk may be given an NSD
name.
2.5.2 SAP NetWeaver Landscape Virtualisation Management
All the base considerations are described in the SAP Notes Note 1527538 - SAP NetWeaver
Landscape Virtualization Management 1.0 and Note 1644520 - Integration of IBM
Components into SAP LVM
SAP LVM distinguishes in its configuration between:
Services
A service is a whole or a part of a SAP System (database service, central instance
service, application server service). The SAP LVM use-cases typically are executed on
the services.
and
Resources
A resource describes an operating system installed on a physical or virtual server. SAP
services may be relocated between resources, or a resource may be the target for a new
SAP system created via SAP system copy or cloning. A new resource may be
deployed (via SAP LVM and IBM Systems Director) so that it can be used as a target
for a relocate operation or a SAP system copy use-case. SAP LVM monitors resources
through SAP Host Agents running on the operating systems.
2.5.2.1 SAP LVM Virtualization Management
To extend resource monitoring data available via SAP Host Agents, to provision new
resources, or to perform load balancing by starting, stopping and relocating the resources,
SAP LVM interacts with the virtualization managers:
SAP LVM virtualization managers (adapters) for
IBM Systems Director/ VMControl (respectively IBM Flex Systems Manager)
or
IBM Hardware Management Console for IBM Power
Both allow
Monitoring of IBM Power physical servers and logical partitions (LPARs)
Management of LPARs (Power on/ Power off)
Relocation of LPARs using Live Partition Mobility
Additionally the “IBM Systems Director/ VMControl” adapter provides provisioning
capabilities for both AIX OS and Storage.
See SAP Note 1644520 - Integration of IBM Components into SAP LVM for detailed
description of supported configurations and features.
2.5.2.2 SAP LVM Storage Management
For cloning, copying and refreshing of SAP systems and for preparing, un-preparing and
relocation of SAP services on / between resources SAP LVM interact with the storage
manager:
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 27 of 169
SAP LVM storage manager (adapter) for IBM Tivoli Storage FlashCopy Manager
(FCM)
This adapter supports invocation of FlashCopies/ SnapShots within the storage system to
provide a fast, efficient, and application-aware cloning of the source system within the
storage. At the time of writing this document (July 2012) this adapter supports following
storage system types:
IBM XIV Storage System
IBM Storwize V7000 storage system
IBM SAN Volume Controller (incl. any supported backend storage system)
In addition, the adapter provides the required functionality for the management of the storage
to attach/ detach volumes and filesystems to/ from the LPARs during the prepare and
unprepared activities.
See SAP Note 1644520 - Integration of IBM Components into SAP LVM for detailed
description of supported configurations and features.
In this PoC we used the following configuration for SAP System Copy use cases:
IBM XIV Storage System, Version 10.2 with XIV CLI 3.0
Storage volumes in the storage system connected to the IBM Power system LPARs
either by dedicated physical fibre channel adapters or by virtual fibre channel adapters
(N-Port ID virtualization, NPIV).
The volumes of the source system are connected to the source LPAR only. The source
volumes must not be connected to the target LPAR.
The SAP Host Agent (located in /usr/sap/hostctrl) is installed on a storage volume that
is not part of any cloning scenario: We recommend creating a /usr/sap filesystem in
“rootvg”.
The target host LPAR is defined as host in the storage system. We recommend
deploying the AIX operating system of the target LPAR to volumes within the same
storage system: then this requirement is fulfilled automatically.
2.5.2.2.1 SAN zoning requirements
As a prerequisite for all the SAP System Copy use cases (SAP System Clone, SAP System
Copy, SAP System Refresh) SAN storage is required with the IBM Tivoli Storage FlashCopy
Manager (FCM) Adapter. SAN zoning on the target resource LPAR must be pre-configured:
This can be easily fulfilled if the volume(s) for the AIX operating system (rootvg) of the
target LPAR is located on the same storage like the SAP systems in scope of the System
Copy. Then the existing access to the rootvg on the target LPAR provides SAN access for the
SAP application volumes too. In this case, the “host” and “adapter” definition in the IBM XIV
storage system exist already, and the LPAR has an already zoned access in the SAN to the
storage system. The SAP application volumes are assigned to the LPAR by “LUN masking”
commands out of the SAP LVM storage adapter.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 28 of 169
The storage system should be zoned according to “single initiator zones”. Each zone set
should contain one (active) virtual fibre channel host port only, but can include multiple
storage adapter ports. (E.g. between three and six adapter ports for an IBM XIV storage
system). For IBM XIV storage the maximum number of paths that can be zoned to an AIX
LPAR is 32. However there is no benefit in having more than 24 paths.
Although not mandatory it is best practise to define the name of the host in the IBM XIV
storage system identical to the name of the LPAR or its hostname. If virtual fibre channel
adapters are used, then all the WWPNs need to be included in the IBM XIV port definition, as
well as in the SAN zoning definition. This includes all “active” WWPNs via the VIOS pair, as
well as the secondary WWPNs used during a Live Partition Mobility relocation event.
2.5.3 Considerations for the SAP LVM resources
Several storage attachment considerations need to be obeyed, and additional software
components need to be installed on the resource LPARs. All these are described shortly in this
section
2.5.3.1 VIOS / PowerVM and SAN attachment
For all the SAP System clone/System Copy scenarios that invoke storage-based cloning, all
the storage volumes for both the source and the target system need to be attached via virtual
Fibre Channel (vFC) adapters and using NPIV. Figure 8 illustrates the redundant attachment:
two redundant VIO Servers are installed in the POWER system. Each of the VIO servers has
one or more physical dual-port fibre channel adapters assigned to the partition. The two ports
of the fibre channel adapter card are connected to two different SAN fabrics. Also, the XIV
storage system is connected to both fabrics. If a LPAR has only one fibre channel adapter,
then zone it to three IBM XIV modules. Zone it to six modules only in case the LPAR has
very high throughput requirements.
Typically, each client LPAR is defined with four virtual fibre channel adapters: On each VIO
server two virtual fibre channel server adapters are created for each client LPAR. These server
adapters are assigned to the corresponding virtual fibre channel client adapters for each
LPAR. Four pairs of WWPNs are created (WWPN1, WWPN2, WWPN3, WWPN4, plus the
additional WWPNs for the LPM scenario).
LPAR
IBM XIV Storage
connected via vFC
(NPIV)
vFC Client
WWPN1
vFC Client
WWPN2
vFC Client
WWPN3
vFC Client
WWPN4
VIOS #1 VIOS #2
vFC ServervFC Server vFC ServervFC Server
FC adapter
port 1
FC adapter
port 0
FC adapter
port 1
FC adapter
port 0
FC adapter
port 1
FC adapter
port 0
FC adapter
port 1
FC adapter
port 0
Fabric #1
Fabric #2
Figure 8 redundant virtual fibre channel attachment
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 29 of 169
Use single initiator zones: Each zone set contains one (active) virtual fibre channel host port
and between three and six XIV ports. The maximum number of paths that can be zoned to an
AIX LPAR is 32. However there is no benefit in having more than 24 paths.
2.5.3.2 AIX operating system specifics
The AIX operating system on the managed systems needs to fulfil certain criteria:
IPSec is required for network fencing
All required filesets need to be installed. IPSec needs to be activated in the OS
DNS should be exploited
For the sake of simplicity, all the virtual hostnames of the SAP services should be
provided by DNS.
A central user management is recommended.
In the test environment, LDAP is used for that purpose. The LDAP filesets need to be
installed, and the OS needs to be configured as LDAP client.
No central repository for the services entries was used.
The /etc/services file need to be prepared and distributed to all resource LPARs
2.5.3.3 IBM Tivoli Storage FlashCopy Manager
IBM Tivoli Storage FlashCopy Manager software provides fast application-aware backups
and restores, so leveraging advanced snapshot technologies in IBM storage systems. The
software provides following features in general:
Near-instant application-aware snapshot backups, with minimal performance impact
for IBM DB2, Oracle, SAP, Microsoft SQL Server, and Exchange
Improvement in application availability and service levels through high-performance,
near-instant restore capabilities that reduce downtime
Integration with IBM Storwize V7000, IBM System Storage DS8000, IBM System
Storage SAN Volume Controller and IBM XIV Storage System on AIX, Solaris,
Linux, and Microsoft Windows
Fulfilment of advanced data protection and data reduction needs with optional
integration with IBM Tivoli Storage Manager
In the context of SAP landscape management, IBM Tivoli FlashCopy Manager is utilized to
create instantaneously a consistent copy of an online SAP system.
2.5.3.4 XCLI
The IBM XIV Storage System command-line interface (XCLI) provides a mechanism for
issuing commands to manage and maintain the XIV storage systems. XCLI commands are
entered on the XCLI client either by a user or by another application. The XCLI client and the
XIV system communicate using Secure Sockets Layer (SSL) over TCP/IP.
2.5.3.5 SAP Host Agent
The SAP Host Agent package contains all the required elements for centrally monitoring of
any hosts. It is used by SAP LVM to monitor and manage SAP services and resources also.
So the SAP Host Agent package needs to be installed and configured on each host containing
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 30 of 169
an SAP component managed by the SAP NetWeaver Landscape Virtualization Management
software.
The SAP Host Agent package is automatically installed as part of the “SAPinst” installation
procedure for any components of SAP NetWeaver 7.0 EHP2 and higher. However it may
need to be updated to a higher patch level to exploit all SAP LVM management features. It is
also required to install SAP Host Agent manually on operating systems without any SAP
systems. In our case these are operating systems that act as targets in SAP System Copy and
SAP Service Relocation use cases.
2.5.4 Considerations for the SAP LVM Services (SAP Systems)
All SAP systems need to be installed compliant to a SAP LVM environment. This includes
different considerations for user management, network, and storage. Some best practises are
discussed in the following.
2.5.4.1 User Management
Ensure that the administrative users are available with same UID and GID on all the
LPARs involved in the scenario.
A central user management solution provides the same UIDs, GIDs and passwords for
one user on all the computing nodes. In the environment described here this is
achieved by using LDAP.
2.5.4.2 Network considerations
The following guidelines for the network setup for the SAP system to be managed by SAP
LVM needs to be considered:
Provide a dedicated virtual hostname for each SAP service. The different services of
the SAP system (SAP database, SAP Central Services, SAP instances) shouldn‟t share
an IP address. Each service should own a dedicated IP address.
Provide a dedicated IP address for each virtual host name. Do not use multiple virtual
hostnames (IP alias names) referring to one and the same IP address.
Ensure that all the virtual host names can be resolved on the SAP LVM system.
Ensure that the reverse lookup on the SAP LVM server provides a one-to-one
mapping.
Ensure that all the virtual host names can be resolved on any resource LPAR. Ensure
that the reverse lookup on the LPAR provides a one-to-one mapping.
2.5.4.3 Storage Layout
Separate all storage for the different services. During the prepare/unprepare activities
is must be possible to handle the storage independent of the services:
o Create different AIX volume group for the filesystems related to the SAP
Central Instance and the Database
o If a Solution Manager Diagnostics agent exists for the SAP System, place its
filesystems to the Central Instance.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 31 of 169
o If there is a dedicated Solution Manager diagnostics agent for the Database,
then its filesystems need to be part of the volume group of the database
o If you intend to use FlashCopy backup for the production database also, then
split the database content into three VGs: database data (tablespaces), database
online logs, and other database files (instance directory, executables, archive
logs etc …)
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 32 of 169
3 Installation
Table 6 gives an overview about the operational model. In the test environment, both the
management environment and the managed environment are hosted on IBM POWER using
AIX operating system. Most of the deployment units for the managed environments were
directly added to the AIX operating system image: By this means they are part of the NIM
deployment process for a new SAP LVM resource and are automatically available on the new
system.
Management environment
Node Deployment Unit
Management LPAR SAP LVM
AIX
LDAP Client (IBM Tivoli Directory Server Client)
SAP NetWeaver 7.3 Java System / DB2 UDB
SAP NetWeaver Landscape Virtualization Management 1.0 SP03
IBM Tivoli Storage FlashCopy Manager [acsd]
XIVGUI XCLI
SAPHOSTAGENT package and IBM partner libraries
Management LPAR IBM Systems Director
AIX
LDAP Client (IBM Tivoli Directory Server Client) optional
IBM Systems Director incl. plug-in VMControl
IBM Systems Director Storage Control required for end-to-end provisioning of new LPARs
NIM Server
AIX, setup as NIM Master
LDAP Client (IBM Tivoli Directory Server Client) optional
IBM Systems Director VMControl subagent for NIM
LDAP Server AIX
IBM Tivoli Directory Server
DNS Server AIX bind
Managed Environment
Node Deployment Unit Part of
“golden AIX image”
on all SAP LVM resource LPARs
AIX Yes
LDAP Client (IBM Tivoli Directory Server Client) Yes
SAPHOSTAGENT package and IBM partner libraries Yes
IBM Tivoli Storage FlashCopy Manager [acsgen] Yes
XIVGUI XCLI Yes
SAPInst rename required for SAP System Copy on “target resource”
No (accessed
from a central NFS share)
on one SAP LVM resource LPAR
SAP ECC 6.0 source system / DB2 UDB
SAP PostCopyAutomation tool optional, allows for key-functionality in SAP System Copy
Table 6 Operational Model
This chapter describes the installation of the components for both the management and
managed environments for an AIX/ POWER environment. Not all the installation steps will
be described in all details: It is assumed that the reader has sufficient background knowledge
to perform those.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 33 of 169
In principle the SAP NetWeaver Landscape Virtualization Management and IBM Systems
Director could be deployed to another platform, e.g. LINUX/INTEL or MS Windows Intel:
Then the deployment unit “IBM Tivoli Storage FlashCopy Manager [acsd]” would need to be
moved to another AIX server (e.g. to the NIM server). XIVGUI XCLI and SAP Host Agent
package and partner libraries then would need to be installed on this server too.
3.1 Naming Conventions
In a SAP LVM environment SAP services can potentially run on any SAP LVM enabled
server. For example, you could install all service relevant data (database, binaries, and
working directories) on a distributed file system. In this case, we recommended putting this
data under a common root directory on that distributed file system.
A careful planning is required to avoid the collision of various names like IP names, user
names, user ids and group ids, directory structures, etc. One method is to include the SAP SID
(which is typically unique in a customer environment) in the required service IP names and
root directories of the service data. In addition, if you want to run multiple services on a single
compute node, you have to plan for unique SAP instance numbers as well, as you cannot run
two services with the same instance number (but different SIDs) within the same OS image.
3.1.1 Sample Service IP Names for SID C01
The following is one possible naming convention to build SAP service IP names. There are no
specific requirements for IP names in the SAP LVM infrastructure, however choosing and
sticking to one convention will help to simplify the administration.
c01db01 IP name for database server
c01ci01 IP name for central instance
c01as01 IP name for first application server instance
c01as02 IP name for second application server instance
3.1.2 Sample directory structure for SID C01 on a Distributed File System
Each SAP system should have a common high level subdirectory name to provide a standard
entry point to all data that belongs to the same system. For example if the distributed file
system is mounted at /sapfs, then all data belonging to SAP system C01 could be allocated
under the subdirectory /sapfs/sapC01. The “root” directory for the SAP system C01 would
then be /sapfs/sapC01. In that root directory, all instance relevant subdirectories are stored.
The mappings between the physical locations on the storage subsystem and the directory
locations as they are expected by the SAP kernel have to be defined in SAP LVM. The
controller will then create the required mapping between the new data location and the
required standard path before starting a service on a server. Table 7 shows a sample directory
structure for SID C01:
Directory Content Path in standard
installation
<root>/home/c01adm Home directory for SAP admin user /home/c01adm
<root>/db2 <root>/oracle
Database Server, Client, instance /db2 /oracle
<root>/sapmnt/C01 Directory for Executables, Profiles, Shared files /sapmnt/C01
<root>/usr/sap/c01ci01 Instance Directories for central instance /usr/sap/C01
<root>/usr/sap/c01as01 Instance Directories for application server 01 /usr/sap/C01
/sapfs/saptrans01 Common transport directory /usr/sap/trans
Table 7 Sample directory structure for a distributed file system implementation
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 34 of 169
The screenshot in Figure 9 shows the service configuration screen on the SAP LVM
Controller which illustrates the mapping between physical layout and the required layout for
the SAP database service EC1. The storage type is Distributed File System (DFS) with a
subtype of “gpfs", which refers to a General Parallel File System installation.
The entries with mount option dirmount define the required mapping between the physical
location and the mount point that is expected by the SAP kernel.
Figure 9 Service configuration in SAP LVM with distributed file system layout
3.1.3 Sample directory structure using Network Attached Storage (NAS)
The same concept as in the previous section can be used to layout the physical data locations
on a NAS device. A typical organization unit with NAS is called a volume where you store
directories and files. Those resources then have to be exported to the clients that need to
access them. The root directory for service C01 could for example look like
nas_server:/vol/sapvol1/sapC01, where nas_server is the IP name of the NAS device. Again
the mappings between physical locations and expected locations have to be defined in the
SAP LVM Controller. The controller will then create the required mappings (in this case
using client NFS mount instead of creating symbolic links) before starting a service on a
server.
3.2 Install Storage Infrastructure
3.2.1 Exploit GPFS as “adaptive storage”
This section provides a brief summary of setup tasks to get started with a GPFS cluster using
the ISICC SAP LVM landscape implementation as an example.
Using GPFS it is possible to carry out the use cases of the standard edition of LVM, that is in
general start, stop, and relocate SAP systems.
We do not cover in detail typical GPFS administration tasks like GPFS installation and
administration, the creation and management of storage pools, definition of file placement
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 35 of 169
policies, etc. For those tasks you need to refer to the standard product documentation and
available Redbooks. Especially to configure data availability and disaster recovery scenario,
you should carefully review the GPFS Base documentation (e.g. the “GPFS Concepts,
Planning, and Installation Guide” and the “GPFS Advanced Administration Guide”).
Additional information can be found in the GPFS Wiki
3.2.1.1 Planning for GPFS
3.2.1.1.1 Quorum Nodes
The first task is to decide on a node quorum strategy. As described in the “GPFS Concepts,
Planning, and Installation Guide”, GPFS enforces a node quorum rule to prevent multiple
nodes from assuming the role of the file system manager in the event of a network
communication problem in the cluster. A majority of quorum nodes must remain active in
order for the cluster to sustain normal file system usage (multi-node quorum).
The ISICC implementation uses a new node quorum mechanism that was introduced with
GPFS V2.3 – node quorum with tiebreaker disks, which allows you to run with as little as one
quorum node available as long as you have access to a majority of the quorum disks. Please
refer to Chapter 2 in the “GPFS Concepts, Planning, and Installation Guide” for a detailed
description of the quorum concept and guidelines to select quorum nodes. In the ISICC setup,
we use two quorum nodes and one tiebreaker disk.
3.2.1.2 Install and setup GPFS
3.2.1.2.1 Install GPFS software
You need to install the following filesets:
gpfs.base
gpfs.msg.en_US
This is best accomplished by including these filesets in a NIM installp bundle.
3.2.1.2.2 Create GPFS Cluster
The mmcrcluster is used to create a GPFS cluster. Our setup uses two quorum nodes: siccps10
and siccps12. The same nodes are also used as primary (option –p) and secondary (option –s)
configuration server to store the GPFS cluster configuration data. For cluster communication
we use ssh and scp instead of rsh and rcp, which has to be specified with the –r and –R
options. The –A option specifies that GPFS daemons are to be automatically started when
nodes come up. The default is not to start daemons automatically. Our cluster is named
siccfs.isicc (-C option).
root> mmcrcluster –N siccps12:manager-quorum,siccps10:manager-quorum –p siccps12 –s siccps10 –
r /usr/bin/ssh –R /usr/bin/scp –C siccfs.isicc –A
After a successful creation of the cluster, you can verify your setup using the mmlscluster
command.
root> mmlscluster
GPFS cluster information
========================
GPFS cluster name: siccfs.isicc
GPFS cluster id: 691765743288319456
GPFS UID domain: siccfs.isicc
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 36 of 169
GPFS cluster configuration servers:
-----------------------------------
Primary server: siccps12
Secondary server: siccps10
Node Daemon node name IP address Admin node name Designation
----------------------------------------------------------------------------------------------
1 siccps12 9.153.165.71 siccps12 quorum-manager
2 siccps10 9.153.165.69 siccps10 quorum-manager
Then you need to startup GPFS on all nodes and you should check the state of the cluster:
root> mmstartup -a
root> mmgetstate -aL
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
------------------------------------------------------------------------------------
1 siccps12 2 2 8 active quorum node
2 siccps10 2 2 8 active quorum node
3.2.1.2.3 Define Network Shared Disk (NSD) Devices
GPFS uses Network Shared Disk (NSD) devices to provide access to the disks. A NSD device
can either be physically attached (using a SAN infrastructure) to a node or it can be accessed
using a virtual connection (through the network) using a primary and, if specified, backup
NSD server. GPFS determines if a node has physical or virtual connectivity to an underlying
NSD through a sequence of commands invoked from the GPFS daemon. This determination
is called disk discovery and occurs at both initial GPFS startup as well as whenever a file
system is mounted.
In the ISICC PoC landscape, we used the following configuration file to create the NSD
devices:
/var/mmfs/config/nsd.disks:
#Description of disk attributes
#<disk name>:<primary NSD server>:<2ndary NSD server>:<disk usage>:<failure group>:<NSD name>
#Tiebreaker disk
hdisk3:::descOnly:-1:nsd_tbrk_01:
#Data and metadata disk for /siccfs
hdisk4:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_01:
hdisk5:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_02:
The first disk (hdisk3) holds file descriptor data only and is used as a tiebreaker disk. hdisk4
and hdisk5 are designated as data disks for our LVM cluster file system. We use our quorum
nodes as primary and backup NSD servers as well, although all of our nodes do have physical
access to the disks. This provides some automatic fallback capability should the SAN access
become unavailable.
Create the NSD devices using the mmcrnsd command and verify your setup using the
mmlsnsd command (the option -v yes specifies that the NSD are to be created only if the disk
has not been formatted by a previous invocation of the mmcrnsd command):
root> mmcrnsd -F /var/mmfs/config/nsd.disks -v yes
root> mmlsnsd -aL
File system Disk name NSD volume ID Primary node Backup node
---------------------------------------------------------------------------------------------
siccfs nsd_siccfs_01 0999A547447D7EFE siccps12 siccps10
siccfs nsd_siccfs_02 0999A547447D7F00 siccps12 siccps10
(free disk) nsd_tbrk_01 0999A547447D7DA6 (directly attached)
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 37 of 169
The specified disk file is rewritten during NSD creation. Each hdisk entry is preceded by a
comment (# sign) and followed by the equivalent NSD entry. The new file can then be reused
for file system creation during subsequent configuration steps. The file system create
command uses NSD devices and not AIX hdisks.
3.2.1.2.4 Activate Node Quorum using Tiebreaker Disks
You need to shutdown the GPFS cluster and use the mmchconfig to activate the tiebreaker
disks. Use the following commands to activate the NSD device nsd_tbrk_01 as a tiebreaker
disk:
root> mmshutdown –a
root> mmchconfig tiebreakerDisks="nsd_tbrk_01"
root> mmstartup –a
You can use the mmlsconfig and mmgetstate commands to check your configuration after the
switch to tiebreaker mode:
root> mmlsconfig
Configuration data for cluster siccfs.isicc:
-------------------------------------------
clusterName siccfs.isicc
clusterId 691765743288319456
clusterType lc
autoload no
useDiskLease yes
maxFeatureLevelAllowed 903
tiebreakerDisks nsd_tbrk_01
[siccps12]
takeOverSdrServ yes
File systems in cluster siccfs.isicc:
------------------------------------
/dev/siccfs
root> mmgetstate -aL
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
------------------------------------------------------------------------------------
1 siccps12 1* 2 8 active quorum node
2 siccps10 1* 2 8 active quorum node
In the mmlsconfig output you can see that device nsd_tbrk_01 is used as tiebreaker disk. The
quorum information in the mmgetstate output is displayed as “1*”, which indicates that this is
a two node tiebreaker disk cluster.
3.2.1.2.5 Create GPFS File System
The last step is to create the file system itself. We used the following excerpt of the disk file
that was created by the mmcrnsd command as input to create file system command:
/var/mmfs/config/disks.siccfs:
#Description of disk attributes
#<disk name>:<primary NSD server>:<2ndary NSD server>:<disk usage>:<failure group>:<NSD name>
#Data and metadata disk for /siccfs
# hdisk4:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_01:
nsd_siccfs_01:::dataAndMetadata:1::
# hdisk5:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_02:
nsd_siccfs_02:::dataAndMetadata:1::
The mmcrfs command was used to create the /siccfs cluster file system:
root> mmcrfs /siccfs /dev/siccfs -F /var/mmfs/config/disks.siccfs –A automount -B 256K -v yes
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 38 of 169
We used the –A automount option to specify that the file system is automatically mounted
when its first accessed.
3.2.1.2.6 Mount GPFS File System
If you did not use the automount option during file system creation, you can use the standard
AIX mount command to mount the cluster filesystem:
root> mount /siccfs
3.2.1.3 Considerations for SAP Service Installation on GPFS
A SAP system has to be up and running before it can be registered and configured in the
Landscape Virtualization Management software. SAP LVM is not able to create the required
environment (mount target directories) for a new installation, this has to be done manually
(see also section 3.4.5). You should first create the directory structure on the GPFS file
system according to your naming conventions and then mount the subdirectories at the mount
points that are expected by the SAP installation routines.
The following commands create a sample directory structure for system C01:
mkdir /siccfs/sapC01
mkdir /siccfs/sapC01/oracle
mkdir /siccfs/sapC01/sapmnt
mkdir /siccfs/sapC01/usrsap
mkdir /siccfs/sapC01/trans
mkdir /siccfs/sapC01/c01adm
Then you need to create the required mapping before you can start the installation. This is
done with a normal mount command using the option –v mmfs (you may have to create some
of the directories for the mount points first, e.g. mkdir /home/c01adm).
mount -v mmfs /siccfs/sapC01/oracle /oracle
mount -v mmfs /siccfs/sapC01/sapmnt /sapmnt/C01
mount -v mmfs /siccfs/sapC01/usrsap /usr/sap/C01
mount -v mmfs /siccfs/sapC01/trans /usr/sap/trans
mount -v mmfs /siccfs/sapC01/c01adm /home/c01adm
After the installation is finished and the new SAP service is registered at the Solution
Manager, you should unmount all directories and proceed to the SAP LVM configuration
tasks.
3.2.1.4 Service Configuration in SAP LVM for GPFS
When you enable a SAP service in SAP LVM, you have to configure the data storage on the
Services tab in the Configuration menu (for further details see chapter 4.4).
Data residing on a GPFS file system has to be defined using the Storage Type DFS. The
Partner Id is ibm and FS/SRID Type is gpfs. Please note that these values are case sensitive
and have to be entered using lower case. The Export Path designates the directories as they
are stored in the GPFS file system and the Mount Point/SRID is the mount point as it is
expected by the SAP kernel. The last field is Mount Options, the available options for GPFS
file systems are fsmount and dirmount. The first one can be used to mount the GPFS file
system itself (in case you don‟t use the automount option) and the second option to specify
the subsequent subdirectory mounts.
For example to mount subdirectory /usr/sap/EC1 on a GPFS file system that doesn‟t use the
automount option, you would need the following two entries:
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 39 of 169
Storage Type DFS
Mount Point/SRID /siccfs
Export Path /dev/siccfs
Mount Options fsmount
FS/SRID Type gpfs
Partner ID ibm
Storage Type DFS
Mount Point/SRID /usr/sap/EC1
Export Path /siccfs/sapEC1/usrsap
Mount Options dirmount
FS/SRID Type gpfs
Partner ID ibm
Figure 10shows an example configuration of a GPFS based SAP system within the ISICC
SAP LVM landscape. The database used in this example. For DB2 based systems the
filesystems need to be created and mounted accordingly to the DB2 file system structure.
Note that a separate resource pool for SAP systems using the GPFS file systems was created
in SAP LVM.
Figure 10 Example configuration of an SAP system using GPFS
3.2.1.5 Add a node to GPFS Cluster
You need to add an AIX node to the GPFS cluster before you can mount the cluster file
system. This is done with the mmaddnode command after the GPFS software installation is
completed:
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 40 of 169
root> mmaddnode –N siccps18
You can remove a node from the cluster using the mmdelnode command and we recommend
that you remove the node before you start a NIM re-installation of a specific node.
You can list the state of cluster nodes with the mmgetstate command:
root> mmgetstate -aL
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
------------------------------------------------------------------------------------
1 siccps12 1* 2 8 active quorum node
2 siccps10 1* 2 8 active quorum node
3 siccps17 1* 2 8 active
5 siccps13 1* 2 8 active
6 siccps14 1* 2 8 active
7 siccps15 1* 2 8 active
8 siccps16 1* 2 8 active
9 siccps18 1* 2 8 active
3.2.2 SAN Storage Management
The SAP NetWeaver Landscape Virtualization Management allows the direct management of
natively attached storage without having the need of a shared file system like GPFS.
IBM provides a deep integration of IBM storage functionality with SAP LVM. This includes,
but is not limited to, creation and mapping of storage volumes to hosts or LPARs, discovering
the volumes within the operating system and create file systems, creation of snapshots of
volumes and mapping the snapshots to hosts / LPARs, and re-mapping of volumes.
This allows automated and fast moving, cloning and copying of SAP systems.
This chapter briefly describes the storage components and setup used in this PoC. You can
find the detailed description how the components were installed and configured in chapter
3.4.4.
3.2.2.1 Storage Components
Storage system:
IBM XIV Storage System
Firmware Version: 10.2.4a
79 TB usable space
A separate pool „ISICC_cloud‟ was created for the volumes of the test environment
Storage Management Software:
XIV CLI version (XCLI) version: 3.1 build 18
IBM Tivoli FlashCopy Manager: 3.1.0.1
3.2.2.2 SAP LVM SAN configuration and file systems
You need to install the SAP instances to be managed with SAP LVM according to the SAP
installation guides.
The following file system layout is recommended, and required if IBM Tivoli® Storage
FlashCopy® Manager is used to create a FlashCopy backup or clone of a SAP DB instance;
typically this applies to SAP production systems. The number of sapdata file systems, see
chapter 3.4.5.1, might vary, the default is four. Three different volume groups need to be
configured, as also shown in Figure 11:
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 41 of 169
Figure 11 Volume Group and Filesystem layout
List of file systems used: root> mount
node mounted mounted over vfs date options
-------- --------------- --------------- ------ ------------ ---------------
/dev/hd4 / jfs2 Jul 02 09:47 rw,log=/dev/hd8
/dev/hd2 /usr jfs2 Jul 02 09:47 rw,log=/dev/hd8
/dev/hd9var /var jfs2 Jul 02 09:47 rw,log=/dev/hd8
/dev/hd3 /tmp jfs2 Jul 02 09:47 rw,log=/dev/hd8
/dev/hd1 /home jfs2 Jul 02 09:48 rw,log=/dev/hd8
/dev/hd11admin /admin jfs2 Jul 02 09:48 rw,log=/dev/hd8
/proc /proc procfs Jul 02 09:48 rw
/dev/hd10opt /opt jfs2 Jul 02 09:48 rw,log=/dev/hd8
/dev/livedump /var/adm/ras/livedump jfs2 Jul 02 09:48 rw,log=/dev/hd8
siccserv /siccfs /home/siccfs nfs4 Jul 02 09:48 ro,bg,hard,intr,vers=4,sec=sys
/dev/lv10 /sapmnt/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv00
/dev/lv11 /usr/sap/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv00
/dev/lv13 /home/cl6adm jfs2 Jul 02 11:02 rw,log=/dev/loglv00
/dev/lv12 /sapinst jfs2 Jul 02 11:02 rw,log=/dev/loglv00
/dev/lv15 /usr/sap/DAA/SMDA97 jfs2 Jul 02 11:02 rw,log=/dev/loglv00
/dev/lv09 /db2 jfs2 Jul 02 11:02 rw,log=/dev/loglv01
/dev/lv16 /db2/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv01
/dev/lv01 /db2/CL6/db2cl6 jfs2 Jul 02 11:02 rw,log=/dev/loglv02
/dev/lv03 /db2/CL6/log_dir jfs2 Jul 02 11:02 rw,log=/dev/loglv03
/dev/lv04 /db2/CL6/log_archive jfs2 Jul 02 11:02 rw,log=/dev/loglv01
/dev/lv02 /db2/CL6/db2dump jfs2 Jul 02 11:02 rw,log=/dev/loglv01
/dev/lv05 /db2/CL6/sapdata1 jfs2 Jul 02 11:02 rw,log=/dev/loglv02
/dev/lv06 /db2/CL6/sapdata2 jfs2 Jul 02 11:02 rw,log=/dev/loglv02
/dev/lv07 /db2/CL6/sapdata3 jfs2 Jul 02 11:02 rw,log=/dev/loglv02
/dev/lv08 /db2/CL6/sapdata4 jfs2 Jul 02 11:02 rw,log=/dev/loglv02
/dev/lv14 /var/db2 jfs2 Jul 02 11:02 rw,log=/dev/loglv01
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 42 of 169
List of volume groups used:
root> lsvg -l vg00 vg01 vg02 vg03
vg00:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
loglv00 jfs2log 1 1 1 open/syncd N/A
lv12 jfs2 256 256 1 open/syncd /sapinst
lv10 jfs2 251 251 1 open/syncd /sapmnt/CL6
lv11 jfs2 376 376 1 open/syncd /usr/sap/CL6
lv13 jfs2 8 8 1 open/syncd /home/cl6adm
lv15 jfs2 62 62 1 open/syncd /usr/sap/DAA/SMDA97
vg01:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
loglv01 jfs2log 1 1 1 open/syncd N/A
lv14 jfs2 1 1 1 open/syncd /var/db2
lv02 jfs2 7 7 1 open/syncd /db2/CL6/db2dump
lv04 jfs2 254 254 1 open/syncd /db2/CL6/log_archive
lv09 jfs2 178 178 1 open/syncd /db2
lv16 jfs2 1 1 1 open/syncd /db2/CL6
vg02:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
loglv02 jfs2log 1 1 1 open/syncd N/A
lv01 jfs2 1 1 1 open/syncd /db2/CL6/db2cl6
lv05 jfs2 313 313 1 open/syncd /db2/CL6/sapdata1
lv06 jfs2 313 313 1 open/syncd /db2/CL6/sapdata2
lv07 jfs2 313 313 1 open/syncd /db2/CL6/sapdata3
lv08 jfs2 313 313 1 open/syncd /db2/CL6/sapdata4
vg03:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
loglv03 jfs2log 1 1 1 open/syncd N/A
lv03 jfs2 497 497 1 open/syncd /db2/CL6/log_dir
List of hdisks used:
root> lsvg -p vg00 vg01 vg02 vg03
vg00:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk2 active 1019 65 00..00..00..00..65
vg01:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk1 active 1019 577 77..00..92..204..204
vg02:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk3 active 1279 25 00..00..00..00..25
vg03:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk4 active 1019 521 113..00..00..204..204
The list of volumes defined on the XIV storage system for SID CL6 is shown in Figure 12.
It also shows the snapshots created during the cloning of SAP system CL6 as source system.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 43 of 169
Figure 12 Volumes of SAP system CL6
The volume wwn is part of the hdisk unique_id.
Thisattribute is used to identify the hdisk on the storage system volume, as highlighted in
yellow and green:
XIV 1300360>>vol_list vol=CL6_datavg_1 -t wwn
WWN
00173800016800CA
root> lsattr -l hdisk3 -E -a unique_id
unique_id 2611200173800016800CA072810XIV03IBMfcp
Figure 13 shows the database instance of SAP system CL6 in LVM and the according mount
points.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 44 of 169
Figure 13 SAP System CL6 in LVM and mount points
The host names defined on the XIV storage system to which the volumes are mapped must
exactly match the real host name / node name.
Every storage system must be accessible from any LVM managed node: For SAN Volume
Controller and Storwize V7000, enable and configure ssh. For XIV systems install the XIV
CLI on every node.
IBM Tivoli FlashCopy Manager (FCM) is used to manage the snapshots on the storage
systems for SAP system cloning and copying purposes.
The installation and configuration steps of the XIV CLI and the FCM are described in detail
in chapter 3.4.4).
3.3 Install Management Environment
3.3.1 Install SAP NetWeaver Landscape Virtualization Management Software
The current SAP NetWeaver Landscape Virtualization 1.0 Software is an add-on to a SAP
NetWeaver 7.3 system. You must install a SAP NetWeaver Application Server Java of usage
type “Basic” before you can deploy the SAP LVM software packages.
Please download the appropriate installation guide for your intended operating system and
database platform combination and follow the instructions in that guide to install a SAP
NetWeaver Java System:
http://service.sap.com/instguidesNW73
As always please pay particular attention to the SAP notes mentioned in the “SAP Notes for
the Installation” section of the installation guide.
After the installation and configuration of the SAP Java System is complete, you need to
download and deploy the SAP NetWeaver Landscape and Virtualization Management
Software.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 45 of 169
Please download the latest version of the installation guide for your licensed version (standard
edition or enterprise edition):
https://websmp201.sap-ag.de/instlvm
Again it is important to check all SAP notes mentioned on the download page and in the
installation guide itself.
This installation guide describes in detail the necessary steps to install LVM after the initial
setup of SAP NW 7.3 AS Java.
At a high level, the installation of a SAP LVM management system
1. Install SAP NW 7.3 AS Java with usage type “BASIC”
2. Download the required SCA files for SAP LVM from the SAP Service Marketplace
http://service.sap.com/swdc
Software Downloads Browse our Download Catalog SAP NetWeaver and
complementary products SAP NW LANDSC VIRT MGT ENT
3. Deploy the downloaded SCA files using one of the supported implementation tools
(JSPM or telnet)
4. Perform the initial setup using the configuration wizard as described in the post-
installation section
The last step is described in detail on the SAP Help Portal at:
http://help.sap.com/nwlvm10
Initial Configuration Carrying Out the Initial Setup Using the Configuration Wizard
Follow the procedure documented in the help portal to step through the configuration wizard.
After the wizard finished successfully, you should be able to access SAP LVM using the
following URL:
http://<host>:<port>/lvm
3.3.2 Install IBM Tivoli Storage FlashCopy Manager
SAP LVM exploits IBM Tivoli Storage FlashCopy Manager to perform snapshot backups for
file systems related to the SAP Systems. The FCM setup is done according to the FlashCopy
backup of "Custom application environments". The snapshots are managed as backup
versions using the IBM Tivoli Storage FlashCopy Manager management policies. The target
volumes are acquired and mounted on the target LPAR: These are the storage volumes for the
target system.
The Management Agent (acsd) of Tivoli Storage FlashCopy Manager is installed centrally on
one of the management LPARs: "acsd" coordinates the snapshot backup operation and
controls the backup flow and mediates between the other agents. The Management Agent
provides access to the snapshot backup repository, which contains information about the valid
snapshot backups and their relationships to the snapshot capable storage devices.
On each resource LPAR an “acsgen” agent is started during the SAP cloning/ SAP system
copy process: This agent perform the storage related tasks on the source and target resource.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 46 of 169
Figure 14 illustrates the environment:
VIOS
vFC (NPIV)
VIOS
vFC (NPIV)
Management Environment
SAP LVMAIX
Storage Manager native Interfacee.g. IBM XIV XCLI 3.1 build 18/opt/IBM/xiv/XIVGUI3.1/xcli
Managed Environment
IBM Tivoli FlashCopy Manager 3.1.0.1
acsd/home/sapadm/acs
SAP Hostagent/usr/sap/hostctrl
SAP SourceAIX
Storage Manager native Interfacee.g. IBM XIV XCLI 3.1 build 18/opt/IBM/xiv/XIVGUI3.1/xcli
IBM Tivoli FlashCopy Manager 3.1.0.1
acsgen/home/sapadm/acs
SAP Hostagent/usr/sap/hostctrl
SAP TargetAIX
Storage Manager native Interfacee.g. IBM XIV XCLI 3.1 build 18/opt/IBM/xiv/XIVGUI3.1/xcli
IBM Tivoli FlashCopy Manager 3.1.0.1
acsgen/home/sapadm/acs
SAP Hostagent/usr/sap/hostctrl
IBM XIV
SVC / Storwize
Source
Target
SAN Storage
• IBM XIV, IBM SVC, or IBM
Storwize V7000 Storage
System
• Storage connected via vFC
(NPIV)
• Compliant Storage Layout
Figure 14 Overview of the FlashCopy Manager installation
The Tivoli FlashCopy Manager installation will follow the same steps as described in chapter
3.4.4.8 for the setup for a "Custom application environments". Additionally, a profile template
needs to be created on the management node. The following includes a sample profile for the
IBM XIV Storage:
>>> GLOBAL
# ACS_DIR /home/sapadm/acs
ACSD siccplvm 57328
# TRACE NO
<<<
>>> ACSD
ACS_REPOSITORY /home/sapadm/acs/acs_rep
# REPOSITORY_LABEL TSM
<<<
>>> DEVICE_CLASS STANDARD
COPYSERVICES_HARDWARE_TYPE XIV
# CLONE_DATABASE NO
#STORAGE_SYSTEM_ID 3984
PATH_TO_XCLI /opt/IBM/xiv/XIVGUI3.1
COPYSERVICES_SERVERNAME siccxiv
COPYSERVICES_USERNAME edmund
BACKUP_HOST_NAME siccps98
# RECON_INTERVAL 12
# GRACE_PERIOD 24
# USE_WRITABLE_SNAPSHOTS AUTO
# USE_CONSISTENCY_GROUPS YES
<<<
>>> CLIENT
LVM_FREEZE_THAW NO
ENHANCED_PARTITIONING NO
<<<
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 47 of 169
SAP LVM takes this as a template and generates “runtime” templates during the actual
execution. The “ACSD” entry in the GLOBAL section of the template needs to refer to the
hostname of the management system running the FCM acsd agent. SAP LVM may replace the
port number (2nd argument) during the actual run.
The ACS_REPOSITORY in the ACSD section will refer to the repository directory.
In the DEVICE_CLASS section, ensure that the following settings are included for IBM XIV
storage:
COPYSERVICES_HARDWARE_TYPE needs to be set to XIV
PATH_TO_XCLI needs to point to the installation directory of the XIV XCLI (see also
chapter 3.4.4.7)
COPYSERVICES_SERVERNAME needs to include the hostname / IP address of the XIV
system
COPYSERVICES_USERNAME needs to contain an administrative user on the IBM XIV
system. IBM Tivoli Storage FlashCopy Manager will use this user to communicate
with the IBM XIV System. This user needs to be created on the IBM XIV system first,
and the password needs to be entered later on during the FCM setup.
In the CLIENT section, the two entries LVM_FREEZE_THAW NO
ENHANCED_PARTITIONING NO
need to be specified.
3.3.3 Install IBM Systems Director
The IBM Systems Director installation package is available for download at the Systems
Director home page:
http://www-03.ibm.com/systems/software/director/
Select Resources Downloads Management Servers and choose the installation package
or DVD image for your intended target environment. Please use the current version 6.3 and do
not start with the older releases anymore.
The installation is dependent on the chosen platform and is fully documented at the Systems
Director Information Center:
http://pic.dhe.ibm.com/infocenter/director/pubs/index.jsp?topic=%2Fcom.ibm.director.main.h
elps.doc%2Ffqm0_main.html
Please follow the steps for “Installing on the management server” in the Planning and
Installing sections of the information center.
A number of short hands-on product demonstration movies for the Systems Director
installation and first use are available at:
http://www.ibm.com/developerworks/wikis/display/WikiPtype/Movies#Movies-sectiondir63
3.3.4 Install IBM Systems Director Plug-ins
Additional plug-ins are available on the Systems Director download page and might have to
be downloaded and installed separately. Please select the appropriate version that is supported
for Systems Director version 6.3.
3.3.4.1 IBM Systems Director VMControl
The current version of VMControl is already installed with IBM Systems Director
management server installation package. The further steps for discovery of the environment
and the base setup for OS deployment are described later in chapter 4.5.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 48 of 169
3.3.5 LDAP Server
3.3.5.1 User Management
A SAP LVM infrastructure requires a method to synchronize naming services on all enabled
servers. This includes information on users, groups, protocols, services, hosts, etc. Possible
solutions are NIS/NIS+ or the synchronization of local /etc files using tools like rdist.
However, development of NIS has stopped and the recommended solution for centralized user
management is LDAP (Lightweight Directory Access Protocol). LDAP is widely used and
has become the standard protocol for access to information registries, optimized for read
mostly access.
AIX clients can work with any LDAP server as long as it is V3 conform. In our sample
implementation, we implemented the IBM Tivoli Directory Server V6.3 as LDAP server.
More information is available at:
http://www-01.ibm.com/software/tivoli/products/directory-server/
The following introduces some of the LDAP terminology and discusses briefly some
important steps for planning and running LDAP on AIX. This is no replacement for the
product documentation. Especially an enterprise wide usage of LDAP sharing directory
information for other services requires careful planning and you need to consider issues like
namespaces, secure authentication using SSL, and implementing high availability using server
replication. These topics are fully covered in the product documentation and the available
Redbooks and whitepapers.
3.3.5.2 Schema
A schema defines which objects and attributes can be stored in a Directory Information Tree
(DIT). It defines the names of objects and attributes, syntax, and possible values as well as
whether attributes are required or optional for an object. For example, an object could be
people or server and attributes of the objects could be name, homedirectory, uid or ip-
address, hostname.
The following schemas are available for AIX user management and authentication:
AIX The AIX schema includes the aixAccount and aixAccessGroup object classes. This
schema offers the entire AIX user and group attributes. This schema is included to support
legacy LDAP installations prior to Version 5.2.
RFC2307 The RFC2307 schema includes the posixAccount, posixGroup and other NIS
related object classes. This experimental RFC defines a schema that allows NIS maps to be
imported into LDAP. RFC2307 only defines a subset of the AIX user and group attributes.
This schema supports any RFC2307 compliant platforms and AIX 5L Version 5.2.
RFC2307AIX The RFC2307AIX schema includes the RFC2307 schema plus the AIX
specific object classes, aixAuxAccount and aixAuxGroup. The AIX specific object classes
provide attributes to store additional attributes not defined by the RFC2307 standard.
The recommended schema for AIX5.2 and higher is RFC2307AIX and has been used for the
SAP LVM reference implementation. This has several advantages such as RFC2307 is
considered a standard for UNIX systems including Linux.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 49 of 169
3.3.5.3 Namespace
The LDAP naming model defines how entries are identified and organized. Entries are
organized in a tree-like structure called the Directory Information Tree (DIT) based on their
distinguished name (DN). A DN is a unique name that unambiguously identifies a single
entry. Distinguished Names are made up of a sequence of relative distinguished names
(RDN). Each RDN in a DN corresponds to a branch in the DIT leading from the root of the
DIT to the directory entry.
A RDN typically has the form <attribute name> = <value>. A DN is composed of a sequence
of relative distinguished names separated by commas.
The following DIT shows a sample DIT with a suffix o=isicc,c=de:
Figure 15 Sample LDAP Directory Information Tree for AIX user management
3.3.5.4 LDAP Server Installation and Configuration
For LDAP server implementations please review and follow the instructions in the mentioned
references.
3.3.5.5 LDAP client Installation
AIX LDAP client is available in a 32-bit and a 64-bit version. Their filesets are shipped with
the AIX CDs and should be included in the install images for the SAPL LVM client nodes.
3.3.5.6 LDAP client configuration
mksecldap is an AIX command for IBM Directory server and client setup. Please check the
man pages or the online documentation for the full syntax description.
The mksecldap command performs the required updates in various configuration files, starts
the secldapclntd daemon, and puts an entry in /etc/inittab so that the seclpdapclntd daemon
will be automatically started after reboot.
The following command is used in our demo environment to configure an AIX LDAP client:
/usr/sbin/mksecldap -c -h siccldap1,siccldap2 -a cn=root -p ldappw
There are a number of commands available to control the client daemon:
c=de
o=isicc
cn=user cn=group
uid=c01adm
uid=orac01
gid=sapsys
gid=dba
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 50 of 169
start-secldapclntd start ldap client daemon
stop-secldapclntd stop ldap client daemon
restart-secldapclntd restart the ldap client daemon
ls-secldapclntd show the status of the ldap client daemon
flush-secldapclntd clear the cache of the ldap client daemon
3.3.5.7 User Authentication
The last step required to setup user login through LDAP is to change a user‟s authentication
mechanism. User authentication is controlled by the SYSTEM attribute in the
/etc/security/user file. For example to enable LDAP user c01adm to login to the local system,
you could execute the following command;
chuser SYSTEM=LDAP registry=LDAP c01adm
To allow all LDAP users to login to the system, one can set each LDAP user's SYSTEM
attribute to LDAP, or simply set the default stanza's SYSTEM attribute to LDAP. This has to
be done manually in the /etc/security/user file, where you need to change the following two
attributes in the default stanza.
Default:
SYSTEM = "LDAP or compat"
registry = LDAP
We recommend using local authentication for the root user so that you can still login to the
server in case there is something wrong with the LDAP setup. With local authentication the
root user‟s stanza should look like:
root:
SYSTEM = "compat"
registry = files
In our sample environment the complete client setup is automated in a NIM script. The script
uses mksecldap to configure the client and then runs a Perl command to automatically change
the default and root user‟s stanza to the appropriate:
3.3.5.7.1 Sample customizing script for automated LDAP client setup
#Setup LDAP Client
#AIX users are managed by LDAP
/usr/sbin/mksecldap -c -h siccldap1,siccldap2 -a cn=root -p ldappw
perl -p -i -e 'if ( /default:/ .. /SYSTEM =/ ) {
s/"compat"/"LDAP or compat"\n\tregistry = LDAP/;
} elsif ( /root:/ .. /SYSTEM =/ ) {
s/"compat"/"compat"\n\tregistry = files/;
}' /etc/security/user
3.3.5.8 Create Users and Groups
The AIX mkuser command is able to create a user on the LDAP server using the load module
option “-R LDAP”. Unfortunately the “smit mkuser” panel doesn‟t allow for this option. You
should create a new user using the mkuser command. Once a user is created, you can use the
“smit chuser” panel to change user attributes. For example to create a user c01adm, use the
command
mkuser –R LDAP registry=LDAP c01adm
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 51 of 169
Then use the “smit chuser” panel to change the other required attributes (of course, you could
also set them directly using the appropriate attribute options with the mkuser command).
The problem also applies to the mkgroup command, so you have to create the required groups
manually as well: mkgroup –R LDAP sapsys
Consider using some naming conventions for the numerical user id, for example by reserving
certain ranges for the SAP and database administration users:
mkgroup -R LDAP registry=LDAP id=400 dbc01adm
mkgroup -R LDAP registry=LDAP id=401 dbc01ctl
mkgroup -R LDAP registry=LDAP id=402 dbc01mnt
mkgroup -R LDAP registry=LDAP id=403 dbc01mon
mkgroup -R LDAP registry=LDAP id=404 sapsys
mkgroup -R LDAP registry=LDAP id=405 sapinst
mkuser -R LDAP registry=LDAP id=400 pgrp=sapsys
groups='sapsys,dbc01ctl,dbc01mon,sapinst' home='/home/c01adm' gecos='SAP
System Administrator' shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1
core=2097151 c01adm
mkuser -R LDAP registry=LDAP id=401 pgrp=dbc01adm groups='dbc01adm'
home='/db2/db2c01' gecos='SAP Database Administrator'
shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 db2c01
mkuser -R LDAP registry=LDAP id=402 pgrp=dbc01mnt groups='dbc01mnt'
home='/home/sapc01' gecos='ABAP Database Connect User'
shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 sapc01
mkuser -R LDAP registry=LDAP id=403 pgrp=dbc01mon groups='dbc01mon'
home='/home/sapc01db' gecos='Java Database Connect User'
shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 sapc01db
3.3.6 Other Central Management Components
3.3.6.1 OS Management
We highly recommend a centralized OS software management concept. There are two
common solutions to manage the installation and maintenance of AIX images in a server
landscape. The first one is using the AIX built-in Network Installation Management (NIM)
features. The second one is to use the VMControl plugin of IBM Systems Director.
VMControl uses NIM under the cover as well, but will simplify some of the administration
tasks.
NIM allows the installation and maintenance of AIX base operating system and additional
software from a central repository. As images are sent across the network to the partition to be
installed, the installation steps can be performed fully remotely without needing physical
access to the server. During and after base installation, you can customize the machines by
running specific customizing scripts. Later on, you can deploy fixes and new software images
to the server partitions from a central administration point.
NIM supports several different machine types: diskless, dataless and standalone. A diskless
machine has no own disks, all file systems and other resources are installed remotely and
accessed via the network. A dataless machine has local paging and dump devices, the file
systems for / and /usr are located on a remote server, all the other file systems are located
either local or remote. On a standalone machine, all file systems and resources are installed
on local disks. Due to performance and availability reasons, we recommend using local (or
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 52 of 169
SAN attached disks) on the computing nodes and install them as standalone machines. It is
possible to use diskless or dataless machines for test and development systems. They boot of
the network and mount the OS from the NIM server.
The following two Redbooks describe in detail the concepts of centralized OS management
using NIM or VMcontrol:
NIM from A to Z in AIX 5L
http://www.redbooks.ibm.com/abstracts/sg247296.html?Open
IBM Systems Director VMControl Implementation Guide on IBM Power Systems
http://www.redbooks.ibm.com/abstracts/sg247829.html?Open
3.3.6.2 Network Infrastructure
There are no special network infrastructure requirements for an SAP LVM landscape.
Common standard today and recommended environment is a switched Gigabit Ethernet
network infrastructure.
The major difference compared to a traditional SAP implementation is the consequent usage
of virtual IP names and addresses assigned to a SAP service. These addresses need to be
implemented in the customer‟s name services infrastructure, typically based on DNS (please
refer to the section 3.1. for some recommendations about naming of virtual IP names). If the
decision is to pick virtual network addresses in new subnets, one might have to update router
tables, saprouter definitions, etc. to ensure that the end user can access the SAP service using
the virtual IP name. You also have to change SAP logon groups to reflect the virtual IP
names.
Note 962955 - Use of virtual TCP/IP host names
3.4 Install Managed environments
The following section provides an overview about the preparation steps required for the
environment. This includes considerations about
PowerVM / VIOS
IBM Systems Director
IBM HMC
OS setup for the resources
SAP Installation
3.4.1 VIOS/ PowerVM
Base assumption is that the volume(s) for the AIX operating system (rootvg) is/are located on
the same storage system like the SAP system also, and that the “rootvg” storage volumes are
also attached via virtual fibre channel (vFC). In this case, the “host” and “adapter” definition
in the storage system required for the access to the “rootvg” volumes will ensure the required
connectivity to the SAP application related volumes in the storage system also.
3.4.2 IBM Systems Director Setup
The following describes high-level configuration steps required for the integration of either
IBM Systems Director or IBM FlexSystem Manager with SAP LVM: Further details are
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 53 of 169
described in SAP Note 1728222 - Configuration of IBM Systems Director VMControl for
SAP LVM
In the text the term “virtualization manager” is applied for both of the integration scenarios:
Both “IBM Systems Director” and “IBM FlexSystem Manager” require the identical steps for
the setup. The tasks on the virtualization manager include:
Discover the IBM HMCs on the virtualization manager and configure access to them.
Perform a "collect inventory" task on the virtualization manager for IBM HMC, IBM
Power servers, and for all logical partitions (LPARs).
Discover operating systems running on LPARs, configure access and perform "collect
inventory" task on virtualization manager for all the discovered operating systems.
Define a user for of SAP LVM access on the virtualization manager
The default administrator user can be used as user for SAP LVM too: The
administrators in SAP LVM then have the same credentials and get data for all the
IBM Power Systems configured in the virtualization manager. To restrict the access to
the specific set of IBM Power Systems that are desired to be managed by SAP LVM
(and so from SAP LVM administrators) a specific user can be created in the
virtualization manager: This user doesn‟t have all the credentials for the whole set of
IBM Power Systems defined in the virtualization manager, but only to those in scope
of the SAP LVM environment. The configuration is described in chapter 4.5.
Create a security certificate on the virtualization manager
Per default, the virtualization manager uses Port 8422 for communication:
Communication via this port must be allowed in the network and firewall settings for
the environment.
o Open the URL “https://<virtualization manager host>:8422" in a browser
window. During the first connection attempt the browser displays a notification
window to accept the certificate. This virtualization manager certificate is
stored in the internet browser configuration locally, and can be exported as
certificate file.
o The virtualization manager certificate needs to be imported into the SAP
NetWeaver CE system where the SAP LVM software is deployed to.
The certificate store must be "TrustedCAs".
Check that the hostname in the certificate is valid.
For more information on this topic see also the SAP documentation "Using the
AS Java Key Storage".
Using the virtualization manager for AIX OS provisioning requires to discover, to
access and to perform "collect inventory" tasks for NIM, VIO servers, FC switches
and storage managers in addition.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 54 of 169
It must be possible to
capture and deploy an AIX operating system via NIM (4.5).
select a particular IBM Power server as target for deployment
select a storage system for deployment
within the different layers of the virtualization manager (VMControl, Storage
Control). Then similar provisioning operations can be triggered out of SAP LVM.
Using the virtualization manager for LPAR relocation (Live Partition Mobility)
requires configuration of “virtual farms”. If it is intended to migrate LPAR "X" hosted
on IBM Power system "A" to IBM Power system "B", then both Power system "A"
and Power system "B" must be members of the same virtual farm.
If must be possible to relocate a LPAR using virtual farms out of the virtualization
manager. Then the same operation can be triggered out of SAP LVM.
3.4.3 HMC setup
The recommended setup for SAP LVM is to use IBM Systems Director (or IBM FlexSystem
Manager) as Virtualization Manager. However the management via IBM Power HMC
instead of IBM systems Director is possible too. This is described in detail in SAP Note
1728293 - Configuration of IBM Power HMC for SAP LVM.
3.4.4 Install compute nodes
This section describes the base configuration steps on an AIX operating system level for the
compute nodes. All the required steps are described, and need to be executed at least on the
first system image: This image then can be captured (NIM, mksysb) and can be used as
“golden image” template for the deployment of all the additional compute nodes. Main steps
include
Activate IPSec
Check/ Update DNS configuration
Configuration of the LDAP client
Adapt /etc/services base configuration
Check/adapt filesystem sizes
Install SAP Host Agent package and IBM libraries
Install XIVGUI (IBM XIV Storage systems)
Install IBM Tivoli Storage FlashCopy Manager
3.4.4.1 Activate IPSec
Each of the resource LPARs that may participate in a system provisioning use case for SAP
system cloning or SAP system copy needs to be prepared for the network isolation. In an AIX
environment, the network isolation is provided via IPSec: IP Security, known commonly as
IPSec, is a protocol developed by the Internet Engineering Task Force (IETF), designed to
provide "end-to-end" cryptographically-based security for IP network connections. IPSec
implementations include a method of restricting connections to various services, based on
their origin and destination. This feature is known as packet filtering. All packets on an IP
network originate from an IP address and a port, and are destined for another IP address and
port. A packet filter is a physical or virtual device that sits between the endpoints of a
connection and determines whether the packet should be permitted to continue to its
destination. The decision is made by comparing various attributes of the packet to rules that
are defined by the administrator of the packet filter. Those attributes include source address
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 55 of 169
and subnet, source port, destination address and subnet, destination port, protocol, direction of
the connection, and fragmentation of the packet.
Make sure that APAR IZ98741 "DYNAMIC FILTER RULE ID NOT MODIFIED BY
RMFILT -N ALL" is installed in the AIX OS image
IPSec is activated via the AIX SMIT tool using fastpath “ips4_start”:
# smitty ips4_start
And then switching to the next panel:
Figure 16 Start IPSec
Alternatively, IPSec can be started directly via the command line:
# /usr/sbin/mkdev -c ipsec -t 4
No manual definitions are required for packet filters: The packet filters for the system
isolation are defined/released automatically during the execution of the System Copy/ System
Clone workflows by SAP LVM.
Figure 17 illustrates an additionally required step during the resource definition for the
resource LPAR. The checkbox “Isolation Ready” must be enabled so that the LPAR can
participate as a target resource in a System Copy/System Clone scenario:
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 56 of 169
Figure 17 Define the Resource as “Isolation Ready” in SAP LVM
This is described in detail in Chapter 4.3
3.4.4.2 DNS Name resolution
Ensure that the search order for DNS name resolution is first based on the local /etc/hosts file,
and then on the external DNS nameserver.
#vi /etc/netsvc.conf
hosts = local,bind4
In our test environment, the content of the /etc/host file includes the IP label for the hostname
of the resource only. All external addresses are resolved solely via DNS. In this case,
communication to the DNS server needs to be allowed during the “isolated”, “network
fenced” state within the System Clone / System Copy workflow also: The required settings
for allowing DNS nameserver communication are is illustrated in chapter 4.4, Figure 41.
3.4.4.3 Configure LDAP client
As prerequisite for the use of LDAP the IBM Tivoli Directory Server Client needs to be
installed on each LPAR.
idsldap.clt32bit61
idsldap.clt64bit61
The LDAP client needs to be defined on the LPAR in the same way as described in chapter
3.3.5.5 and 3.3.5.6.
3.4.4.4 Adapt /etc/services entries
Table 8 illustrates the ports to be predefined for ABAP instances. For the sake of simplicity
all ports covering the full instance number range from 00 to 99 are added to /etc/services.
In total 400 ports for sapdpXX, sapgwXX, sapdpXXs and sapgwXXs are added. If only a
subset of the ports is required in the organization (e.g. due to SAP instance number standards)
then it is sufficient to add only those specific ports to the /etc/services file.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 57 of 169
Port Description /etc/services content
sapdpXX
XX { 00, 01, 02, … 99 }
SAP System Dispatcher Ports sapdp00 3200/tcp
…
sapdp99 3299/tcp
sapgwXX
XX { 00, 01, 02, … 99 }
SAP System Gateway Ports sapgw00 3300/tcp
…
sapgw99 3399/tcp
sapdpXXs
XX { 00, 01, 02, … 99 }
SAP System Dispatcher Security Ports sapdp00s 4700/tcp
…
sapdp99s 4799/tcp
sapgwXXs
XX { 00, 01, 02, … 99 }
SAP System Gateway Security Port sapgw00s 4800/tcp
…
sapgw99s 4899/tcp
Table 8 Entries in /etc/services
3.4.4.5 Adapt filesystem sizes
Several software components need to be available on each of the LPARs of the managed
environment. If all the software components are installed locally on the LPAR, then sufficient
free space is required in certain filesystems for the management components. Table 9 gives an
example about required sizes for the components in the “local” filesystems/ directories:
Directory Size
(approx.)
MByte
VG Content
/usr/sap/hostctrl 200 rootvg or saplocalvg
SAP Host Agent Package, together with the IBM libraries
/opt/IBM/xiv/XIVGUI3.1 (example) 350 rootvg XIV XCLI Installation
/usr/tivoli/tsfcm 700 rootvg Installation directory for IBM Tivoli Storage
/home/sapadm/acs (example) 700 rootvg or saplocalvg
Work directory for IBM Tivoli Storage FlashCopy Manager (SAP LVM context)
Table 9 Local Filesystem layout
Increase the size of the /usr filesystem to allow to install the SAP Host Agent package
and the IBM Tivoli Storage FlashCopy Manager software (FCM)
Increase the size of the /home filesystem for the copy of the FCM package in
/home/sapadm/acs
Increase the size of the /opt filesystem for the XIVGUI package
# chfs -a size=+1500M /usr
# chfs –a size=+1G /home
# chfs -a size=+1G /opt
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 58 of 169
3.4.4.6 Install SAP Host Agent and IBM partner libraries
The runtime user for the SAP Host Agent is user “sapadm”: SAP LVM uses this user id to
authenticate to the SAP Host Agent on the managed system. Base assumption is that user
“sapadm” has been created in the LDAP system and is available on all the compute nodes
before the SAP Host Agent installation.
The SAP Host Agent package requires “root” user credentials for the installation.
The group “sapsys” has to exist on the host. If it doesn‟t exist, create the group, for
example with the command “mkgroup”.
The user “sapadm” has to exist on the host, and must be a member of the “sapsys”
group. User “sapadm” is used for authentication from SAP LVM to the host. If the
user does not yet exist, create the user, for example with the command “mkuser”.
Create home directory for sapadm on the resource LPAR. The FCM setup on the
managed LPAR for SAP LVM is located below the home directory of user sapadm
#mkdir /home/sapadm
#chown sapadm.sapsys /home/sapadm
Download the recent SAPHOSTAGENT.SAR archive and uncompress it using the
SAPCAR tool.
The archive contains the “saphostexec” program. The SAP Host Agent package is
installed/ upgraded by executing “saphostexec –install”, respectively “saphostexec –
upgrade” as root user. Install the SAP Host Agent package to directory
/usr/sap/hostctrl.
Some SAP Host Agent runtime executables have the S-User bit set. So privileged
commands can be executed during the prepare/ unprepare operations.
Copy the IBM specific libraries for the SAP Host Agent
# mkdir /usr/sap/hostctrl/exe
# mkdir /usr/sap/hostctrl/exe/operations.d
# mkdir /usr/sap/hostctrl/exe/operations.d/IBM
# mkdir /usr/sap/hostctrl/exe/operations.d/IBM/traces
SAP LVM calls the IBM storage library for mount/ unmount operations as hooks on top
of the ACOSPrepare and PostACOSUnprepare steps. During the execution of these hooks
the script /usr/sap/hostctrl/exe/operations.d/IBM/ibm_lvm_mgmt.ksh is called.
Verify that the settings for ACOSPrepare and PostACOSUnprepare hooks are
included in the profile /usr/sap/hostctrl/exe/host_profile of the SAP Host Agent.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 59 of 169
Figure 18 illustrates the configuration for the host_profile:
Figure 18 Configuration of host_profile
o Ensure that the script ibm_lvm_mgmt.ksh is located in the
/usr/sap/hostctrl/exe/operations.d/IBM directory, that it is readable and has the
execute bit set.
SAP LVM calls to the IBM Tivoli Storage FlashCopy Manager (FCMCLI) are
managed as a “Host Agent Registered Script” (See SAP Note #1465491 for details).
Make sure that the configuration files FCMCLI.conf is available in the
/usr/sap/hostctrl/exe/operations.d directory on all the resource LPARs, and on the
management server hosting SAP LVM (Figure 19):
Figure 19 Content of .conf file FCMCLI.conf
Make sure that the file fcmcli.sh exists in directory
/usr/sap/hostctrl/exe/operations.d/IBM and is readable and executable.
SAP LVM calls to the XIV command line interface (XCLI) are managed as a “Host
Agent Registered Script” (See SAP Note #1465491 for details). Make sure that the
.conf files XCLI.conf is available in the /usr/sap/hostctrl/exe/operations.d on the
management server hosting SAP LVM (Figure 20)
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 60 of 169
Figure 20 Content of .conf file XCLI.conf
Make sure that the file xcli.sh exists in directory
/usr/sap/hostctrl/exe/operations.d/IBM on the management server hosting SAP LVM
and is readable and executable.
Start/ Restart the SAP Host Agent
If there were any changes to the profile or additional configuraton files, restart the
host agent on the affected LPAR.
Additional considerations for the SAP Hostagent are found in the SAP Note
Note 1292836 - Logfiles of the SAP Hostagent
3.4.4.7 Install XIVGUI
In the PoC environment XIVGUI 3.1 build 18 is installed. The XIVGUI image is deployed to
the directory /opt/IBM/xiv (XIVGUI is part of the OS image).
Parameter Value
XCLI Installation Directory /opt/IBM/xiv/XIVGUI3.1
Table 10 XIVGUI Install directory
At first create the target directories for the XIVGUI image
mkdir /opt/IBM/xiv
mkdir /opt/IBM/xiv/XIVGUI3.1
Install additional rpm packages for the XIVGUI image.
Although this is a prerequisite this is not explicitly mentioned in the XIV XCLI
installation guide: Required rpm packages include:
o libgcc
o libstdc
o readline-5
readline-5 additionally requires
o bash
o info
#rpm -i libgcc-4.2.0-3.aix6.1.ppc.rpm
#rpm -i libstdc++-4.2.0-3.aix6.1.ppc.rpm
#rpm -i gcc-4.2.0-3.aix6.1.ppc.rpm
#rpm -i bash-4.2-6.aix5.1.ppc.rpm
#rpm -i info-4.6-1.aix5.1.ppc.rpm
#rpm -i readline-5.2-3.aix5.1.ppc.rpm
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 61 of 169
Adjust the library settings and the LIBPATH to include these shared libraries
#cd /opt/freeware/lib
#ln -s /opt/freeware/lib/gcc/powerpc-ibm-aix6.1.0.0/4.2.0/libgcc_s.a
#ln -s /opt/freeware/lib/gcc/powerpc-ibm-aix6.1.0.0/4.2.0/libstdc++.a
#export LIBPATH=$LIBPATH:/opt/freeware/lib
Extract the XIVGUI software from the Software Image Repository
# ./xivgui-3.1-build18-aix.bin
Adapt the xcli script
There is a bug in the xcli script: xcli can be started only out of the installation path.
xcli needs to be patched to be able to start it from any location.
Edit the xcli script and add the line
prg="$0"
in the upper section of the xcli script
Validate the XCLI by a test connection to the IBM XIV storage system
# /opt/IBM/xiv/XIVGUI3.1/xcli
User Name: ...
Password: ...
Machine IP/Hostname: ...
connecting.
XIV 1300360>>
3.4.4.8 Install IBM Tivoli Storage FlashCopy Manager
SAP LVM exploits IBM Tivoli FlashCopy Manager for the backup of "Custom application
environments". On the managed systems, the “acsgen” agent is used to invoke the different
storage related activities on the LPAR during the System Copy/ System Cloning process.
Base Installation of IBM Tivoli Storage FlashCopy Manager
./3.1.0.1-TIV-TSFCMFTP-AIX.bin
Parameters:
- Choose Locale: 2- English
- PRESS <ENTER> TO CONTINUE
- Press Enter to continue viewing the license agreement, or enter "1" to
accept the agreement
- Choose the Install Set to be installed by this installer: 4 - IBM
Tivoli Storage FlashCopy (R) Manager (Custom Applications)
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 62 of 169
- ENTER AN ABSOLUTE PATH: /usr/tivoli/tsfcm/acs_3.1.0.1
- IS THIS CORRECT? (Y/N): Y
- PRESS <ENTER> TO CONTINUE
- PRESS <ENTER> TO EXIT THE INSTALLER
Check the license file (contained in IBM Tivoli FlashCopy Manager base installation).
If the FCM 3.1.0.1 package was downloaded as an update package, it will not include
the license file. The license file included in the FCM 3.1.0.0 base package needs to be
copied to the directory:
o Copy license file “tsmacs.lic” from the FCM 3.1.0.0 base package to the FCM
install directory /usr/tivoli/tsfcm/acs_3.1.0.1
Create Directory for FCM profile / environment:
mkdir /home/sapadm
chown sapadm:sapsys /home/sapadm
mkdir /home/sapadm/acs
chown sapadm:sapsys /home/sapadm/acs
o Run the setup for FCM custom application
cd /usr/tivoli/tsfcm/acs_3.1.0.1
./setup_gen.sh
Script not started from 'acs' sub-directory of the Home directory of a
custom application user. Trying to locate the correct Home directory or
install IBM Tivoli Storage FlashCopy(R) Manager in a new Home directory ...
Please enter the Home directory:
/home/sapadm
This is a new installation of IBM Tivoli Storage FlashCopy(R) Manager into
this Home directory.
Please enter the name of the user of this Home directory:
sapadm
Please enter the group name of the user:
sapsys
Installing/Updating IBM Tivoli Storage FlashCopy(R) Manager installation
for /home/sapadm/acs
Checking environment
- - - - - - - - - - -
+ OK: installation directory exists.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 63 of 169
+ OK: setup_gen.sh runs in IBM Tivoli Storage FlashCopy(R) Manager
installation directory '/usr/tivoli/tsfcm/acs_3.1.0.1'
+ OK: IBM Tivoli Storage FlashCopy(R) Manager installation will be
installed to '/home/sapadm' for the first time.
Installation messages
- - - - - - - - - - -
+ OK: New installation !
+ OK: IBM Tivoli Storage FlashCopy(R) Manager directory '/home/sapadm/acs'
created successfully.
+ OK: Files successfully copied.
checking /home/sapadm/acs/acsgen ...
OK
Select one of these configurations:
(1) On-Site Production System configuration with optional remote Backup
System configuration
(2) On-Site Backup System configuration
Enter '1' to configure IBM Tivoli Storage FlashCopy(R) Manager on the
production server with the option to remotely synchronize the configuration
of one or multiple backup systems using Secure Shell.
Enter '2' to configure IBM Tivoli Storage FlashCopy(R) Manager on the
backup system as a 'separate installation'.
2
Going to install on backup server
Please enter the directory where the profile will be stored:
[/home/sapadm/acs]
Deactivating IBM Tivoli Storage FlashCopy(R) Manager management daemon.
Are you going to perform off-loaded backups to Tivoli Storage Manager?
[y|n]
n
Profile configuration
=====================
Special commands:
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 64 of 169
'?' ==> Show help for current parameter
'!d' ==> Delete current parameter instance (only applicable to
multi-instance parameters)
FMM8422I Creating new profile '/home/sapadm/acs/profile' for application
'GENERIC' ...
****** Profile parameters for section GLOBAL : ******
Path of the IBM Tivoli Storage FlashCopy(R) Manager directory {ACS_DIR} =
[/home/sapadm/acs]
Hostname and port of machine running Management Agent {ACSD} (<hostname>
<port>) = [] siccplvm 57328
Low level tracing {TRACE} (YES|NO) = [NO]
FMM8426I Saving profile '/home/sapadm/acs/profile' ...
FMM1555I Profile successfully created.
Please enter the password for authentication with the ACS daemon:
Please re-enter password for verification:
Creating password file at /home/sapadm/acs/shared/pwd.acsd.
A copy of this file needs to be available to all components that connect to
acsd.
Enter the device classes to use for this Backup System. Make sure each
device class is separated by a comma.
STANDARD
IBM Tivoli Storage FlashCopy(R) Manager requires at least two daemon
processes to be running at all times.
IBM Tivoli Storage FlashCopy(R) Manager can add the necessary daemon
processes to the inittab.
Alternatively you may choose to start and stop these processes yourself,
for instance if you want to include the processes in your HA policy.
Do you want IBM Tivoli Storage FlashCopy(R) Manager to create the inittab
entries for you? [y|n]
n
Please run the following daemon processes:
/home/sapadm/acs/acsgen -D -M -s STANDARD
Please make sure that the command '/home/sapadm/acs/tsm4acs' is started by
a scheduler or manually.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 65 of 169
No special customization for the FCM profile is required on the resource LPAR.
SAP LVM will derive the required content for the profile out of the master profile on the
FCM management LPAR (“acsd”), and copy the profile to the source and target resource
during the execution of the SAP System Cloning/SAP System Copy workflow.
o Tivoli Storage FlashCopy Manager password file
The FlashCopy Manager configuration on the central node (acsd) generates a password file in
/home/sapadm/acs/shared/pwd.acsd. The password file needs to be copied to all the
FlashCopy Manager directories on the target resources:
It contains the connection password to the acsd, and also the connection password to the
Storage System.
3.4.5 SAP System
In the following section, base considerations for the installation of a new SAP system, as well
as the eventually required modifications of an existing SAP system are given. These
considerations need to be taken into account, before the SAP system then can be registered
with SAP LVM. The definition of the new system in the SAP LVM context is described in
chapter 4.4
3.4.5.1 Filesystem considerations for the SAP Systems (SAP System Copy)
The two tables below illustrate two different storage layouts for a SAP System based on DB2
UDB. The first example (Table 11) includes a simple layout, having a few different
filesystems only and just separating between SAP application and database filesystems. The
filesystem layout fullfills the requirements for the relocation and SAP system copy scenarios
with SAP LVM.
Filesystem Volume Group Content
/db2 vg01 Contains the instance directory /db2/db2<sid> and all other database related directories. All DB2 database specific data is stored in a directory structure below the filesystem /db2
/var/db2 vg01 Filesystem for DB2 global registry
/home/<sid>adm vg00 Filesystem for home directory of SAP admin user
/sapmnt/<SID> vg00 Filesystem for SAP profile, SAP work/interface data, and SAP Kernel source
/usr/sap/<SID> vg00 Filesystem for SAP Instance directories
/usr/sap/DAA/SMDA97 vg00 Filesystem for SMD agent
Table 11 Minimum filesystem layout (DB2 UDB)
In the second example (Table 12) the filesystem layout is more complex: The database layout
is split-up into different volume groups. The filesystem layout fulfils the requirements for the
relocation and SAP system copy scenarios with SAP LVM as well as the capability for
performing SnapShot/FlashCopy backups for the database with IBM Tivoli Storage
FlashCopy manager:
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 66 of 169
Filesystem Volume Group Content
/db2 vg01 Contains the instance directory /db2/db2<sid> and all other database related directories. The instance directory /db2/db2<sid> is *not* created as a filesystem, as SAP LVM V1.0 then is not able to handle that during a SAP System Copy
/db2/<SID>/db2dump vg01 Own filesystem for the db2diag.log for convenience
/db2/<SID>/log_archive vg01 Separate filesystem for DB2 inactive logs
/db2/<SID>/log_dir vg03 Separate filesystem for DB2 active logs: In case SnapShot backups are regularly taken and should be used for a fast restore, this filesystem *must* be separate from database data.
/db2/<SID>/db2<sid> vg02 Contains the database directory. In case that regular SnapShot backups are intended for the database, the content needs to be included in the SnapShot image together with the DB2 database containers/ storage pathes
/db2/<SID>/sapdataX vg02 Contains the database containers / storage pathes
/home/<sid>adm vg00 Filesystem for home directory of SAP admin user
/sapmnt/<SID> vg00 Filesystem for SAP profile, SAP work/interface data, and SAP Kernel source
/usr/sap/<SID> vg00 Filesystem for SAP Instance directories
/usr/sap/DAA/SMDA97 vg00 Filesystem for SMD agent
/var/db2 vg01 Filesystem for DB2 global registry
Table 12 Extended filesystem layout (DB2 UDB)
If such FlashCopy backups for the database are a requirement, then a layout similar to the
second example has to be implemented.
3.4.5.2 Install a new SAP system
For the SAP system installation, select one resource node of the SAP LVM environment that
has no active SAP services.
The filesystem layout needs to be compliant to chapter 3.4.5.1
For the SAP system installation the virtual IP addresses for the database service and
the central instance/ central services need to be present. Define the virtual IP addresses
to the network adapter using the “ifconfig” command. These addresses must not be
defined as persistent, as the virtual IP addresses for the SAP services later on will be
managed exclusively by SAP LVM during the prepare/unprepare workflows.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 67 of 169
# ifconfig <nw interface> alias <virtual IP CI> netmask <netmask>
# ifconfig <nw interface> alias <virtual IP DB> netmask <netmask>
During the SAP system installation later on, use the SAPINST_USE_HOSTNAME
parameter to specify the virtual hostname for the service to be installed.
Users and Groups have to be defined centrally in the LDAP environment. As SAPinst
currently cannot handle LDAP users properly, create users and groups for SAP and
DB administration manually before the SAP system installation.
The detailled installation procedure needs to follow all the steps as described in the
SAP installation documentation, and the according SAP notes.
3.4.5.3 Adapt an already existing SAP system
File System Layout
The filesystem layout needs to be compliant to chapter 3.4.5.1. So it may be required
to copy or move the content of the standard installation directories into the SAP LVM
compliant storage layout.
Virtual IP Address and hostname
Within the SAP LVM environment the database server and the central instance/
central services can be operated separately on different nodes. Two different virtual IP
addresses are required for them. If one of the virtual IP addresses was defined as base
address of a network adapter, redefine the network adapter to a new address. Define
the virtual IP addresses as IP alias to the network adapter using the ifconfig command:
# ifconfig <nw interface> alias <virtual IP CI> netmask <netmask>
# ifconfig <nw interface> alias <virtual IP DB> netmask <netmask>
If the hostname for the SAP instance is changed to a new virtual hostname, make sure
to adapt all occurrences of it in the filename and contents of DEFAULT, START, and
instance profile. Create and update the following parameters in the Instance Profile to
the virtual hostname too.
SAPLOCALHOST = <virtual hostname CI>
SAPLOCALHOSTFULL = <full qualified virtual hostname CI>
icm/host_name_full = <full qualified virtual hostname CI>
For example, in the default profile DEFAULT.PFL, the SAPDBHOST has to reflect
the virtual IP label of the DB server, and the virtual IP label of the CI will be part of
the mshost, vbname, enqname, btcname, and gateway parameters:
SAPDBHOST = c01db01
rdisp/mshost = c01ci01
rdisp/sna_gateway = c01ci01
rdisp/sna_gw_service = sapgw10
rdisp/vbname = c01ci01_C01_10
rdisp/enqname = c01ci01_C01_10
rdisp/btcname = c01ci01_C01_10
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 68 of 169
Dependent on your individual environment, you may have to adapt settings in further
interfaces / SAP systems too.
(Re-) create the Users in LDAP
The administrative users for Database and SAP system have to be created with the
same numeric UID in the LDAP directory. If the UID for the users violate uniqueness
within the SAP LVM environment, the ownership of all files needs to be changed to
an unique UID.
Adapt user environment files
During Shell startup, several files are sourced to get the proper environment for SAP
admin and DB admin users. During installation time, these files are created containing
the hostname within their name. Rename them in the home directory of both SAP
admin and DB admin user.
# mv .sapenv_<virtual hostname>.sh .sapenv.sh
# mv .apoenv_<virtual hostname>.sh .apoenv.sh
# mv .dbenv_<virtual hostname>.sh .dbenv.sh
# mv .sapenv_<virtual hostname>.csh .sapenv.csh
# mv .apoenv_<virtual hostname>.csh .apoenv.csh
# mv .dbenv_<virtual hostname>.csh .dbenv.csh
You also need to check the shell start profiles (.cshrc) that it sources all required
environment files correctly. Each entry should look like
# RDBMS environment
if ( -e $HOME/.dbenv_`hostname`.csh ) then
source $HOME/.dbenv_`hostname`.csh
else if ( -e $HOME/.dbenv.csh ) then
source $HOME/.dbenv.csh
endif
This means the shell first searches for a file containing the hostname, will not find it
and then uses the corresponding file without the hostname. Sometimes the “else if”
clause is missing and you need to create it manually.
Adapt database configuration files to virtual IP address
o ORACLE: Adapt configuration files tnsnames.ora and listener.ora (ADDRESS =
(COMMUNITY = SAP.WORLD)
(PROTOCOL = TCP)
(HOST = c01db01)
(PORT = 1537)
)
The stanzas in tnsnames.ora and listener.ora have to reflect the virtual
hostname of the DB instance. The communication port should be unique within
the environment.
o DB2 UDB: Update db2cli.ini to the virtual DB server address
; Comment lines start with a semi-colon.
[C01]
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 69 of 169
Database=C01
Protocol=tcpip
Hostname=c01db01
Servicename=5912
[COMMON]
Diagpath=/usr/sap/C01/SYS/global/db6/db2dump
The “Hostname” in the db2cli.ini file needs to reflect the virtual hostname of
the DB instance. The service name reflects the communication port and should
be unique within the environment.
Perform a test connection to the database: Start the database (and, in case of DB
ORACLE, the listener process). As user <sid>adm, perform a test connection to the
database using R3trans:
c01adm> R3trans –d –v
This is R3trans version 6.22 (release 720 - 03.06.11 - 17:57:00).
unicode enabled version
R3trans finished (0000).
R3trans should return 0 for a successful DB connection.
3.4.5.4 Additional SAP profile parameters
See also SAP Note 1438774 - New profile parameter system/uuid and system/description:
SAP LVM will verify the system/uuid profile parameter against its configuration prior to
any operation (start, stop, relocate) on any instance of a (cloned) system, to ensured that the
operation is triggered on the correct instances. The system/description profile parameter
contains a verbal description of the system.
The parameter can be added to the SAP system later on also during the LVM configuration
phase. However, the SAP system then needs to be restarted to activate the parameter.
3.4.5.5 SAP license considerations
The license key for an SAP system is checked on the resource on which the message server is
running. If the message server is moved during a relocation event to another resource, a
license key for the new environment may be required. Request and import in advance license
keys for all the systems on which the message server may run. The license key depends on the
hardware key: The hardware key is different for different physical servers. (All the LPARs
within one physical IBM Power server have the same hardware key).
The license key is checked during the (re-) start of the message server:
If the relocation is initialized virtualization-based (Live-Partition Mobility), then the
message server keeps running in the same state as before.
o Even if there is no license key for the target server available the SAP system
continuous its operation. The attempt to restart the SAP system on the target
server while having no valid license for the hardware key of the target server
would then fail.
If the relocation is initialized adaptive-based (stop and unprepare on the source
resource/ prepare and start on the target resource), then the message server is restarted
on the target resource, and a valid license key for the hardware key of the target server
is required.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 70 of 169
SAP has introduced a new license key feature called Flexible License Mechanism to simplify
the administration of SAP license keys in system landscapes where the message server can
move between several physical servers. With that method the license key is no longer tied to
the hardware key of the message server. The flexible license mechanism uses a separate ID
generator, which creates a unique network ID for each message server. This network ID is
hardware independent and therefore it is possible to move the message server to a different
host and retain the unique ID. It is possible to configure multiple ID generators to eliminate
any potential single point of failure.
Refer to the current SAP NetWeaver Help documentation for a detailed description and
instructions of how to set up this new Flexible License Mechanism:
SAP NetWeaver 7.0 EHP2 and SAP NetWeaver 7.3
3.4.6 Install SAP LVM Prerequisites on the management node
3.4.6.1 XIV XCLI
The XIV CLI is used for the communication of LPARs with the storage system. It needs to
run each of the management as well as on the managed LPARs.
It is a software which needs to be installed separately
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 71 of 169
4 Initial LVM Configuration
4.1 Engine Settings
The SAP LVM engine settings are normally configured when running the Initial
Configuration Wizard. If you need to change some of the settings as for example the default
user name or password for the Host Agents, you can do this under Setup General Settings
LVM Engine Settings.
Figure 21 LVM Engine Settings
4.2 Storage Manager Configuration
The IBM Storage Manager settings (IBM XIV Storage System, IBM Tivoli Storage
FlashCopy Manager) are defined via the Configuration Infrastructure Storage Manager
Panel. The start panel is illustrated in Figure 22: Navigate to this panel, and press “Add” to
define a Storage Manager in SAP LVM.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 72 of 169
Figure 22 Storage Manager Configuration entry panel
In the next screen (Figure 23) the Storage Manager Type can be selected. Select Vendor
“IBM”, Product “IBM FCM Adapter” and click “Next”.
Figure 23 Storage Manager Configuration: Select Storage Manager type
In the next panel (Figure 24) the IBM Tivoli Storage FlashCopy Manager related settings are
specified (See also chapter 3.3.2, as a lot of these parameters need to be provided from the
FCM installation):
The “FCM/ XIV CLI Host Name” field specifies the node on which the IBM Tivoli Storage
FlashCopy Manager Management Agent is installed: The Management Agent (acsd) controls
the backup flow and mediates between the other agents running on the “resource” LPARs.
The Management Agent also provides access to the snapshot backup repository, which
contains information about the valid snapshot backups and their relationships to snapshot
capable storage devices. During the SAP system cloning / SAP system copy process SAP
LVM starts the Management Agent on the “FCM CLI Host Name” node.
The “FCM CLI Path” specifies the working directory for the IBM Tivoli Storage FlashCopy
Manager on all of the nodes. This directory reflects the path which was specified during the
installation customization of the IBM Tivoli Storage FlashCopy Manager (FlashCopy backup
for generic applications in “setup_gen.sh”)
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 73 of 169
The “IBM FCM template” reflects the filename of the profile template located in the “FCM
CLI Path”.
Figure 24 Storage Manager Configuration: Define FlashCopy Manager settings
When all required entries are defined, press “Test Configuration”. SAP LVM will initiate a
communication test to FlashCopy Manager and the storage subsystem CLI. The configuration
test should return “Connection successful: SAP LVM successfully gets data from storage
managers through IBM Tivoli Storage FlashCopy Manager”.
After a successful connection test, select “Next” to get to a user and password configuration
screen. Here you can enter a user and password for the configured storage subsystem. SAP
LVM stores the storage subsystem password internally: if the user or password is changed in
the storage subsystem, you have to update the LVM Storage Manager configuration too.
Figure 25 Storage Manager Configuration: user and password
Click “Next”. The summary screen appears; the SAP LVM Storage Manager definitions for
IBM XIV and IBM Tivoli Storage FlashCopy Manager are complete.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 74 of 169
Figure 26 Storage Manager Configuration Summary
4.3 Discovery of resources (LPARs)
To discover a new resource in SAP LVM, the SAP Host Agent must be installed and running
on the target partition. Select Configuration Resources Add Resources to get to the “Find
Resources” panel. Here you have various options to perform a resource detection, in this
example we select “Detect Using Host and Instance Agent” as Source and “Resources only”
under Find.
Figure 27 Add resource – find Resources
Enter the host name of the partition you want to discover. You can use the default credentials
for the SAP Host Agent user id as configured in the engine settings or choose other
authentication types like specific user/password settings or a X.509 client certificate. Click the
Detect button and you should get a message that new resources are found.
On the next panel select the resource pool you want to assign this partition to and click the
next button.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 75 of 169
Figure 28 Add resource - assign pool
Click the Save button on the summary screen to add this newly discovered resource.
The next step is to change a number of properties for this new resource. Select the new host
on the Resources Overview panel and click “Edit”. The “Basic Configuration” step should
already have the correct settings, click Next to get to the Resource Properties panel.
Figure 29 Add resource – define properties
You have to select the AC-Enabled flag and add a network interface that is used to bind the
virtual IP address of a SAP service. You also have to provide a capacity figure (SAPS
provided) for this resource. If the partition should be used as a target resource for cloning and
system copy operations, then you need to select the “Isolation Ready” flag too. Press the
“Save” button to store the changes.
4.4 Service Configuration/ Discovery in SAP LVM
The discovery for a new system is done in several steps. In the “base discovery” the new SAP
system is discovered and added to the SAP LVM configuration. Then the configuration of the
newly discovered services is completed. This includes the detailed storage configuration for
database and for the central instance.
4.4.1.1 Discovery for a new SAP system
The base discovery of new SAP systems is started out of the in the tab “Services” in the
“Configuration” view. A new SAP System is discovered by using the “Add Services” button
(Figure 30):
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 76 of 169
Figure 30 Add new SAP services
The new system needs to meet the following prerequisites:
the resource hosting the LPAR is already discovered in SAP LVM
the SAP Host Agent is installed and active on the resource
the SAP system is installed in an “adaptive” configuration:
o virtual hostnames are defined for both database and central instance
o the storage layout supports separation of database and central instance
the SAP system is active
all the user credentials (e.g. <sid>adm) and their passwords are known to the SAP
LVM admin
For older SAP releases (without having enhanced security measures implemented) the SAP
Host Agent is able to discover both database and SAP instances running on the LPAR. For
higher SAP releases the SAP Host Agent may not be able to detect the SAP instance and all
the details. Due to enhanced security settings, the “sapadm” user is not authorized any more to
retrieve all the instance parameters during the discovery. Discovery of the SAP instance
requires to contact the instance agent using the <sid>adm credentials to retrieve all relevant
instance data. The “Expert Mode” in the configuration panel has to be used.
Specify the connection data to the SAP Host Agent (IP address of the resource LPAR), as
well as the connection details to the Instance agent (Figure 31):
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 77 of 169
Figure 31 Expert Mode: Detect new services using host agent and instance agent
The discovery is then started by pressing the “Detect Button”. The Database and SAP Central
Service services are identified, and the data is integrated to the “Add Services” view. In this
view, the services are assigned to the appropriate resource pool and the SAP system is
described with additional details (Figure 32).
Figure 32 Add Services View
Select a target resource pool using the selection menu in the “Pool for New Systems”. When
the selection dialog screen appears (Figure 33), select the appropriate pool for the new
system.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 78 of 169
Figure 33 Selection of the Resource Pool
After the “Resource Pool” was selected, the detailed description for the SAP System needs to
be specified (Figure 34):
Figure 34 Enter SAP System Description
Base configuration for the new system is done,
Figure 35 Add Services sub-menu
and finally the new system data is saved (Figure 36).
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 79 of 169
Figure 36 Summary view for the new system
Base system discovery is complete, and now configuration details for the SAP system, and
specifics for the database and the SAP central instance are started. These additional
configuration steps are required to enable the system for SAP LVM.
4.4.1.2 Base configuration
Select the “System Details” view for the new SAP System, and press the “Edit” button to
define all the detailed settings (Figure 37):
Figure 37 SAP System configuration
The customization is started (as indicated in Figure 38) and will run through the steps
Basic Configuration
ACM Settings
Network Isolation
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 80 of 169
Figure 38 SAP System configuration: general settings
Basic Configuration
The Provisioning Use-Cases allowed for the SAP System are selected in the “Basic
Configuration” (Figure 39). These include:
Cloning
Copying
Renaming
Application Server (Un-) Installation
If either copying or renaming are check-marked, then a RFC destination (target client(s) for
post copy automation) needs to be defined. The customization panel for a RFC destination
will open automatically by checking the “Configure RFC Destination” flag.
Figure 39 SAP System configuration. Overall system details
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 81 of 169
ACM Settings
ACM was not evaluated in the test installation (Figure 40). Click “Next” to go to the next
step.
Figure 40 SAP system configuration: ACM
Network Isolation
In the network isolation step, all network connections which are still allowed during the
“Network fencing” step are configured. The settings can still be adapted during the SAP
System Copy/System cloning definition steps. The settings defined here will appear as default
allowed connections during the System Copy / System Cloning step (Figure 41).
Figure 41 SAP System configuration: Network fencing defaults
As examples, the network connections to “siccserv2” (DNS server for the test environment,
Figure 42), and the “ssh” protocol (Figure 43) were explicitly added as allowed per default to
the network fencing configuration.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 82 of 169
Figure 42 SAP System configuration: Add allowed communications
Figure 43 SAP System configuration: Add allowed communications
Complete with “Save” afterwards.
4.4.1.3 DB Configuration
After the base configuration for the SAP system is done, the next step will configure details
for the SAP database service. In the “Services” tab in the “Configuration” view expand the
new SAP System, and select the database. Press the “Edit” button (Figure 44).
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 83 of 169
Figure 44 DB Configuration: Entry screen
Configuration of the database will be done in the order
Basic Configuration
Service Properties
Mount Points
Basic Configuration
Keep (or change) the settings in the basic configuration view (Figure 45), afterwards click
“Next”:
Figure 45 DB Configuration: Basic configuration
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 84 of 169
Service Properties
Check-mark the box “AC-Enabled” in the Adaptive Enablement session of the screen (Figure
46). Select the appropriate Network. Additional values will be shown in the “Requirements”
section of the screen. Enter the required SAPS and required Memory values for the service.
These attributes will be used during Start- and Relocate actions to determine which target
resource is a candidate for the service. The definition of the target resource needs to satisfy
the conditions defined here in this dialog.
Select the allowed OS levels for the service, and press “Next”
Figure 46 DB Configuration: Add resource details
Mount Points
In this step, the storage configuration for the system is defined. This step will depend on the
storage provider. For the IBM storage environment, some data needs to be gathered on the
resource LPAR. In detail the required steps are:
Create a file containing all the filesystems related to the database of the SAP System
to be discovered on the LPAR
Use the script ibm_lvm_mgmt.ksh from the IBM Storage library to create a XML
structure referencing all required storage information for the database
Copy this information and paste it into the LVM configuration screen
Un-check the flag “OS Managed Mounts /Automounter)”. The screen changes its
appearance. Select “Extract Mount Points” to enter the XML data gathered on the resource
(Figure 47).
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 85 of 169
Figure 47 SAP LVM Storage Definition Screen
Logon to the LPAR hosting the SAP system and create a list for the database filesystems.
Figure 48 illustrates a sample list of database filesystems stored in a config file.
Figure 48 Create a list of the DB filesystems
This list is then used as input data for the ibm_lvm_mgmt.ksh script. Additional arguments
for the script are the name of the XIV Storage System and the “mountconfig” keyword
(Figure 49). Specify this as arguments and execute the ibm_lvm_mgmt.ksh script:
/usr/sap/hostctrl/exe/operations.d/IBM/ibm_lvm_mgmt.ksh mountconfig [filename] [XIV
name]
The execution of the script generates a XML structure for the mount data to standard out:
Each filesystem from the list will get an own mount stanza in the XML output.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 86 of 169
Figure 49 Create XML-File for mountconfig
Copy the XML structure from the screen of the LPAR (Figure 49) and paste it into the SAP
LVM “Input for Extraction Window” (Figure 50):
Figure 50 Paste Mountconfig in LVM storage config
Then press the “Extract Mount” Button in the SAP LVM screen: SAP LVM transfers the
XML data into its filesystem data model (Figure 51):
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 87 of 169
Figure 51 Extract Mount Points
All the mount points for the database are integrated into the SAP LVM configuration, and the
database service configuration can be saved (Button “Save”, Figure 52).
Figure 52 Final LVM Mount point configuration
Database configuration is complete.
4.4.1.4 Configuration of SAP Central Instance
After the configuration of the database instance is done, the next step will configure details
for the SAP Central Instance. This is similar to the “database configuration”.
In the “Services” tab in the “Configuration” view expand the new SAP System, and select the
Central Instance. Press the “Edit” button (Figure 53):
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 88 of 169
Figure 53 Service Details Central Instance
Configuration of the database will be done in the order
Basic Configuration
Service Properties
Mount Points
Basic Configuration
Keep (or change) the settings in the basic configuration view (Figure 54), afterwards click
“Next”:
Figure 54 Basic Configuration for Central Instance
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 89 of 169
Service Properties
Check-mark the box “AC-Enabled” in the Adaptive Enablement session of the screen (Figure
55). Select the appropriate Network. Additional values will be shown in the “Requirements”
section of the screen: Enter the required SAPS and required Memory values for the service.
These attributes will be used during Start- and Relocate operations to determine which target
resource is a candidate for the service. The definition of the target resource needs to satisfy
the conditions defined here in this dialog.
Select the allowed OS levels for the service, and press “Next”.
Figure 55 Service properties for Central Instance
Mount Points
In this step, the storage configuration for the system is defined. This step will depend on the
storage provider. For the IBM storage environment, some data needs to be gathered on the
resource LPAR. In detail the required steps are:
Create a file containing all the filesystems related to the central instance of the SAP
System to be discovered on the LPAR
Use the script ibm_lvm_mgmt.ksh from the IBM Storage library to create a XML
structure referencing all required storage information for the central instance
Copy this information and paste it into the LVM configuration screen
Un-check the flag “OS Managed Mounts /Automounter)”. The screen changes its appearance.
Select “Extract Mount Points” to enter the XML data gathered on the resource (Figure 56):
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 90 of 169
Figure 56 Mount Config for Central Instance
Logon to the LPAR hosting the SAP system and create a list for the central instance
filesystems. Figure 57 illustrates a sample list of database filesystems stored in a config file.
Figure 57 Create List of filesystems for central instance
This list is then used as input data for the ibm_lvm_mgmt.ksh script. Additional arguments
for the script are the name of the XIV Storage System and the “mountconfig” keyword
(Figure 58). The execution of the script generates a XML structure for the mount data to
standard out.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 91 of 169
Figure 58 Extract Mount Config for Central Instance
Copy the XML structure from the screen of the LPAR (Figure 58) and paste it into the SAP
LVM “Input for Extraction Window” (Figure 59):
Figure 59 Transfer Mount Config to SAP LVM
All the mount points for the database are integrated into the SAP LVM configuration, and the
database service configuration can be saved (Button “Save”, Figure 60).
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 92 of 169
Figure 60 Apply Mount Config to SAP LVM
SAP Central Instance configuration is complete
4.5 Setup for the environment in IBM Systems Director
4.5.1 Discover the OS deployment components in IBM Systems Director
Execution of image deployment requires activities on the Hardware Management Console
(HMC), the virtual I/O server(s), and the NIM server. These components need to be
discovered in IBM Systems Director first. A OS mksysb image then needs to be captured
from the NIM server. New deployments using this image then can be started.
All the screenshots shown here are based on IBM Systems Director 6.2.1 and VMControl
2.3.1: So the screens with IBM Systems Director 6.3. / VMControl 2.4 may look slightly
different. The overall procedure however is still valid.
4.5.1.1 Discover the HMC
Figure 61 illustrates the HMC discovery steps: In the Systems Director GUI go to:
Inventory System Discovery Enter the IP address of the HMC. Then choose
“Discover Now”
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 93 of 169
Figure 61 HMC Discovery in IBM Systems Director
Request access to the HMC (Figure 62)
Choose the entry reflecting the operating system Enter the HMC‟s user id and
password. Then choose “Request Access”
Figure 62 Request access to the HMC
Collect HMC inventory (Figure 63)
In the Systems Director GUI go to:
Inventory View and Collect Inventory choose „Browse‟ in the target section
select the two HMC entries choose “Add” choose “OK”.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 94 of 169
Figure 63 HMC inventory collection
An inventory collection is started in the background. When the collection job is
completed, you can view the discovered inventory (Figure 64):
Figure 64 View HMC inventory
4.5.1.2 Discover the VIO server(s)
Discovery of the VIO Server(s) follows a similar sequence as illustrated before for
discovery of the HMC. In the Systems Director GUI go to:
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 95 of 169
Inventory System Discovery Enter the IP address of the VIO server choose
“Discover Now”
Request access to the VIO server
Choose the entry reflecting the operating system Enter the VIO server‟s user id and
password choose “Request Access”
Collect VIO server inventory
In the Systems Director GUI go to:
Inventory View and Collect Inventory choose „Browse‟ in the target section
select the VIO server entry choose “Add” choose “OK”.
An inventory collection is started in the background. When the collection job is
completed, you can view the discovered inventory (Figure 65):
Figure 65 View VIO server inventory
4.5.1.3 Discover the NIM server
Discovery of the NIM Server LPAR follows a similar sequence as illustrated before
for discovery of the HMC. In the Systems Director GUI go to:
Inventory System Discovery Enter the IP address of the NIM server choose
“Discover Now”
Request access to the NIM server
Choose the entry reflecting the operating system Enter the NIM server‟s user id
and password choose „Request Access‟
Collect NIM server inventory
In the Systems Director GUI go to:
Inventory View and Collect Inventory choose “Browse” in the target section
select the NIM server entry choose “Add” choose “OK”.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 96 of 169
An inventory collection is started in the background. When the collection job is
completed, you can view the discovered inventory (Figure 66):
Figure 66 View NIM server inventory
4.5.2 Setup VMControl
VMControl can be customized when HMC, VIOS and NIM Server are discovered.
VMControl interfaces with the NIM server via an agent: This agent needs to be installed on
the NIM server LPAR first.
4.5.2.1 Install NIM server agent on NIM server
Important:
The following agents have to be installed on the NIM server to ensure a successful
connection to Systems Director. The agent version correlates to the VMControl
version in place:
root> lslpp -l | grep -i agent
DirectorCommonAgent 6.2.1.3 COMMITTED All required files of
Director Common Agent, including JRE
DirectorPlatformAgent 6.2.1.2 COMMITTED Director Platform Agent for
Tivoli_Management_Agent.client.rte
cas.agent 1.4.2.32 COMMITTED Common Agent Services Agent
If you find other versions installed on the LPAR the agents may need to be removed
completely before a new version can be installed: Update of the agents doesn‟t work in
all cases. To ensure a clean discovery, remove the existing NIM server settings from
the existing Systems Director Database first.
In the Systems Director GUI go to:
Navigate Resources Choose “All Systems” Search and right click the NIM
server Choose “Remove”
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 97 of 169
Figure 67 Remove NIM server
(Re-) Discover the NIM server
In the Systems Director GUI go to (Figure 68):
Inventory Discover System Enter the IP Address of the NIM server Click
“Search”
Figure 68 Discover NIM server
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 98 of 169
Request Access to the NIM server
In the Systems Director GUI go to:
Navigate Resources Choose “All Systems” Search and right click the NIM
server Choose “Request Access”
Collect inventory
In the Systems Director GUI go to:
Inventory View and Collect Inventory Select or Browse the target system
Click “Collect Inventory”
Figure 69 Collect Inventory of NIM server
Install the NIM subagent
In the Systems Director GUI go to:
Welcome VMControl Enterprise Edition In the “Common tasks” table, click on
“Install agents” (Figure 70) Click “Next” Choose “Common Agent Subagent
Packages” Choose “CommonAgentSubagent_VMControl_NIM-2.x.x.x”
(according to the current VMControl version) and click “Add” Choose the NIM
server and click “Add” Click “Finish”
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 99 of 169
Figure 70 Installation of NIM subagent on the NIM server with VMControl
Collect the inventory once again
In the Systems Director GUI go to:
Inventory View and Collect Inventory Select or Browse the target system
Click “Collect Inventory”
4.5.2.2 Prepare NFS filesystem for the OS images on the NIM server
On the NIM server a NFS filesystem is created for the OS images. During the capture
activities, this filesystem is mounted via NFS on the NIM client: It will be the target directory
for the “mksysb” image. In smitty, export the related directory, in our case it is
/export/mksysb (Figure 71):
Change Attributes of an Exported Directory
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[TOP] [Entry Fields]
* Pathname of directory to export /export/mksysb
* Version of exported directory to be changed 3
Anonymous UID [-2]
Public filesystem? [no] +
* Change export now, system restart or both both +
Pathname of alternate exports file []
Allow access by NFS versions [3]
External name of directory (NFS V4 access only) []
Referral locations (NFS V4 access only) []
Replica locations []
Ensure primary hostname in replica list yes +
Allow delegation? []
Scatter none +
Security method 1 [sys,krb5p,krb5i,krb5,dh] +
Mode to export directory [read-mostly] +
Hostname list. If exported read-mostly [siccps99]
Hosts & netgroups allowed client access []
Hosts allowed root access [siccps99]
Security method 2 [] +
Figure 71 Export mksysb directory
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 100 of 169
4.5.2.3 Create mksysb backup on the target LPAR
On the LPAR to be backed up, mount the NFS share mount siccnim2:/export/mksysb /mnt
On the LPAR to be backed up, enter following command: mksysb -i /mnt/sap_lvm_generic.mksysb
4.5.2.4 Import “mksysb” as Virtual Appliance into VMControl
In the Systems Director CLI, enter: root> smcli captureva -r <resource number> -F repos:<location of the
mksysb> -n <image name> -A "cpushare=<number of physical
CPUs>,memsize=<memory size>,disksize=<disk size>"
Example: root> smcli captureva -r 74091 -F
repos://export/mksysb/sap_lvm_generic.mksysb -n "SAP LVM generic" -A
"cpushare=0.1,memsize=4096,disksize=20000000000"
Tip: to get the resource number, enter: root> smcli lsrepos –o
Tip: to delete an image from systems director, enter root> smcli rmva -V <ApplianceID>
where the value of the ApplianceID can be obtained using: root> smcli lsva –l
4.5.2.5 Deployment test for the Virtual Appliance
When the image is successfully captured in VMControl, it can be deployed now:
In the Systems Director GUI go to:
Welcome VMControl Enterprise Edition Change to the „Virtual Appliances‟ tab
Select the virtual appliance you want to deploy Click „Deploy Virtual Appliance‟
Follow the wizard Click „Finish‟
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 101 of 169
Figure 72 Trigger of Virtual Appliance Deployment
4.5.3 Setup of IBM Systems Director Security
You can use IBM Systems Director administrator users for integration with SAP LVM. Some
virtualization use cases (OS provisioning and LPAR relocation using Live Partition Mobility)
require administration credentials. For virtualization monitoring and management, you may
use restricted credentials. IBM Systems Director administrators may allow monitoring and
management of particular elements only (e.g. of certain physical servers, LPARs, operating
systems, etc.) and thus restrict a view of SAP LVM users. Below we describe how to extend
IBM Systems Director predefined users so that you may use them for integration with SAP
LVM.
Log in to operating system where IBM Systems Director is running on and create a group
(e.g. using mkgroup) and a user (e.g. using mkuser) that you want to use for integration with
SAP LVM. The user must belong to the group you created previously.
Log in to IBM Systems Director console as administrator and navigate to „Users and
Groups“. Press „Authorize Groups“ and authorize the group created in previous step. All the
users that belong to this group will appear as valid IBM Systems Director users with minimal
access rights.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 102 of 169
Figure 73 Setup of IBM Systems Director Security – Authorize Groups
Navigate to „Roles“ and create a copy of predefined roles „SMManager“ and „SMUser“ by
clicking on „Create Like...“ The copy of „SMManager“ (named „LVM_Manager“ in our
example) is a role for SAP LVM management authorization. The copy of „SMUser“ (named
„LVM_User“ in our example) is a role for SAP LVM monitoring authorization.
Figure 74 Setup of IBM Systems Director Security – Copy Predefined Roles
Edit both of the copied roles and extend them by assignment of additional permissions.
Extend the copied SMUser Role with following permissions:
General Create Group, General Edit (Description: Provides an editor for
changing an existing group)
System Status and Health Monitors
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 103 of 169
Extend the copied SMManagement Role with following permissions:
Power On/Off Power Off Now, Power On, Shutdown and Power Off
Figure 75 Setup of IBM Systems Director Security – Modify Copied Roles
Navigate to “Resource Explorer” and create two static groups that include any types of
resources. The first group (monitoring) should include physical machines, virtual appliances,
storage pools and virtual servers that will be visible in SAP LVM. The second group
(management) should include only the virtual servers that will be manageable through the
SAP LVM. If possible, use IBM Systems Director administration credentials for first
integration with SAP LVM. In that case SAP LVM virtualization adapter for IBM Systems
Director would create a list of dynamic groups containing all the resources that may be shown
in SAP LVM. You can then navigate to these dynamic groups and add the resources you are
interested into the static groups.
Figure 76 Setup of IBM Systems Director Security – Define Resource Groups
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 104 of 169
Navigate back to „Users and Groups“ and choose the user that you would like to configure.
Click „Assign Role...“ in order to step into the assignment dialog.
Figure 77 Setup of IBM Systems Director Security – Assign Roles to User
Associate the created groups with created roles, whereby the copy of SMUser role becomes
associated with monitoring group and the copy of SMManagement role becomes associated
with management group. Add these groups / roles combinations to the list of “Assigned
Roles”.
Figure 78 Setup of IBM Systems Director Security – Assign Groups and Roles to User
After finishing this configuration you may use the configured user for integration of IBM
Systems Director with SAP LVM.
4.6 Virtualization Manager Configuration in SAP LVM
The IBM Virtualization Manager settings (IBM Systems Director/ VMControl) are defined
via the Configuration Virtualization Managers Panel. SAP Note 1728222 – Configuration
of IBM Systems Director VMControl for SAP LVM describes further background
information about IBM Systems Director VMControl integration with SAP LVM for this
step. The start panel is illustrated in Figure 79. Navigate to this panel, and press “Add” to
define a new virtualization manager in SAP LVM:
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 105 of 169
Figure 79 Virtualization Manager Configuration
Select “IBM Systems Director VMControl” as platform (Figure 80):
Figure 80 Virtualization Manager: IBM Systems Director VMControl
Then enter all the relevant values for the virtualization manager.
As Basic properties provide a label in SAP LVM, the user name and password of the
“administration” user that was created for the IBM Systems Director in chapter 4.5, and the
URL pointing to the IBM Systems Director Management server.
In the Additional Properties provide the Systems Director Port (Default is 8422), the
monitoring interval (how often SAP LVM tries to retrieve data from IBM Systems Director),
and the Storage Connection “NPIV” for LPAR deployments1
Press “Test Connection” to validate all the settings.
1 The deployment of LPARs end-to-end (LPAR creation, storage assignment, and OS installation) requires the
Storage Control plug-in and customizing for all data sources (SAN switches, Storage Systems).
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 106 of 169
Figure 81 Virtualization Manager properties
Press “Next” to proceed to the next screen (Figure 82 ): Press “Save” to confirm all settings.
Figure 82 Virtualization manager summary
4.7 Additional Configuration Steps
4.7.1 Network definition in SAP LVM
As a prerequisite for SAP System Cloning / Copy / Refresh, the network definitions need to
be completed in SAP LVM. The configuration dialog is started via
Configuration Infrastructure Network
Press “Add” to define a new network (Figure 83):
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 107 of 169
1
2
3
4
5
1
2
3
4
5
Figure 83 Configure Network
In the first panel “Basic Configuration” settings are specified. They includes Name, Subnet
Mask, and the Broadcast Address of the network.
In the next step, “Advanced Configuration” settings are specified. They include Gateway
Server, DNS Zone, Reverse DNS Zone and Update IP Range.
The “Update IP Range” can restrict the range for new IP addresses that are provisioned out of
SAP LVM if automatic DNS update is configured. If the "Auto Assign IP Address" option is
selected in the "Host Names" step during a clone or copy, a random free IP address is selected
from within this range. If an IP address is specified manually, it must still be within this
range.
IP addresses are expected in dotted quad notation.
Multiple IP ranges can be separated by ";"
The lower and upper parts of a range must be present in each range and are separated
by ";" or ":"
Ranges are calculated by using the IPs as numbers.
Example: The range 12.48.2.3:12.48.4.2 includes 12.48.2.4, 12.48.2.255, 12.48.3.0,
12.48.3.255, 12.48.4.0, 12.48.4.1, and 12.48.4.2.
An IP range must not be a single IP. However, a single IP can be represented as a
range. Example: 12.48.2.38 as 12.48.2.38:12.48.2.38
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 108 of 169
4.7.2 Configuration for SAP System Copy
During a SAP System Copy and a SAP System Rename activity SAP LVM invokes the
SAPInst rename tool on the target LPAR to change the SAP system ID of the system. SAP
LVM expects that the SAPInst tool is available in an installation folder which can be accessed
on the target LPAR: In the test environment a central NFS server is used as repository, and
the directories are mounted to all the (target) LPARs. This path is declared in Configuration
Extended Settings System and Provisioning for each different target OS type:
Figure 84 illustrates the configuration settings:
Figure 84 Customizing SAPInst rename path
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 109 of 169
5 Detailed Description / Execution of Use Cases
5.1 LVM Standard Edition Use Cases
This chapter describes the use cases which are included in the standard version of SAP LVM.
Basically the SAP LVM Standard Edition allows to Start, Stop, and Relocate SAP Systems.
5.1.1 Description of the LVM Standard Edition Use Cases
This chapter introduces the use cases of LVM Standard edition. Following use cases are
possible:
Prepare / unprepare SAP instances and systems
Start / stop SAP instances and systems
Relocate SAP instances and systems
Start / stop resources
Monitoring
5.1.1.1 Prepare / unprepare SAP instances and systems
The Standard Edition of SAP LVM is able to prepare and unprepare SAP systems and
instances. That means, in case of a prepare, the SAP instance or system is made ready to be
started on a resource like an LPAR. During the prepare phase, the storage volumes belonging
to an SAP system are mapped to the LPAR, the volume groups are imported, the file system
are mounted, and the SAP instance service is started.
In case of an unprepare, the SAP instance service is stopped, file systems volumes belonging
to an SAP system are unmounted, the volume groups are exported, and the storage are
unmapped. This allows to map and start the SAP system to/on the same LPAR, or to/on
another LPAR.
The prepare / unprepare can be run on SAP instance or on a complete SAP system including
all instance belonging to an SAP system. If the complete SAP system is affected, the function
is called mass operation.
Prepare / unprepare triggers following calls:
Storage CLI commands in order to attach or detach storage volumes containing SAP
systems to / from certain OS hosts
OS specific storage libraries for handling the storage mapping
OS specific libraries for handling the SAP instance service
5.1.1.2 Start / stop SAP systems and instances
In order to start an SAP instance or system, it needs to be successfully prepared. After an
SAP instance or system has been stopped, it can be unprepared.
The prepare+start and stop+unprepare can be executed as a single workflow each.
As for prepare / unprepare mass operations allow to start / stop complete SAP systems versus
single SAP instances.
Start / stop triggers following calls:
OS specific libraries for handling the SAP tasks start / stop
5.1.1.3 Relocate SAP instances and systems
The relocation of SAP instances and systems allows to move SAP instances or systems to
another LPAR.
There are two flavors of SAP instance or system relocation:
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 110 of 169
IBM PowerVM based relocation
This means that the complete LPAR including the SAP and operation system is moved
(also called migrated) from one physical host to another physical host. In that case
LVM uses IBM PowerVM Live Partition Mobility (LPM). The LPAR and SAP
system stay running during the execution of LPM. LPM is used e.g. for hardware
maintenance activities.
Hypervisor based relocation triggers following calls:
o IBM PowerVM specific libraries for handling Live Partition mobilty in the
HMC or Systems Director
SAP adaptive based relocation
This means that the SAP system is moved from one operating system or LPAR to
another operating system or LPAR. This causes a short SAP downtime as the SAP
system needs to be stopped on the previous operating system and started on the „new‟
operating system. The process automates following functions:
o Stop SAP instance or system
o Unprepare SAP instance or system
o Prepare SAP instance or system on the „new‟ LPAR
o Start SAP instance or system on the „new‟ LPAR
As for prepare / unprepare or start / stop mass operations allow to relocate complete
SAP systems versus single SAP instances.
SAP based relocation is used e.g. for operating system based maintenance.
SAP based relocation triggers following calls:
o Storage CLI commands in order to attach or detach storage volumes containing
SAP systems to / from certain OS hosts
o OS specific storage libraries for handling the storage mapping
o OS specific libraries for handling the SAP instance service
Figure 85 and Figure 86 illustrate the two relocation scenarios.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 111 of 169
Figure 85 IBM PowerVM based relocation
Figure 86 SAP adaptive based relocation
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 112 of 169
5.1.1.4 Start / stop resources
With SAP LVM LPARs can be started or stopped.
Caution: Powering off LPARs without stopping and unpreparing the SAP systems running
inside the LPAR may cause data loss or data corruption.
Starting or stopping LPARs triggers following calls:
IBM PowerVM specific libraries for handling Live Partition mobilty in the HMC or
Systems Director
5.1.1.5 Monitoring
SAP LVM provides several types and levels of monitoring, e.g. CPU consumption, memory
consumption, dashboard, history data etc.
SAP LVM communicates with the virtualization manager in order to retrieve monitoring data
about IBM Power physical servers and logical partitions. Optionally SAP LVM tries to gather
CPU utilization data for discovered elements (physical servers and LPARs). SAP Notes
1728222 (for IBM Systems Director VMControl adapter) and 1728293 (for IBM HMC
adapter) provide configuration details of components involved in this scenario.
Following kinds of setup are supported:
Monitoring through IBM Systems Director VMControl adapter
Monitoring through HMC adapter
5.1.2 Executing the LVM Standard Edition Use Cases
5.1.2.1 Prepare / unprepare SAP instances and systems
The following section describes the required steps to prepare a SAP instance. As the
unprepare process is very similar, it will not be shown separately.
As a prerequisite, the SAP system needs to be installed and discovered in SAP LVM with
correct storage volumes and mount points.
To prepare a SAP instance:
Log in to SAP LVM Chose „Operations‟ Chose tab „Services‟ Enhance the SAP
system you want you work with Select the instance to be prepared Select or enter a
resource Chose the small arrow in the right lower corner of the button „Prepare and Stop‟
Chose ‚Prepare„
Figure 87 shows an example on how to prepare a SAP instance.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 113 of 169
Figure 87 Prepare SAP instance
You can monitor the progress in the „Logs‟ tab, see Figure 89.
5.1.2.2 Start / stop SAP systems and instances
The following section describes the required steps to start a complete SAP system including
the prepare step. As the complete SAP system is started, this is a mass operation.
The stopping process is very similar, it will not be shown separately.
As a prerequisite, the SAP system needs to be installed and discovered in SAP LVM with
correct storage volumes and mount points.
To start a complete SAP instance including preparation:
Log in to SAP LVM Chose „Operations‟ Chose tab „Services‟ Select the SAP
system be started Select or enter the resource for each instance to be started Select
„Mass Start (including prepare if possible) Chose ‚Execute„
Figure 88 shows the steps to start a complete SAP system as mass start operation.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 114 of 169
Figure 88 Start SAP system as mass operation
Figure shows the logs.
Figure 89 Logs of the 'Start SAP system' operation
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 115 of 169
5.1.2.3 Relocate SAP instances and systems
The following section describes the required steps to relocate a complete SAP system using
the adaptive relocation. That is, only the SAP system (not the operating system) is moved to
another LPAR. As the complete SAP system is relocated, this is a mass operation.
As a prerequisite, the SAP system needs to be installed and discovered in SAP LVM with
correct storage volumes and mount points, and it needs to be running.
To start the relocation of a complete SAP system:
Log in to SAP LVM Chose „Operations‟ Chose tab „Services‟ Select the SAP
system be relocated Select both instances to be relocated Select or enter the target
resource for each instance Select „Mass Relocate All in State „Running‟‟ Chose
‚Execute„
Figure 90 shows how to trigger the relocation of SAP system as a mass operation.
Figure 90 Relocation of SAP system as mass operation
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 116 of 169
5.1.2.4 Start / stop resources
The following section describes the required steps to stop a LPAR. As the unprepare process
is very similar, it will not be shown separately.
Make sure that all SAP systems and instances are stopped and unprepared. Otherwise there
might be data loss, or the database might get corrupted.
To stop a LPAR:
Log in to SAP LVM Chose „Operations‟ Chose tab „Resources‟ Select the LPAR to
be stopped Chose tab Virtual Resource Chose „Deactivate‟ and select „Power Off‟
Figure 91 shows how to stop an LPAR.
Figure 91 Stop LPAR
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 117 of 169
5.1.2.5 Monitoring
The following section describes how to access the different monitoring functions.
Dashboard
The Dashboard is the first screen that is opened per default when you log in to LVM. It
provides information e.g. regarding services (SAP systems), physical resources, and virtual
resources. By selecting „Add Pool‟ you can open a „window‟ showing an additional subset of
information. Figure 92 shows an example of the Dashboard.
Figure 92 Dashboard in SAP LVM
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 118 of 169
Visualization
The „Visualization‟ tab provides a visual view on the infrastructure and SAP systems and
instances running on the infrastructure.
Figure 93 shows an example of the Visualization functionality.
Figure 93 Infrastructure Visualization in SAP LVM
Performance monitoring
Performance monitoring enables you to display metrics for specific entities (such as SAP
systems, specific instances, or virtual machines) for a specific time period. The historical
monitoring data is fed to automatic capacity management (ACM) to provide response time
data.
To open the performance monitor:
Log in to SAP LVM Choose Monitoring Chose tab „Performance‟ select one or
more entities on the left (for example, application server) click the metric you want to
display (for example, CPU usage) and the period (for example, hour)
To add entities to the display of a metric, specify the entity in the Entity field (or choose the
entity using the value selector) and choose Add.
To add the metric to the dashboard as a pod, choose Add to Dashboard. If you have added
entities to the display of the metric, the metric is displayed on the dashboard only for the first
entity.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 119 of 169
Figure 94 shows an example of the Performance Monitor.
Figure 94 Performance monitoring in SAP LVM
5.1.3 OS Provisioning
SAP LVM provides an interface for provisioning of AIX partitions through IBM Systems
Director VMControl. SAP LVM and VMControl coordinate and automatically execute all the
provisioning workflow steps that are:
Optional creation of „mksysb“ OS image on NIM server
Definition of LPAR profile on IBM Power HMC
Allocation of storage volumes required for rootvg
Connection of storage volumes with the LPAR incl. FC zoning definition, VIOS
mapping or NPIV tunneling and host bus adapter masking on storage subsystem
Installing of „mksysb“ image on a newly created partition
Registration of the provisioned operating system in IBM Systems Director and
gathering of the OS inventory data
SAP LVM interacts in this scenario as a user interface and workflow coordinator. All the
steps are explicitly or implicitly executed on IBM Systems Director VMControl itself. SAP
LVM provides three scenarios where OS provisioning is involved:
Provisioning of a new SAP LVM virtual resource. By that scenario SAP LVM
user chooses a source operating system or a predefined “mksysb” image as
template and provisions an additional SAP LVM virtual resource based on this
template. In case a template is a running partition – SAP LVM creates a temporary
“mksysb” image first. That image is deleted at the end of the provisioning process.
This scenario is available in SAP LVM standard and enterprise editions. We
describe this scenario below in this chapter.
Provision a new SAP system clone / copy target. This scenario is a subtask of SAP
clone / copy scenarios described in chapters 5.2.35.2.5 and requires SAP LVM
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 120 of 169
enterprise edition. Instead of (re-) using an existing resource a new resource is
created during the process and is used as target for the SAP system clone/ copy.
Create an SAP clone / copy by provisioning of a new operating system. By this
scenario SAP LVM takes a partition where a SAP source system is active as
provisioning template: The provisioned operating system includes a clone of SAP
source system. SAP LVM prepares this clone for later usage and integrates it into
SAP LVM landscape. This scenario is not described in this cookbook because it
requires that SAP system is installed in the “rootvg” which is typically not desired
in customer environments. SAP LVM enterprise edition is required for the
execution of this use case.
All these scenarios require configuration of SAP LVM virtualization manager for IBM
Systems Director VMControl as described in chapter 4.6. SAP Note 1728222 – Configuration
of IBM Systems Director VMControl for SAP LVM describes what should be done on IBM
Systems Director VMControl itself in order to integrate it with SAP LVM. If you are able to
capture and deploy an operating system via IBM Systems Director VMControl using NIM
and selecting a particular IBM Power server and storage system then you can perform similar
provisioning operation over SAP LVM.
5.1.3.1 Provisioning of a new SAP LVM virtual resource
This scenario includes following subtasks:
Gathering user input for the provisioning process.
Optional creation of an “mksysb” image through IBM Systems Director
VMControl in case a running partition is taken as provisioning template. This
subtask corresponds to IBM Systems Director VMControl “Capture Image” task.
Provisioning of a new operating system. This subtask corresponds to IBM Systems
Director VMControl “Deploy Workload” task.
Register a provisioned operating system in IBM Systems Director. This subtask
corresponds to IBM Systems Director “System Discovery”, “Configure Access”
and “Collect Inventory” tasks.
Optional deletion of “mksysb” image in case it was implicitly created for the
provisioning process.
Below we provide screenshots (Figure 95) that show execution of OS provisioning process in
SAP LVM.
Navigate to Provisioning Virtual Resource Provisioning and choose SAP LVM
virtualization manager where the provisioning process should be triggered. You see a list of
preconfigured “mksysb” images and running partition which may be taken as provisioning
templates. An “mksysb” image must occur in IBM Systems Director VMControl as a valid
deployment image. Otherwise it will be not shown in this dialog. Only running LPARs with
AIX operating system that is discovered, unlocked and for which inventory is collected by
IBM Systems Director may be taken as templates for provisioning. Partitions that are stopped
or where corresponding operating system is not known on IBM Systems Director are not
shown in this dialog. Choose an image or a partition and press “Provision”.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 121 of 169
Figure 95 OS Provisioning – Template Selection
Define a name for LPAR (Figure 96). We strongly recommend that you use not fully qualified
hostname of the operating system as LPAR name.
Figure 96 OS Provisioning – LPAR Name Definition
Choose IBM Power server that should host target LPAR (Figure 97). You may provision
LPARs which run in dedicated or in shared CPU usage mode. If you choose IBM Power
server or “Default Pool” – the provisioned partition will run in dedicated CPU usage mode. If
you provision LPAR into “Physical Shared CPU Pool” – the partition will share its CPU
cycles with other partitions running in that pool.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 122 of 169
Figure 97 OS Provisioning – Target Server Selection
Define storage pool where storage volumes for rootvg of the provisioned system should be
allocated (Figure 98). Storage pools shown in the dialog correspond to the pools provided in
IBM Systems Director VMControl dialog for deployment of a new workload. These may be
external storage subsystems and storage pools available on VIO servers.
Figure 98 OS Provisioning – Storage Pool Selection
Define parameters required for execution of provisioning process (Figure 99). The number of
configuration parameters varies depending on what is chosen as provisioning template.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 123 of 169
Figure 99 Provisioning – Definition of Provisioning Parameters
In case you use an “mksysb” image as provisioning template you may / must specify
following parameters:
Hostname for the operating system
CPU Entitlement. The guaranteed physical CPU capacity for the virtual
system. This parameter is relevant in case you provision into “Physical Shared
CPU Pool” and is not relevant in case you partition gets dedicated CPUs. The
shown value is a value that was configured as default in the LPAR profile of
the captured partition. You may specify values within the certain range that is
shown in description of this parameter. The value configured here must fit to
the number of virtual CPUs specified by another parameter.
Virtual CPUs. The number of online virtual CPUs. The shown value is a value
that was configured as default in the LPAR profile of the captured partition.
You may specify values within the certain range that is shown in description of
this parameter. The value configured here must fit to the CPU entitlement
specified by another parameter.
Partition Memory. Online partition memory. The shown value is a value that
was configured as default in the LPAR profile of the captured partition. You
may specify values within the certain range that is shown in description of this
parameter.
OS User / OS Password. The OS credentials are required for gathering
inventory on the deployed operating system. The inventory gathering is
required in case you plan to use the new partition as source for further
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 124 of 169
deployments. These parameters are optional but we strongly recommend to
define them.
Virtual Network. Virtual network adapter (VLAN) of the primary network.
Use F4/OVS to see possible input values.
Additional network specific parameters (Short host name for the system, Static
IP address and netwrk mask for the network adapter specified by parameter
„Virtual Network“, Default IPv4 gateway, DNS domain name for the system,
IP addresses of DNS servers for system). Specify parameters required for
configuration of the OS network adapter.
NIM Resource or Resource Group. Specify the name of an existing NIM
Resource or NIM Resource Group to allocate during the deployment. Any
defined NIM Resource Group, or Resource of class "resources" can be
specified, except: mksysb, spot, lpp_source, ovf_vm, master.
In case you use a running LPAR as provisioning template you may / must specify following
parameters:
Hostname for the operating system.
OS User / OS Password. The OS credentials are required for gathering inventory
on the deployed operating system. The inventory gathering is required in case you
plan to use the new partition as source for further deployments. These parameters
are optional but we strongly recommend to define them.
Virtual Network. Virtual network adapter (VLAN) of the primary network. Use
F4/OVS to see possible input values.
Image Repository. Define repository that is capable to store images of the source
partition. The image created trough this process will be deleted automatically. Use
F4/OVS to see possible input values.
In the last configuration step (Figure 100) you may validate your configuration and choose if
you would like to register the operating system that will be provisioned as SAP LVM virtual
resource. The checkbox „Power On Virtual Resource After Provision“ is not relevant for
provisioning using NIM through IBM Systems Director VMControl. A provisioned system
will be started automatically at the end of the provisioning process.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 125 of 169
Figure 100 Provisioning – Final Parameter Confirmation
After you click on provisioning SAP LVM starts the process. You may navigate to
monitoring and check the status (Figure 101). After some minutes (23 minutes in our
environment) you may use the new additional SAP LVM virtual resource.
Figure 101 OS Provisioning – Process Monitoring
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 126 of 169
5.2 SAP LVM Enterprise Edition Use Cases
The execution of the SAP LVM Enterprise edition use-case require that
LVM Enterprise Edition (VCM_ENT.SCA) is deployed to the managing SAP
NetWeaver 7.3 Java Web AS system.
5.2.1 SAP System Copy Use Cases
The SAP LVM Enterprise Edition Use-Cases include the SAP System Copy Use-Cases
Figure 102 illustrates the three different scenarios for SAP System copies in SAP LVM:
SAP System Clone
Initial SAP System Copy
SAP System Copy Refresh
Figure 102 SAP System Copy scenarios supported by SAP LVM (Source SAP)
These are discussed in more detail in the following chapter.
5.2.1.1 SAP System Clone
In the SAP System Clone scenario, an exact duplicate of an existing SAP system is created.
Content and system configuration are 100% identical in all aspects to the original system. To
avoid conflicts e.g. with interfaces dealing with the original system, the clone system is fully
isolated from the rest of the environment before the SAP target system can be started:
Therefore, the clone system is established in an isolated networking environment. AIX ipsec
allows to restrict the outbound TCP/IP network communication to hosts/ ports which don‟t
participate in any interface communication of the SAP system.
SAP System cloning is realized in two different versions:
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 127 of 169
Storage based System Clone
For storage based cloning the system to be cloned must be “adaptive enabled”: This
includes
o virtual IP addresses for both database and central instance
o Storage Setup separating all the SAP and database entities from the operating
system image. The SAP and database entities must be deployed to storage
supported by the SAP LVM storage adapter.
The target resource must match all prerequisites for a relocation target of adaptively
installed systems, e.g. the users, services, etc. must exist on it.
Virtualization based
For virtualization based System Cloning the system to be cloned must be installed into
the rootvg of a LPAR that is managed the SAP LVM virtualization manager, and
which is registered with SAP LVM. In addition virtualization based cloning is
supported with an “offline database copy” only. During virtualization based System
Cloning, the SAP System is stopped, the LPAR is captured via IBM Systems Director/
VMControl, and is deployed to the new target LPAR afterwards.
5.2.1.2 Initial SAP System Copy
During a SAP System Copy a duplicate of an existing SAP system is created with a different
system name and a unique system ID. The target system is updated with the new system
information, and can run in the same network environment as other SAP systems.
As a first step the System Copy involves the creation of a system clone to create the initial
target system. Then this initial target system is modified during subsequent execution steps to
update the settings according to the new SAP System ID. A special version of SAPInst,
“SAPInst Rename” is used to change the SAP system ID during this process. Additional post
configuration tasks in the SAP System are achieved via the Post Copy Automation (PCA)
tool. PCA provides task lists, with a predefined sequence of configuration tasks to configure
the extensive technical scenarios automatically. In the SAP LVM environment the execution
of a PCA task list is embedded in the overall System Copy workflow.
5.2.1.3 SAP System Refresh
Given are two systems 'SAP source' and 'SAP target' which exist and are configured in a
copy-relationship (target is copied from source) SAP LVM allows to 'refresh' the 'target'
system by:
exporting configuration data from the 'target' system
SAP System Copy for copying the source system to the 'target' system
(same target SID, same hostnames as previous system copy)
importing the formerly exported configuration data into the new 'target' system
The System Refresh feature has the same requirements as the System Copy.
5.2.1.4 SAP System Rename
The SAP System Copy can be executed in two steps also:
Create a SAP System Clone in a first step
Then use this clone later on, and rename it to a new SAP System ID. Like in scenario
5.2.1.2, the SAPINST rename tool is invoked for the SAP System ID change.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 128 of 169
At first, the cloned SAP system needs to be enabled for the SAP System Rename.
In Configuration Services, select the newly cloned System, and press “Edit”.
In the configuration screen, the activate the checkbox “Renaming” and press “Save”.
1
2
1
2
Figure 103 Enable the cloned System for SAP System Rename
Now, the SAP System Rename can be invoked. (See chapter 5.2.7)
5.2.2 SAP ACM (Automatic Capacity Management) Use-Cases
The SAP ACM Use-Cases are SAP LVM enterprise use cases also: They will however not be
discussed in further detail here. Just note that the following considerations need to be taken
into account for the ACM Use-Cases:
Both adaptively installed SAP systems and traditionally installed SAP systems are
supported for ACM
One dialog instance per resource LPAR is supported
All ACM managed dialog instances must be in the same set of logon groups reserved
for use by ACM.
SAP CCMS must be installed and operational for each managed SAP system.
For ACM-managed VMs in the same LVM pool, all the OS users must be configured
identical and must be defined in the central user store (LDAP).
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 129 of 169
5.2.3 Required preparation steps
Table 13includes an overview about the required initial setup step before a system can be
cloned, copied, refreshed or renamed with SAP LVM. For each of the required steps it
references to the section in this document where the step is described in more detail.
Setup Procedure in SAP LVM
Clone System Copy System Refresh System/ Database
Rename System
Configuring Networks
Required (See chapter 4.7.1)
Configuring Storage Providers
Required, if the source system is adaptively installed. (See chapter 4.2)
Not Required
Configuring User Management
Not part of the test environment: All users for the target systems had been pre-defined in LDAP
Configuring Name Servers
Not part of the test environment: All virtual hostnames and IP addresses had been pre-defined in DNS.
(See chapter 5.2.4.1 for additional information)
Configuring Remote Function Call Destinations
Not Required Required
(See chapter 5.2.5.2)
Setting the Infrastructure Assignment
Required (See chapter 4.4.1.1 and 4.4.1.2)
Configuring System Rename Configurations
Not Required Required
(See chapter 4.7.2)
Configuring Mount Points
Required, if the source system is adaptively installed (See chapter 4.4.1.3 and 4.4.1.4)
Configuring Java Post Copy Automation
Not Required Not part of the test environment:
SAP ABAP Stack tests only
Configuring a universally unique identifier (UUID)
Required (See chapter 3.4.5.4)
Not Required
Enabling the system for (See chapter 5.2.1.4, Figure 103)
cloning copying refreshing renaming
Table 13 Required setup procedures
5.2.4 Execution of a SAP System Clone
The following section describes the required steps for starting the SAP system copy in SAP
LVM. As a prerequisite, a SAP source system need to be installed and discovered in SAP
LVM, and need to be enabled for SAP System cloning in the SAP LVM configuration
A SAP System Copy is started in the “System and AS Provisioning” tab in the “Provisioning”
view. A SAP System is selected, and the “Clone System” Button is entered. See Figure 104:
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 130 of 169
11
Figure 104 Invoke SAP System Clone
SAP LVM starts the System Cloning dialog: In the first screen, the Storage Pool is shown,
and a short name and a description for the target system are specified (Figure 105).
Accept the default settings, and press “Next” to get to the next screen.
11
Figure 105 SAP System Clone: Basic data for the target system
In the next screen (Figure 106) the target resource for the SAP system clone is specified:
1. In this example the default value “Use Existing Resource” was choosen. SAP LVM
takes one existing LPAR out or the pool of resources. The SAP System will be cloned
and attached to this already existing LPAR.
Alternatively “Provision New Resource” could be selected. Then a new LPAR will be
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 131 of 169
created as the target resource for the SAP System Clone. SAP LVM calls the
virtualization manager (IBM system Director/VMControl) to deploy the new target
resource.
2. Press “Next” to continue with the next selection screen
1
2
1
2
Figure 106 SAP System Clone: Resource selection
In the next screen, the virtual host names and the network for the SAP system clone are
defined (Figure 107). The virtual hostname per default is derived from naming conventions.
The short name specified in Figure 105 is inserted into the virtual hostname.
5.2.4.1 Excursus: DNS Name resolution considerations
DNS name server considerations
All target IP addresses for the “clone” IP labels are predefined in DNS by an appropriate
address range. Assuming the pre-defined IP addresses (see the example in Table 14),
Role IP Label (DNS server) IP Address
SAP Source cl6ci01 192.168.165.116
cl6db01 192.168.165.126
SAP Target cl6cloneci01 192.168.165.136
cl6clonedb01 192.168.165.146
Table 14 Examples for IP addresses
SAP Note 1572841 - Setup of name server update of SAP LVM describes an alternative
solution where SAP LVM automatically updates/creates virtual hostnames in the DNS server.
SAP LVM then creates the respective A (forward) and PTR (reverse) lookup records for IPv4
addresses in the DNS server at the runtime during the SAP System Clone operation. During a
SAP System Destroy operation these records previously added are removed. This mechanism
ensures that in the DNS server the new IP addresses are assigned to the new IP label. In the
test environment however all virtual hostnames and IP addresses had been pre-defined in
DNS: The automatic name server update was not tested here.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 132 of 169
DNS behaviour on the target resource
On the target resource, things may look slightly different: SAP LVM acquires the new IP
Adresses on the target resource during the “prepare” operation, e.g.
On the target resource:
root> netstat -in
Name Mtu Network Address
en0 1500 link#2 86.1f.b8.54.6b.2
en0 1500 9.153.164 9.153.164.137
en0 1500 192.168.164 192.168.165.146
en0 1500 192.168.164 192.168.165.136
However, the Name resolution on the target resource still states the “source” IP labels. This is
required for SAP system operation of the clone. The clone system is a 100% identical clone of
the source system, and so the virtual IP labels need to be unchanged on the target resource (as
e.g. the virtual IP labels appear in the instance profiles, and start scripts, etc.).
On the target resource:
root> netstat -in
Name Mtu Network Address
en0 1500 link#2 86.1f.b8.54.6b.2
en0 1500 9.153.164 siccp137
en0 1500 192.168.164 cl6db01
en0 1500 192.168.164 cl6ci01
To achieve that behaviour, SAP LVM adds new entries to the local /etc/hosts file:
On the target resource:
root> cat /etc/hosts
…
192.168.165.136 cl6ci01.isicc.de.ibm.com cl6ci01 #SAP-LM: Added to
modify local hostname resolution for cloning DO NOT MODIFY
192.168.165.146 cl6db01.isicc.de.ibm.com cl6db01 #SAP-LM: Added to
modify local hostname resolution for cloning DO NOT MODIFY
So, with the “local” name resolution the new “target” IP addresses are mapped to the “source”
IP labels.To ensure that the mechanism works it is mandatory that the DNS search order
evaluates the local /etc/hosts first. This ensures that the local host file supersedes the
(external) nameserver. The nameserver gets contacted only in case the record to be resolved is
not found in the local hosts file.
root> cat /etc/netsvc.conf
# order of search: 1. /etc/hosts 2. nameserver
hosts = local,bind4
However, back to the selection screen of the SAP system clone (Figure 107):
1. The network name needs to be selected
2. Click “Next” to continue with the next screen
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 133 of 169
1
2
1
2
Figure 107 SAP System Clone: Network definition
In the next screen, all the storage definitions are made for the SAP system clone (Figure 108).
1. Select the Consistency Method: By selecting “Online Clone”, SAP LVM will place
the DB2 UDB database into “write suspend” state before the FlashCopy/ SnapShot is
triggered in the storage system. When the FlashCopy / SnapShot is finished, SAP
LVM will initiate a “write resume” to bring back the database into operational state.
2. Selection of “Full Clone” allows to create a new, independent set of target volumes
(which no longer depend on the source volume: So theoretically the source volume
could be removed later on)
3. Press next to get to the next selection screen.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 134 of 169
1
2
3
1
2
3
Figure 108 SAP System Clone: Storage Definitions
In the next screen, all the port definitions for the network fencing of the SAP system clone are
specified (Figure 109).
The highlighted item in the allowed connections is the DNS name server: In the environment,
the standard /etc/hosts file didn‟t include entries for the virtual IP labels. All the name
resolution is done via DNS to the DNS server. So it is important that the LPAR can
communicate with the DNS server in fenced state also. Therefore the DNS server (here:
siccserv2) was added as allowed communication already during the system setup in SAP
LVM 4.4, Figure 41).
1. Select “Next” to get to the final confirmation screen
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 135 of 169
11
Figure 109 SAP System Clone: Network Fencing
The final confirmation screen appears (Figure 110): It is possible to review all the parameters
defined in all the steps before.
1. Press “Start System Cloning” to finally start SAP System Clone execution
11
Figure 110 SAP System Clone: Final parameter confirmation
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 136 of 169
After starting the SAP System Cloning process, the Monitoring screen is launched and the
progress of the SAP System Clone can be tracked.
5.2.5 Execution of a SAP System Copy
The following section describes the required steps for starting the SAP system copy in SAP
LVM. As a prerequisite, a SAP source system need to be installed and discovered in SAP
LVM, and need to be enabled for SAP System copies in the SAP LVM configuration.
5.2.5.1 Preparations for Post Copy Automation (PCA)
Post Copy Automation (PCA) is part of SAP NetWeaver Landscape Virtualization
Management, enterprise edition solution. This section illustrates how to set-up PCA initially.
Several preparation steps before set-up of SAP Post Copy Automation need to be executed in
the source system first: Details are described in the document "ABAP Post Copy Automation
Guide for Enterprise Edition" . See SAP NetWeaver Landscape Virtualization Management -
Installation & Upgrade Guides SAP NetWeaver Landscape Virtualization Management
1.0 and in SAP Note 1614266 - System Copy: Post Copy Automation (PCA) / LVM 1.0.
The file "Note_Overview.pdf" (which is attached to SAP Note 1614266) list a set of
additional SAP notes which need to be implemented as a prerequisite for PCA in the source
system. Dependent on the actual support package stack of the SAP system they may (or may
not be) already included in the system, and corrections may need to be applied to the SAP
system (via TA SNOTES). The ABAP program Z_SAP_NOTE_ANALYZER (which is also
attached to SAP Note 1614266) automates this procedure. The SAP system is checked, and
according to its actual support package stack the required SAP notes content is downloaded
directly during execution of the from Z_SAP_NOTE_ANALYZER report, the content is
transferred into the SNOTES notes assistant transaction and added to the queue. If all the pre-
requisites are met, then PCA is installed via the “Post Copy Automation Installer” (PCAI)
automatically into the managed source system. See also
Note 1589145 - Task Manager for Technical Configuration and
Note 1589175 - System Copy: Task Content for Task Manager
Further details are described in chapter 3 and chapter 4 of the “ABAP Post Copy Automation
Guide for Enterprise Edition“.
PCA is part of the enterprise edition of the SAP NetWeaver Landscape Virtualization
Management solution. PCA is shipped within SAP NetWeaver packages, but is deactivated
per default. The License Enabler add-on PCAI_ENT 1.0 is activates PCA and is so required in
the system to enable the execution of PCA task lists. Install the add-on PCAI_ENT 1.0 before
the first system copy of the system using transaction SAINT. Details are described in chapter
5 of the “ABAP Post Copy Automation Guide for Enterprise Edition“.
To grant the privileges to execute PCA activities for the administrator (dialog user) the
corresponding roles SAP_BC_STC need to be assigned to all the relevant users. This includes
the system administrator user responsible for PCA activities, and the technical users that will
be used via remote access by SAP LVM. Details are described in chapter 7 of the “ABAP
Post Copy Automation Guide for Enterprise Edition“.
PCA provides task lists, with a predefined sequence of configuration tasks to configure
extensive technical scenarios automatically. To prepare the run of these task lists the ABAP-
based “task manager for technical configuration” is used. In the SAP LVM environment the
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 137 of 169
execution of such a task list is induced without calling the task manager for technical
configuration directly. Further details are described in chapter 8 and chapter 9 of the “ABAP
Post Copy Automation Guide for Enterprise Edition“.
5.2.5.2 Add the RFC destinations for PCA to the SAP System configuration
In the services configuration, define the RFC destinations that will be used for the PCA
execution for the SAP system. Define the logon parameters for the main destinations, and
press “Test Connection”.: An attempt to connect to the SAP system is made.
Press “Retrieve Clients” and the list will be populated with all the clients of the existing
system. Different PCA activities may be executed specific for each client of the SAP system.
Adjust the Users accordingly, and define the passwords.
Figure 111 Define RFC destination for PCA execution
5.2.5.3 Start the SAP System Copy
A SAP System Copy is started in the “System and AS Provisioning” tab in the “Provisioning”
view. A SAP System is selected, and the “System Copy” Button is entered. Figure 112 then
shows an example for the screen of the first selection step. First configuration data for the
target system needs to be entered.
The SAP System ID for the target system is specified, and the master password needs to be
entered. SAP System ID and master password will be used later on during the
“SAPINST_RENAME” step to adapt the configuration. The additional description may be
adjusted additionally, or left on the default value. This will be the description for the new
system in the SAP LVM configuration.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 138 of 169
Figure 112 SAP System Copy – Basic data
Finally, data entry for Step 1 is confirmed by clicking the “Next” button.
In the second step the target resource(s) for the system copy is defined. Figure 113 shows the
screen for the second “Resource Selection” step. By selecting type “Use Existing Resource”
SAP LVM suggests a potential target, which fulfils the boundary conditions (capacity
requirements in SAPS and memory) for the target system. LPARs satisfying these conditions
can be selected in a drop-down list.
Alternatively “Provision New Resource” will trigger IBM Systems Director/ VMControl
respectively IBM Flex Systems Manager to provision a new LPAR as target for the system.
The services for the target system (Central Instance, Database) can be provisioned either
together to one target LPAR, or separated to different target LPARs.
Figure 113 SAP System Copy - Resources
Data entry for Step 2 is confirmed by clicking the “Next” button.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 139 of 169
In the next steps the network configuration for the target system is specified (Figure 114). The
IP addresses for the target system are proposed according to the naming conventions for the
virtual IP labels (<sid>ciXX, <sid>dbXX). The addresses and labels are either predefined in
the DNS (respectively in the local host files of the SAP LVM system and the involved
resources) already.
Otherwise, SAP LVM may directly update the DNS domain name server for the new entries,
however that need to be defined and configured according to SAP Note 1572841 - Setup of
name server update of SAP LVM first.
A network name for the Network needs to be selected by using the drop-down list (The
network name was defined before, see chapter 4.7.1).
Figure 114 SAP System Copy – Network settings
Data entry for Step 3 is confirmed by clicking the “Next” button.
In Step 4 the Instance number(s) for the Instances of the target system are defined (Figure
115). Per default, the target system will be installed using the same instance numbers like the
source system.
Figure 115 SAP System Copy – Instance Number
Data entry for Step 4 is confirmed by clicking the “Next” button.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 140 of 169
In step 5 the storage configuration for the target system is defined (Figure 116). The storage
image for the target system is created as a SnapShot/ FlashCopy of all source volumes to the
target volumes. The main structure like the overall number of AIX volume groups and their
content cannot be changed. Filesystem names can be adjusted however. Per default, all SAP
System IDs in the filesystem names are updated to the new SAP System ID of the target
system.
The consistency settings need to be specified: In the IBM Tivoli Storage FlashCopy Manager
scenario, specify “Database Consistency (Online Clone)”.
Figure 116 SAP System Copy – Storage Definition
Data entry for Step 5 is confirmed by clicking the “Next” button.
In Step 6 the OS administration users are specified for the target SAP system (Figure 117). If
LDAP is used for central user management, then the users need to be created manually on the
target resource before the SAP system Copy is started. “sapinst” currently is not able to
handle LDAP users correctly, and also the SAP LVM “Create User” step doesn‟t work with
AIX.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 141 of 169
Figure 117 SAP System Copy – User definitions
Data entry for Step 6 is confirmed by clicking the “Next” button.
In Step 7 further additional settings for DB2 schema user and DB2 instance are specified for
the new SAP System ID (Figure 118). For DB2 UDB, the schema user cannot be changed
during the process, and must be set identical to the schema user of the source database. Name
of the database instance for the target system is derived from the SAP System ID according to
the SAP naming conventions.
Figure 118 SAP System Copy – Rename SAP system ID
Data entry for Step 7 is confirmed by clicking the “Next” button.
In Step 8, the allowed ports/ connections for the network fencing step are defined (Figure
119). Additional ports/ hosts allowed for communication can be selected out of the set of
“Current connections on host” from the source system, and then added to the “allowed
outgoing connections on the target hosts”. To avoid additional manual steps it is best-practise
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 142 of 169
to include all the necessary hosts/ ports required for outbound connections of the system
already in the SAP system definition.
Figure 119 SAP System Copy – Network fencing
Data entry for Step 8 is confirmed by clicking the “Next” button.
In Step 9 the SAP clients for executing the Post-Copy Automation (PCA) steps can be
defined. In the test environments the SAP system copy was run without subsequent execution
of the PCA for the first test runs in any of the clients. This can be achieved by removing all
RFC destinations in this PCA section. (See Figure 120). Although the RFC destinations are
removed from the configuration here, the RFC definitions need to be present at first in the
base configuration of the SAP system in SAP LVM to enable it for the SAP system copy.
Figure 120 SAP System Copy - Select RFC destination for PCA
Data entry for Step 9 is confirmed by clicking the “Next” button.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 143 of 169
Before actually starting the SAP System Copy a summary screen is displayed: All the
parameter settings can be reviewed by expanding the different subsections, and eventually
adjust them once again (Figure 121).
Figure 121 SAP System Copy – Summary view
The SAP System Copy process is finally started by clicking the “Start System Copy” button.
After starting the SAP System Copy, the Monitoring screen is launched and the progress of
the system copy can be tracked.
5.2.6 Execution of SAP System Refresh
Prerequisite for the SAP System Refresh is a SAP System Copy run that was previously
completed successfully. The execution of the SAP System Rename is very similar to the steps
before. Start with the Provisioning view. Press “Refresh System” to start the parameter dialog
(Figure 122):
11
Figure 122 SAP System Refresh: Start system refresh
In the first screen base configuration parameters can be adjusted (Figure 123):
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 144 of 169
1. Specify and confirm the Master Password
2. Press “Next” to continue with the next selection screen
1
2
1
2
Figure 123 SAP System Refresh: Basic Data
Acknowledge target resource description in the following screen and press “Next” (Figure
124):
11
Figure 124 SAP System Refresh: Resource Data
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 145 of 169
Also, acknowledge the virtual hostnames and network settings (Figure 125). Press Next:
11
Figure 125 SAP System Refresh: Host Names
In the next screen, the storage settings are defined (Figure 126):
1. Decide if a full clone (target volumes are fully independent from the source volumes)
shall be created
2. Select the appropriate consistency method (typically “Online”)
3. Click “Next” to proceed with the next screen
1
2
3
1
2
3
Figure 126 SAP System Refresh: Storage Settings
Acknowledge the User settings. As it is a SAP system refresh, the Users should already exist.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 146 of 169
Press Next (Figure 127).
11
Figure 127 SAP System Refresh: Users
Enter the password for the schema user (Figure 128). Press “Next” to proceed.
1
2
1
2
Figure 128 SAP System Refresh: DB Schema
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 147 of 169
Acknowledge or eventually adjust the isolation settings during the network fencing (Figure
129).
11
Figure 129 SAP System Refresh: Network Isolation
Adjust or confirm the PCA settings (Figure 130). Press “Next” to enter the summary screen.
11
Figure 130 SAP System Refresh: PCA settings
Eventually review the settings. Finally invoke the SAP System Refresh by pressing “Start
System Refresh” (Figure 131).
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 148 of 169
11
Figure 131 SAP System Refresh: Parameter summary
5.2.7 Execution of SAP System Rename
The execution of the SAP System Rename is very similar to the steps before. Start with the
Provisioning view. The system needs to be enabled for SAP System Rename: Only then the
appropriate button is shown. Press “Rename system” to start the parameter dialog (Figure
132).
11
Figure 132 Start SAP System Rename
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 149 of 169
In the following screen (Figure 133) enter the basic data for the SAP System Rename.
1. Enter and confirm the Master-Password
2. Press Next
1
2
1
2
Figure 133 SAP system Rename: Basic Data
The following screen appears (Figure 134). For convenience, the virtual hostname can be
changed here, as the one from the system clone will not fit to the naming conventions.
1. Then, assign a name to the network
2. Press “Next” to proceed to the next screen
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 150 of 169
1
2
1
2
Figure 134 SAP System Rename: Virtual Hostnames
The SAP Instance number for the renamed system can be specified or, per default, use the
same as the source system (Figure 135). Press “Next to proceed to the next screen.
11
Figure 135 SAP System Rename: Instance numbers
In the following screen, the mount points are adjusted to the new SAP System ID (Figure
136). In most cases you can accept the default names, except if you specifically want to
update certain mount points. Afterwards, proceed with “Next” to the following screen.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 151 of 169
11
Figure 136 SAP System Rename: Mount points
The administrative users for the new SAP System are selected (Figure 137). As SAPinst
currently is not able to create LDAP users correctly, and also the SAP LVM user method
doesn‟t work correctly on AIX, create users and groups manually in LDAP before.
If the users are already existing (in LDAP), then the checkbox “Already exists” is
automatically flagged. Select “Next” to proceed to the following screen.
11
Figure 137 SAP System Rename: Specify users for target system
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 152 of 169
In the next step, the settings for the connect user are specified. For DB2, the connect user
cannot be changed during the System Rename.
1. Specify the password for the connect user
2. Press “Next” to proceed to the following screen
2
1
2
1
Figure 138 SAP System Rename: Specify database parameters
The next screen describes the allowed communications during the network fencing phase
(Figure 139). Ensure that all necessary connections (e.g. to the DNS nameserver) are allowed,
but that all potential dangerous communication is not explicitly allowed (and so prohibited).
Press “Next” to proceed with the following screen.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 153 of 169
11
Figure 139 SAP System Rename: Configure network isolation
In this screen, the SAP target clients for Post Copy Automation execution are specified
(Figure 140). In this example, the Post Copy Automation was explicitly disabled. To disable
PCA, remove all the potential clients from the list.
11
Figure 140 SAP System Rename: Disable Post Copy Automation (I)
If the table is empty (Figure 141), then the PCA won‟t be invoked for any client. Click “Next”
to proceed to the final confirmation screen.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 154 of 169
22
Figure 141 SAP System Rename: Disable Post Copy Automation (II)
In the final selection screen, the parameters can be checked before the actual start (Figure
142). Then press “Start System Rename” to start the execution.
11
Figure 142 SAP System Rename: Summary Screen
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 155 of 169
5.2.8 Additional Considerations for SAPInst Rename
See also SAP Note 1619720 - System Rename for SAP Systems based on SAP NetWeaver
for additional information.
Currently, there are some issues in the transfer of the parameterization for the SAPInst rename
execution from SAP LVM (at least for DB2 UDB/ AIX): The SAP LVM team is currently
working on that.
Additionally, the test environment identified the following issues in context of SAP system
copies:
1. The SAP System Copy in SAP LVM procedure doesn‟t handle DB2 UDB with
Automatic Storage Management (ASM) correctly.
o For the final relocation of the DB2 UDB database on the target system a
parameter file “relocate.template” is generated by the brdb6brt tool on the
source system. These parameter file doesn‟t include the required ASM
information for STORAGE_PATH and CONT_PATH information.
o The ASM directory structure follows the conventions
/db2/<SID>/sapdataX/db2<sid>/NODE0000/<SID>
Not all the <SID> and <sid> entries are adapted by SAP LVM to
the “new” target <SID> / <sid> values of the target system
Resolution: Handled in Event script, see below
2. SAPinst and SAP LVM don‟t handle user creation in LDAP correctly
Make sure that the ABAP Connect User on the target resource has the
following group sets assigned groups=db<src-sid>mon,db<src-
sid>mnt,db<tgt-sid>mon,db<tgt-sid>mnt
Resolution: Handled in Event script, see below
3. SAPINST_RENAME cannot handle the update to db2nodes.cfg if a “full qualified
hostname” is set as hostname on the target resource
If hostname and netname are specified in db2nodes.cfg, only the hostname
is changed. If only the hostname is specified, then it is translated
errornously
Resolution: Handled in Event script, see below
4. An issue with “db2chgpath” was identified in the test environment (DB2 UDB
V9.7 FP5SAP)
Resolution: Fixed in DB2 UDB V9.7 FP6 (APAR IC80849).
DB2 UDB V9.7 FP6 is available, however at the moment not yet certified by SAP.
5. Update to AIX 6.1 TL7 SP4 (6100-07-04-1216) introduced a new issue IV22062
on some of the nodes.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 156 of 169
Resolution: Install eFIX on top of AIX 6.1 TL7 SP4
root> emgr -l
ID STATE LABEL INSTALL TIME UPDATED BY ABSTRACT
======================================================
1 S IV22062s04 06/11/12 12:42:43 Ifix forIV22062@6.1TL7SP4
Item 1.), 2.) and 3.) are addressed by a “Pre-Event” script, which is invoked before SAPInst
rename is started: This is realized via a “Pre-Hook” in SAP LVM. The following outlines how
to implement such a hook. See also SAP Note 1465491 - Provider Implementation Definition
for further details.
5.2.8.1 Create the script and register it with the Host Agent
The script is invoked for both SAP Central Instance and Database. It checks, if it is running
for the database, and then identifies the Source SID based on the environment. Target SID is
specified as argument. The script provides the following functionality
Check and eventually create/ correct the LDAP Users and Groups required for the
target system
Handle DB2 UDB ASM
o Update relocate.template to include storage and container pathes in the
template
o Correct directory structure below the /db2/<SID>/sapdataX filesystems
Update db2nodes.cfg (in the directory of the source instance)
When the script is created, then a .conf file needs to be created in the directory
/usr/sap/hostctrl/exe/operations.d .so that it can be invoked from the Hostagent.
cat PreSAPinst.conf
Name: PreSAPinst
Command: /usr/sap/hostctrl/exe/operations.d/IBM/PreSAPinst.ksh
$[SYSTEM_HOST:#required] $[system_sid:#required] $[SERVICE_CLASS:#required]
$[service_name:#required] $[service_type:#required]
Workdir: /usr/sap/hostctrl/exe/operations.d/IBM/traces
ResultConverter: flat
Platform: Unix
In the .conf file, the Script and its parameters are defined in the „Command:“ section.
The “Workdir:” contains the actual work directory for the script: Make sure that the directory
exists.
Restart the SAP Host Agent to register it.
5.2.8.2 Define the Provider Implementation Definition
For the Provider Implementation Definition, goto Setup Extended Settings Custom
Operations and Custom Hooks Provider Implementation Definition (Figure 143) in SAP
LVM. Choose “Add” to define a new definition.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 157 of 169
Figure 143 Provider Implementation Definition (I)
In the following screen (Figure 144), enter
1. Name and select Type “Script Registered with Host Agent”
2. The hostname running a SAP Host Agent where the script was registered
3. Press “Retrieve Registered Scripts”
2
3
1
2
3
1
Figure 144 Provider Implementation Definition (II)
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 158 of 169
Now, select the Registered Script from the Pick List (Figure 145):
Figure 145 Provider Implementation Definition (III)
The registered script is a Custom Hook, and it shall be invoked for a Service (Database, SAP
Central Instance): Checkmark “Hook” in the row for “Service”. Afterwards, save the new
Definition (Figure 146)
1
2
1
2
Figure 146 Provider Implementation Definition (IV)
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 159 of 169
5.2.8.3 Define the Custom Hook Definition
The “Custom Hook Definition” then maps the script defined before to the “Pre-SAPInst”
Event. Change to the Custom Hook Definition View (Setup Extended Settings Custom
Operations and Custom Hooks Custom Hooks), and press “Add” (Figure 147):
Figure 147 Custom Hook Definition (I)
Figure 148 illustrates the next panel:
1. Enter a name, and select the provider implementation definition defined before
2. Then select the Hook type from the list
1
2
1
2
Figure 148 Custom Hook Definition (II)
1. Select “pre_SAPInst” to link the definition as Pre-Event to SAPInst Rename: During a
System Copy/ System Rename/ System Refresh activity the script is executed on the
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 160 of 169
target resource before the SAPInst step (Figure 149).
2. The picklist disappears, and the screen turns back. Press “Save” to confirm the
changes.
1
2
1
2
Figure 149 Custom Hook Definition (III)
5.2.9 Destroy of a SAP System
During the SAP system destroy action
the storage configuration and all the actual volumes for the systems will be deleted in
the storage system
IP labels, which were created in the DNS server during SAP system creation by SAP
LVM, will be removed from the DNS server once again
The SAP system will be deleted in the SAP LVM repository.
All services for the system needs to be in “unprepared” state before the “Destroy” action can
be started in SAP LVM. If the services are still running, then the system needs to be stopped,
and an uprepare operation needs to be started for it first.
5.2.9.1 Stop and Unprepare a SAP System
Change to the „Operations“ view and check the status of the system: As a prerequisite to be
able to start the “Destroy” operation all the services of the system need to be in state
“Unprepared”. In the example as illustrated in Figure 150 the system is still active: A mass
operation to “Stop and Unprepare” is selected for both the database and the SAP Central
Instance.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 161 of 169
Figure 150 System State in Operations view
Press „Execute“ to start the mass operation: The service gets locked for other operations, and
the “Stop and Unprepare” operation is started (Figure 151):
Figure 151 Mass operation „Stop and Unprepare” started
The progress of the current action can be monitored via the “Monitoring” view. On the upper
section of the screen a status and progress overview is given. By selecting an operation from
the list, and choosing the “Steps” sheet the individual steps of the workflow can be monitored
(Figure 152):
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 162 of 169
Figure 152 Monitoring view
Wait until the „Stop and Unprepare“ operation is completed for all the services (Figure 153):
Figure 153 Mass Stop completed
5.2.9.2 Destroy the SAP System
Change to the „Provisioning“ view and select the system to be destroyed. The System needs
to be stopped and all the services need to be in state “Unprepared” as a prerequisite (Figure
154).
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 163 of 169
Figure 154 Destroy System
Press the button „Destroy“ System: Three additional selection/ information steps are
processed before the final destroy operation takes place.
During the destroy operation all the storage volumes which were deployed for the system
during the Install/ Copy/ Clone process are removed on the storage system. In the first step,
all these storage volumes to be deleted later on are shown (Figure 155):
Figure 155 Destroy operation – delete storage volumes
Press „Next“ to get to the next screen. If SAP LVM is configured for DNS updates and has
generated DNS entries for the system during a SAP LVM deployment process then these
entries will get removed during the destroy operation. This feature was not configured in the
test environment. Instead all IP addresses and corresponding DNS entries were preconfigured
in the DNS before. So in the example case (Figure 156) the list is empty.
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 164 of 169
Figure 156 Destroy operation – delete host names
Click next to proceed to the summary screen. The parameters are shown, and the actual
destroy operation is started by pressing the “Start System Destroy” button (Figure 157).
Figure 157 Destroy Operation – summary screen
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 165 of 169
6 Troubleshooting hints and tricks
6.1 Phase “Prepare Clone Volumes”
Error Message “Error while writing FCM profile <profile> on FCM target host <resource>”
Check .conf file FCMCLI.conf in /usr/sap/hostctrl/exe/operations.d on the host
<resource”. Does the “workdir” exists, and is user “root” allowed to access it?
6.2 Cleanup activities on a resource LPAR
Stop SAP service and all agents for it
Stop the SAP Hostagent on the LPAR
Un-mount all the filesystems belonging to the SAP Service
lsvg |grep -v rootvg |while read i ; do
lsvgfs $i
done | sort -r | while read fs; do
mount | grep -w ${fs}
if [[ $? -eq 0 ]]
then
fuser -cuk $fs
umount $fs
fi
done
Vary-off and export all the volume groups
lsvg |grep -v rootvg |while read i ; do
varyoffvg $i && exportvg $i
done
Delete the disk devices corresponding to those items
Delete virtual hostnames from the network adapter
Remove all IP filters in IPSec
rmfilt -v 4 -n all && mkfilt -v 4 –u
Clean-up /usr/sap/sapservices
Remove all entries set by SAP LVM on the target resource
o /etc/hosts
o /etc/services
Restart SAP Host Agent
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 166 of 169
6.3 Clean-up orphaned volumes in the IBM XIV
Search for volumes which belong to the cloned system and are mapped to the target
resource:
o Un-map all these volumes (“unmap_vol”)
o Delete the volumes (“vol_delete”)
Search for temporary volumes created during the cloning process (Prefix “TSM”,
respectively the prefix defined in the FCM manager profile)
o Delete the volumes
6.4 Delete the System in SAP LVM
Goto Configuration Services
Then select the appropriate system, and press “Remove Service and System” (Figure 158).
Confirm deletion then in the following conformation dialog.
Figure 158 Remove System in SAP LVM
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 167 of 169
7 Resources
7.1 IBM Documentation
7.1.1 Whitepapers
Flyer “Cloud-enabled SAP application management”
IBM Cloud Solution for SAP – Integrating IBM Infrastructure with SAP Landscape
Virtualization Management
Introduction to an Integral IBM / SAP Cloud Solution for SAP Landscape Management
IBM entry cloud configuration for SAP solutions on Power Basic Concepts and High Level
Design
7.1.2 Redbooks / Technotes
Deploying Cloud Components on POWER
IBM Flex System Manager
IBM PureFlex System and IBM Flex System Products and Technology
IBM Flex System p260 and p460 Compute Node
7.1.3 Product Documentation
Requirements and support for AIX using Network Installation Manager (NIM)
Requirements and support for AIX, IBM i, and Linux using storage copy services (SCS)
7.1.4 iRAM
IBM Systems Director VMControl Implementation Perform Guide
7.2 SAP Documentation and SAP Notes
7.2.1 SAP Documentation
SAP NetWeaver Landscape Virtualization Management
SAP Community Network (SCN) Virtualization and Cloud
SAP Help Portal (SAP LVM)
SAP NetWeaver Landscape Virtualization Management, Enterprise
SAP NetWeaver Landscape Virtualization Management at a Glance
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 168 of 169
7.2.2 SAP Notes
SAP Note Title / Description
Note 962955 Use of virtual TCP/IP host names
Note 1292836 Logfiles of the SAP Hostagent
Note 1438774 New profile parameter system/uuid and system/description
Note 1465491 Provider Implementation Definition
Note 1527538 SAP NetWeaver Landscape Virtualization Management 1.0
Note 1572841 Setup of name server update of SAP LVM
Note 1589145 Task Manager for Technical Configuration
Note 1589175 System Copy: Task Content for Task Manager
Note 1614266 System Copy: Post Copy Automation (PCA) / LVM 1.0
Note 1619720 System Rename for SAP Systems based on SAP NetWeaver
Note 1644520 Integration of IBM Components into SAP LVM
Note 1728222 Configuration of IBM Systems Director VMControl for SAP LVM
Note 1728293 Configuration of IBM Power HMC for SAP LVM
Table 15 List of SAP Notes
IBM SAP International Competence Center
Combining IBM Infrastructure with SAP NetWeaver Landscape Virtualization Management © Copyright IBM Corporation, 2012 Page 169 of 169
COPYRIGHT LICENSE
If this information contains sample application programs in source language, which illustrate
programming techniques on various operating platforms, you may copy, modify, and
distribute these sample programs in any form without payment to IBM, for the purposes of
developing, using, marketing or distributing application programs conforming to the
application programming interface for the operating platform for which the sample programs
are written. These examples have not been thoroughly tested under all conditions. IBM,
therefore, cannot guarantee or imply reliability, serviceability, or function of these programs.
The information contained in this document has not been submitted to any formal IBM test
and is provided "AS IS" with no warranties or guarantees either expressed or implied.
All examples cited or described in this document are presented as illustrations of the manner
in which some IBM products can be used and the results that may be achieved. Actual
environmental costs and performance characteristics will vary depending on individual client
configurations and conditions.
IBM is not responsible for printing errors in this document that result in pricing or
information inaccuracies.
A full list of U.S. trademarks owned by IBM may be found at:
http://www.ibm.com/legal/copytrade.shtml
A full list of trademarks owned by SAP may be found at:
http://www.sap.com/company/legal/copyright/trademark.epx
SAP and other SAP products and services mentioned herein, as well as their respective logos,
are trademarks or registered trademarks of SAP AG in Germany and in several other countries
all over the world.
UNIX is a registered trademark in the United States, other countries or both.
Linux is a registered trademark of Linus Torvalds in the United States, other countries or
both.
Microsoft, Windows, Windows NT and the Windows logo are trademarks of Microsoft
Corporation in the United States, other countries or both.
Recommended