Problem Determination and Log Files

HACMP event log file:n /tmp/hacmp.out

> Shows acquiring and releasing of service IP labels

> Service IP labels not associated with the local site should be absent

clstat commandn Will display the state of service IP label where the resource group is

online primary

,Redbooks Workshop

Page 201 of 253e" 09/21/064:56 AM

ibm.com (. r)4,, ni^: rn^^ti mal Ti:i iu Su;!} ^rt Oi^7eniz^i

Summary

Supporting IP Address takeover on XD networks allows thecustomer to better utilize their XD networks

Site-specific IP labels allow clients to access thereapplications when they move from one site to another and theother site has a different subnet

NOTE: IP Address Takeover does not necessarymean that the service IP label will be moved fromone nee to another!

Redbooks WorkshopIBMI18O - kg c TtlmIOM& on o!.* bn

Page 202 of 25309/21 /064:56 AM

Page 203 of 253

09/21 /064:56 AM

nteq ati m a Tug. ri a SIrt O«ruu>i

Break

t . Redbooks Workshopewnao . _ r tsswwno^-

Page 204 of 25309/21/064:56 AM

Ibm.com/redbooks

it t-i):ltion,d Te:lmwol Si r^pcai C) ()nnization

HACMP/XD for Geographic LVM

Redbooks WorkshopIBM ITSO - International Technical Support Organization

Page 205 of 25309/21 /064:56 AM

GLVM with HACMPGLVM is IBM's strategic geographic disaster recovery (DR)technology for pSeries systems running AIX.

Based on AIX LVM; provides Geographic Volume Group Mirrorbelow the LVM layer

Designed to be simpler and easier to use

GLVM is available as part of HACMP /XD 5.2 (since April 2005),and available in HACMP /XD 5.3 , and HACMP 5.4.

Intended to eventually replace HACMP /XD HAGEO

HACMP/XD and GLVM provide together two main functions:• Real time mirroring of data to a remote location (GLVM)• Automated disaster recovery (HACMP)

Comprehensive whitepaper on configuration and use - see:http://www-03.ibm.com/servers/aix /whitepapers/aix g1vm.htm1

Redbooks Workshop

Page 206 of 253

09/21/064:56 AM

Introduction

Usability• No more user managed state maps• Remote copy management becomes LVM's responsibility• Let LVM do what it does best ...• ... and let HACMP what it does best

Reliability/Robustness

Cost effective (GLVM is integrated in AIX 5.3)

Performance

Disaster Recovery / Business Continuity growth area

Redbooks Workshop

Page 207 of 253

09/21/064:56 AM

Terminology

- Remote Physical Volume

- Kernel device driver that runs on the local nodewith a running application

- Kernel module at remote node (server side),receives mirror copy from RPV clients

- a volume group that containsone or more logical volumes that have copies residing onRemote Physical Volumes (RPVs).

Redbooks Workshop

^a 9

Page 208 of 25309/21/064:56 AM

OWN

Terminology: R-PV

RPV composed of two parts:• RPV server - on remote system (where disk resides)• RPV client - on local system (where application runs)

RPV server is a kernel module which provides access toremote disks.

• Each disk should be configured as a RPV server (example: rpvserverO)

RPV client is a device driver which connects to RPV server• Every remote disk should have a client on the local node (example:

hdisk5)

RPV client with RPV server provides access to remote disksas if they are locally attached

Remote system must be connected using TCP/IP network• "Unlimited" distance - the network latency is a factor

Redbooks Workshop

Page 209 of 253

09/21/064:56 AM

00

I lt,, I.l[i_ i Tc:clau -j, SLID)}. ^rt 0^^7ani^:^tion

xampie

EMINAVIN

Redbooks Workshop

Page 210 of 25309/21/064:56 AM

Geographically Mirrored VG

GMVG - Geographically Mirrored Volume Group

An AIX Volume Group that contains the physical disks fromboth local and remote systems

Use Super-Strict inter-disk allocation policy for LogicalVolumes

This forces each complete copy of the logical volume toreside on the separate physical volumes

Make sure• At least one whole mirror copy on local physical volumes• At least one whole mirror copy on remote physical volumes

Redbooks Workshop

Page 211 of 253

09/21/064:56 AM

Placing GMVG under HACMP/XD control

HACMP detects failures and recovers the applications on thebackup site, if required

GMVGs must of configured such that HACMP can varyonGMVG from either sites.

All takeover nodes must have RPV clients and RPV serversconfigured

GMVGs should be included in the HACMP resource groups,just like a regular volume group.

• It's recommended to set the forced varyon to 'true'• It's recommended to turn the LVM quorum OFF

Concurrent disk access for GMVGs is not allowed• Do not include a GMVG in a 'Online On All Nodes' resource group

Redbooks Workshop

NIM

Page 212 of 253

00

09/21/064:56 AM

Data Mirroring - Concepts

The next few slides show the importance of data mirroring.

Single server with one data copy• Single server with standard AIX LVM mirroring• HACMP cluster with standard AIX LVM mirroring• Two site HACMP cluster with Geographic LVM mirroring

Redbooks Workshop

Page 213 of 253

/o**' 09/21 /064:56 AM

Single server with one data copy

A single copy of mission critical data spans across two disks.Either disk is potentially a single point of failure

Redbooks Workshop

Page 214 of 253

`./

09/21 /064:56 AM

Single server with AIX LVM mirroring

PVI=hdisk7 PV3=hdisk9PV2=hdisk8 PV4hdiskl0

Node A

Copy #1 Copy #2VG rcrvg

Two more disks contain a second mirror copy. However, theserver is still a potential single point of failure

Redbooks Workshop

Page 215 of 253

HACMP Cluster with AIX LVM mirroringPV1=hdlsk7 PV3=hdisk9PV2=hdisk8 PV4=hdisklO

Node A

Copy #1 Copy #2

VG rcrvg

Node B

PVl=hdisk5 PV3=hdlsk7PV2=hdisk8 PV4=hdisk8

This eliminates most potential single points of failure. However,a disaster (fire, flood, etc.) can wipe out the entire clusteralong with all copies of mission critical data

...

Redbooks WorkshopIBM IT80- IrMlEOill Twinlwl& pmlppMOlbn

Page 216 of 253

000

09/21/064:56 AM

Two site HACMP cluster with GLVMmirroring

Site A TCP/IP WAN Site B

PV1.hdisk7 PV3=hdI.k9 PV3.hd1sk5 PV1 =hdktk7PV2-hdisk8 PVlshdh}kl0 PV4*hdi8k8 PV2.hdisk8

Node A Node B

PV 41 PV 1 PV2INS Joe

Real copy #1 Virtual Copy #2 Virtual Copy #1 Real Co #2

One volume group actually spans both sites. Each site containsa copy of mission critical data. Instead of extremely long diskcables, a TCP/IP network and the RPV device driver are usedfor remote disk access

Redbooks Workshop

III

Page 217 of 253

ibm com 'nrd ,,R= uU,nat ma lcch ur ^! Sappnrt Organ

GLVM - How it worksSite A -- HACMP cluster

=Redbooks Workshop

Page 218 of 253

Site B

09/21 /064:56 AM

New HACMP Network Types

XD-data• Similar to ethernet network type but has larger values for heartbeat

parameters.• Group bound service IP labels are not supported (therefore no IPAT)• Responsible for GLVM mirrors data traffic• In HACMP 5.3 (and HACMP 5.2) there can be only one XD-data

network• Node-bound service IP labels should be defined on XD_data network• This is HACMP/XD GLVM's counterpart to the Geo_Primary network

type, used with HAGEO

Redbooks Workshop

A"N

Page 219 of 253

0^ 09/21 /064:56 AM

New HACMP Network Types - heartbeat

(cont.)

XD_rs232• This is similar to rs232 network type, except the heartbeat parameters

are modified to allow for the longer distance between sites. (This isHACMP/XD GLVM's counterpart to the Geo_Secondary network type,that is used with HAGEO)

XD_ip• This is similar to ethernet network type, except the heartbeat

parameters are modified to allow for the longer distance between sites.XD_ip networks do not support IP address takeover and cannot beused for GLVM data mirroring

Redbooks Workshop

Page 220 of 25309/21/064:56 AM

HACMP Network Types

eIN

HACMP/XD GLVM requires one XD_data network for datamirroring

Recommended to configure additional heartbeat networks• At least one other heartbeat-only network between the two sites is

highly recommended in order to provide the ability to distinguishbetween a genuine remote site outage and a failure of the XD-datanetwork

• The XD_ip and XD_rs232 network types are provided for this purpose

RPV communication uses TCP port 6192

Redbooks Workshop"

Page 221 of 253

09/21/064:56 AM

Sample HACMP and GLVM cluster

Boston Austin

SERIAL NetworkXD ra232

XDRPV Service P Label RPV

krNode 1 3NodeXDdataStdby P Lbel

PV hdlaki 1.1017 PVDisk May AcrhCbIQ

Di0skk Array

TCPM WANffi3 XD_data Network Y3

S

)(D-dd. XD_dataService IP Label Service IP Label

Node 2 1.1.1 1.1.1 .aNode 4 -.1

XD_dda XD_dedeStarKby P Label Standby P Label

1.10.1. 2

SERIAL Ndwpk

Redbooks Wo kshop,, r

Page 222 of 25309/21/064:56 AM

Quorum for GMVG

Recommended to disable quorum• If quorum were enabled in our example configuration, then a

site failure would cause half of the disks to fail. This wouldresult in a loss of quorum, which would cause the volumegroup to be taken offline at the surviving site.

• The disadvantage of disabling quorum is that you lose out onthe benefits of quorum checking . This puts more burden onthe system administrator to avoid scenarios where theapplication workload may use a stale copy of the missioncritical data.

Hedbooks Workshop

O-N

Page 223 of 253

Forced varyon of a GMVG

Recommend to set Force Varyon to true• If quorum is enabled, a majority of disks must be available in order to

varyon the volume group. When quorum is disabled, all of the disksmust be available in order to varyon the volume group. This meansthat a normal varyon will fail if the remote site is down. This problemcan be avoided by configuring HACMP to perform a forced varyon ifthe normal varyon fails

• The disadvantage of forced varyon is that it increases the possibility ofrunning the application workload using a stale copy of the missioncritical data

Redbooks Workshop

Page 224 of 25309/21 /064:56 AM

PON

Resource recovery if XD data is down

HACMP does not force varyon the GMVG if XD_data networkis down

• HACMP event processing detects that the XD_data network is downand will not automatically perform a forced varyon of the GMVG.

• HACMP event processing flags an error failing to varyon GMVG

Manual Recovery:• Sync GMVGs manually - to synchronize the data.• Force varyon the volume group• Instruct HACMP to continue

Redbooks Workshop

Page 225 of 253

09/21/064:56 AM

GLVM security considerations

It is the system administrator's responsibility to secure theRPV client-server network traffic between the sites

IBM recommends the IP Security (IPsec ) feature of AIX 5.3 fornetwork security

RPV communication uses TCP port 6192

NWO,

Redbooks Workshop

Page 226 of 25309/21/064:56 AM

GLVM enhancements in HACMP 5.4

Initial release characteristics:• Single data mirroring network only• Synchronous mirroring• Supports filesystems or raw logical volumes

HACMP 5.4: Support for multiple data mirroring networks• Up to four networks now supported (XD_data)• Improves overall reliability• Potentially improves throughput and performance (w/ multiple

networks)

Support for enhanced concurrent mode volume groups• Concurrent volume group access within one site with data being

mirrored to the other site (local conc. RG)• Does not support fast disk takeover• Does not support concurrent access from nodes at both sites

simultaneously

Redbooks Workshop

Page 227 of 253

09/21/064:56 AM

Configuring Enhanced Concurrent GMVGsRemote Physical Volumes can be included in Enhanced Concurrent volume groupswhen using HACMP

Use these Resource Group Policies to define a concurrent HACMP resource group:

n Inter-Site Management Policy: Prefer Primary Site or Online On Either Site

n Startup Policy: Online On All Available Nodesn Fallover Policy: Bring Offline (On Error)n Fallback Policy: Never Fallback

Redbooks Workshop

III

Page 228 of 253

114

09/21/064:56 AM

Migration considerations, DARE, Limitations

Migration to HACMP/XD GLVM 5.4 Is supported from prior releases - HACMP/XD 5.2and HACMPIXD 5.3

New features cannot be exploited until all cluster nodes are at HACMP/XD 5.4

Requires AIX 5.3 TL4

fRedbooks Workshop

Page 229 of 253

09/21/064:56 AM

ibm.com rdwok, Int-n:ri on^i Teel ii 3; Supiv^rt O^ganizsriioi

Problem determination

The RPV client / server use bsdlog () logging function used inkernel device drivers

Enable to troubleshoot as follows:• kern.debug /tmp/syslog rotate size 10m files 7 compress

Cluster Manager log /tmp/clstrmgr.debug

Event Script Output /tmp/hacmp.out

r./

Redbooks Workshop

Page 230 of 25309/21/064:56 AM

SummaryHACMP/XD GLVM 5.4 expands upon the function introduced with the initial release ofGLVM in October, 2005

Fills In more of the equivalent function of HACMP/XD HAGEO needed to allow it to bereplaced

TCP/IP based - virtually no limit on distance

GLVM with HACMP provides DR solution

FRedbooks Workshop

Page 231 of 253

09/21 /064:56 AM

mm

Questions ?

rRedbooks Workshop

Page 232 of 25309/21/064:56 AM

ibm.com/redbooks

I

HACMP V5.4 for Linux on System p

Redbooks WorkshopIBM ITSO - International Technical Support Organization

Page 233 of 253

/' 09/21 /064:56 AM

International Technical Support Organization

HACMP V5 .4 for Linux on powerPC

ibm.com Yes, it runs on Linux!

Redbooks WorkshopIBM ITSO - International Technical Support Organizotlon

ibrwcomtrarihooks 1 International Technical Support Organization

HACMP for Linux on PowerWhy Linux ?

- Customers may have several platforms for middle and edge tier applications• Different platforms, different applications , different availability requirements

- Linux is an emerging market - IBM is the world's leader in linux

Why Linux on Power ?- More applications available for Linux on Power than any other platform

Why HACMP on LOP ?- Common HA solution - enterprise wide

- Common look and feel (configuration , management , debug , support)

- Preserves customer investment in HA skills as they grow up and out

Complete support for the entire System p family

Rabooks Wwkdlop

Redbooks Workshop Page 234 of 253IBM ITSO - International Technical Support Organization

r,

eMN

ibm come:, it ,k nl,^rnatm-al Tr,chnical Support O

Over 50 industry leadership benchmarks*

A L.

M

p5-595The IBM System P5 family of systems

OpenPower Editions: p5-550Express

p5-570

p5570Express

p5-575

p5-590

Up to 32-core1.65 GH POWERS

bmxomlr.:- li„^,.E,^, li t:rnaf^.cr al 1ei .h: :al Suppc t O<<;anr_.ai!on

HACMP for Linux on P - what 's supported• Red HatT*' Enterprise Linux (RHEL) 4 or

SUSETM' LINUX Enterprise Server (SLES) 9

• 8 nodes

• Applications- Process monitoring- Custom monitoring

- IP Address Takeover

- IPAT via aliasing

• Limited discovery of networks

• Cluster Synchronization and rsh/rexec via clcomd

• New Linux Installation and Administration Guide

• Clinfo API (local node only)

Rad6oolcs wlmbMlop

Redbooks Workshop Page 235 of 253IBM ITSO - Intematia,al Technical Support Organization

., ,, red & i J1,

SuSE

Ibm com/irdbooks International Technical Support Organization

HACMP for Linux on P - differences from AIX• Disk takeover

- No standard "LVM" on Linux

- Use GPFS or NFS

• Discovery and verification- Manually enter node , network and interface information- No smart assist(s), configuration assist(s)

• XD (disaster recovery) features

• AIX specific features like WLM, Error Notification , GLVM, smit

• DLPAR, CUoD

More features will be added as testing continues

Reftedes Woirkshop

ibm.comlreaYrooks International Technical Support Organization

nstallation• 5 RPMs:

- hacmp.doc.pdf and hacmp . doc.html - contain all Linux and AIX books

- hacmp . client and hacmp . server - runtime

- hacmp . license - software license agreement

• Accept the license agreement- Install hacmp . license

- Run /usr/es/sbin/clusterlnstall_hacmprpm has no built in license acceptance option - install_hacmp runs the IBM LAP then installs theremaining RPMsSpecify "-y" to accept the license agreement automaticallySpecify "-d <path>" to specify a different path to RPMs, „ , j

• Installation makes all necessary changes to s

• Configure and run WebSMIT- New configuration assist supports Apache v1 and

Redbooks Workshop Page 236 of 253IBM ITSO - International Technical Support Organization

0

:onfiguration• Websmit provides access to all configuration and management tools , cluster status

and documentation.• Initial setup

- Configure cluster - provide a cluster name- Configure networks - define ethernet, token ring and rs232 networks

- Configure network interfaces and devices - NIC information (IP, netmask) and tty

- Configure service IP labels - Extended Resource Configuration Path

- Configure Application servers and (optional ) monitors - start/stop/monitor scripts

- Configure Resource Groups - select options and add resources

- Syncrhonize changes - copy application scripts to all nodes

• CSPOC management options- Start/stop cluster services

- Move service IP between interfaces

- Resource group online/offline/move/monitor

- Log viewing and management

Redboaki Woikihap

e"N

bm comh-„Ao^ k,, Intern uonal Technical Support Orel

Log files and troubleshooting• Logging is indentical to HACMP for AIX

- Key log files - hacmp . out, cistrmgr .debug , clcomd.log

- Log file redirection lets you change default log location

- Event summary collection and viewing

- Log file cycling limits disk space used

• Command line information and status tools- clRGinfo

- clstat (single display mode only)

BM

• clsnap collects log files and configuration information for problem reporting

• Many system commands are similar to AIX- ifconfig, netstat, df

Redbooks Workshop Page 237 of 253IBM ITSO - kMematlonal Technical Support OrgantzAt

ibm.comlr^r.br,oks Intornatiomil Technical Support Organization

Summary• HACMP V5.4 is the first version to support multiple platforms

• SUSE and Redhat on System p servers

• Subset of features available on AIX- The feature list will grow but will always be a subset

• Applications and IP address takeover - up to 8 node clusters

• Websmit for configuration, management , status and documentation

., ,, red h a t

IBM System.RI

i Redbooks Workshop Page 238 of 253IBM ITSO - International Technical Support organization

SuSE

eN,

09/21 /064:56 AM

Page 239 of 253

Break

Redbooks Workshop

Page 240 of 25309/21 /064:56 AM

Ibm.com/redbooks Hardware Validated for HACMP V5.3

Redbooks WorkshopIBM ITSO - International Technical Support Organization

Page 241 of 253

r 09/21 /064:56 AM

Why Validate Hardware w/ HACMP?

Provides a broader range of flexible configuration options on allplatforms , including a greater number of:

n Node interconnects and protocolsn Storage systemsn Disk interconnects

Ongoing process that ensures support of latest AIX/System p featuressuch as VLAN, VSCSI and Micropartitioning

IBM conducts significant and thorough validation to assure compatibilitywith latest hardware and AIX releases

Other HW and SW vendors provide validation for their own productsAs HACMP becomes key to driving additional sales , more hardware isnow validated with HACMP

This presentation only shows hardware added to the list of hardwaresupported with HACMP within the past year

liedbooks Workshop

.

Page 242 of 25309/21 /064:56 AM

System p Servers

n IBM System p5-505 (9115-505)

n IBM System p5-510 / 510Q (9110-51 A)n IBM System p5-520 / 520Q (9131-52A)n IBM System p5-550 / 550Q (9131-55A)n IBM System p5-560Q (9116-561)

n IBM System p5-570 (9117-570)

n IBM System p5-575 (9118-575)

Redbooks Workshop

Page 243 of 253

09/21 /064:56 AM

Other Servers

n IBM eServer BladeCenter blade servers

8842-42U and 8842-4TU models (blades)IBM eServer BladeCenter JS20 chassis

> IBM eServer BladeCenter T chassis

n IBM IntelliStation POWER 285 Express> 64-bit deskside workstation with POWER5+ processor

Redbooks Workshop

Page 244 of 25309/21/064:56 AM

AIX 5L V5.3 Virtualization

Micropartitioningn Micropartition : A partition with a sub-processor allocationn Shared processors are allocated in processing units as small as .01 of a processorn Minimum of 0 . 1 processors can be configured for an HACMP node

Virtual LANn Virtual Ethernet Adapter allows partitions to communicate within a managed system without using a

physical Ethernet adaptern Shared Ethernet Adapter connects internal and external VLANs using virtual Ethernet adapter with

"Trunk Adapter" attributen HACMP only supports IPAT via Aliasing with Virtual LAN

Virtual SCSIn Virtual I/O server owns physical resourcesn Implemented as client/server relationshipn Client partition sees standard SCSI devicesn Client partition can boot from virtual SCSI diskn HACMP supports only Enhanced Concurrent Mode volume groups with Virtual SCSI

Redbooks Workshop

Page 245 of 253

09/21/064:56 AM

Storage: Subsystems

IBM System Storage DS4800n Fibre-attached storagen Formerly known as "FAStT"

IBM DS4000 EXP810 Storage Expansion UnitIBM TotaiStorage EXP24

n Scalable, SCSI-attached storagen Model 7031-D24

IBM TotaiStorage SVC V2. 1 Storage softwaren San Volume Controller for storage virtualization

Redbooks Workshop

`./

Page 246 of 25309/21/064:56 AM

Storage: Geographic Mirroring

IBM TotalStorage MetroMirrorn IBM TotalStorage DS8000n IBM TotalStorage DS6000n Combination DS8000 and DS6000

FRedbooks Workshop

Page 247 of 253

09/21 /064:56 AM

Storage: Adapters

n IBM PCI-X Dual Channel Ultra320 SCSI RAID Adapter (FC 5703, FC1975)n IBM PCI-X Dual Channel Ultra320 SCSI RAID Blind Swap Adapter (FC571 1)n IBM PCI-X DDR Dual Channel Ultra320 SCSI RAID Adapter (FC5737, FC1913)n IBM PCI-X DDR Dual Channel Ultra320 SCSI Adapter (FC5736, FC1912)

n IBM 2 Gigabit Fibre Channel PCI-X Adapter (FC1957, FC1977)

Redbooks Workshop

Page 248 of 25309/21/064:56 AM

Network Adapters

InfiniBand now supported with HACMPn InfiniBand GX HCA Adapters(FC1809, FC1810, FC1 811, FC1812)

Gigabit Ethernet adapters supported with HACMPn IBM Gigabit Ethernet-SX PCI-X Adapter (FC1978)n IBM 10/100/1000 Base-TX Ethernet PCI-X Adapter (FC1979, FC1959)

n IBM 2-port 10/100/1000 Base-TX Ethernet PCI-X (FC1983, FC1990)

n Dual Port Gigabit Ethernet-SX PCI-X Adapter (FC1984)n IBM 4-port Ethernet Adapter (FC1954, FC5740)

10 Gigabit Ethernet adapters supported with HACMPn IBM 10 Gigabit Ethernet-SR PCI-X Adapter (FC1981)n IBM 10 Gigabit Ethernet-LR PCI-X Adapter (FC1982)n IBM 10 Gigabit Ethernet-SR/-LR Adapters (FC5718, FC5719)n 10 Gigabit Ethernet-SR/-LR PCI-X 2.0 DDR Adapters(FC5722,

FC5721)Redbooks Workshop

Page 249 of 253

09/21 /064:56 AM

Resources

Flashes are posted within IBM at:n htto://www.ibm.com/suDDort/techdocs/atsmastr.nsf/Web/Flashesn IBM Employees can check SYSGROUPNEWS category of MYNEWS.

Flashes are available to everyone at:n The online product sales manual:

> htt-o://www.ibrrm.com/common/ssi/Ol/Access.wssn http://www.ibm.com/support/us, Search on HACMP + APAR, then sort by "Date -

newest first"

IBM Redbooks are available at:n http://www,redbooks.ibm.com/

Feedback and suggestions can be addressed to:n hafeedbk@us.ibm.com

Redbooks Workshop

Page 250 of 25309/21 /064:56 AM

Questions ?

9•

Redbooks Workshop

Page 251 of 253I

End of Presentation

Please DO NOT forget evaluations!!!

Redbooks Workshop

Page 252 of 25309/21 /064:56 AM

09/21/064:56 AM

Page 253 of 253