Considerations for using TSM in a SANtsm-symposium.oucs.ox.ac.uk/2003/papers/SteveStrutt.pdf · in HACMP cluster TSM 5.2.0 with HACMP on AIX – Devices defined in ‘startserver’

1

IBM Software Group

August 2003 © 2003 IBM Corporation

Considerations for using TSM in a SAN

Steve Strutt, Tivoli Software, IBM [email protected]

Sept 2002Revision 5

IBM Software Group | Tivoli software

Considerations for using TSM in a SANl © 2003 IBM Corporation2

AgendaSAN Exploitation - LAN-Free backupPerformance characteristics

RequirementsLAN, Hardware, Software, Device, dependencies

SAN considerationsDevice fail-overHBA considerationsSAN design considerationsDevice addressing considerations

Going LiveTesting, Diagnosing ProblemsHints and Tips

Question and Answer

2



LAN-free BackupAdvantages:- client data can be local or SAN-attached- transparent to application/database- takes backup traffic off the LAN- reduces CP cycles on backup server (no I/O)- faster speed (usually)- only one backup server needs administration

Disadvantages:- still requires CP cycles on client for backup I/O- careful scheduling to avoid tape drive contention

(or exploit disk pooling)

TAPE

FC device

DISK

LAN

D A TAD A TA

DATA

Direct to tape

Disk pool staging



Performance characteristicsLAN-Free is not necessarily faster

Only network eliminated as bottleneck

Could be other bottlenecksTape drives, Disk subsystemData types

good performance for large files and databasessmall files, performance limited by file system and TSM

architectureLAN-Free to tape

Potentially better performance for large files, as bottleneck becomes file system or tape device

Small files cause tape drives to stop-start more and drives drop out of streaming mode.

LAN-Free to diskIdeal for small files, no stop-start overhead

3



Customer Performance FiguresLarge UK High Street Retailer

TDP for SQL Server on ESS to 3584 LTO Backup 61.4GB hour (17MB/s) to single driveRestore 44.2GB hour (12.3MB/s) from single drive

NT Filesystem on ESS disk to 3584 LTO - small filesBackup 10GB hour to LTO Restore 6.2GB hour from LTO Could be slower than LAN if tape drives do more stop/start operations

Large UK BankTDP for Exchange to 3583 LTO

Backup 52GB/h (14.4M B/s) to single driveRestore 51GB/h (14.2MB/s) from single drive



Agenda - Requirements

Hardware – LAN

– Library support for LAN-Free

– SAN device supportSoftware

– Evolving TSM support for LAN-Free

– TSM code dependencies

4



Requirements - LAN

LAN-Free still requires LAN for meta dataFor Large files and databases

Minimal usage Small files

Maybe the same if not more meta data on LAN than data on SAN if files are very small.

LAN performance and loading still important



SAN Device support

Initially hardware configurations were certified by Tivolimany different combinations, not all could be tested

Common configurations now testedThe Tivoli view is that hardware is transparent to TSM

If the hardware vendors have validated the devices’ use in a SAN and TSM supportsthe device, then it is supported in a SAN

configuration with Tivoli Storage Manager

5



Library support for LAN-FreeNative TSM LAN-Free support for:

– 3494 Libraries

– SCSI Libraries– Controlled via SCSI control path

– SCSI connect– Fibre Channel Connect

NO Native TSM LAN-Free support for– STK and ADIC AML libraries

– Require Gresham EDT on every TSM server and Storage Agent. – STK - ACSLS, Library Station (s390)– ADIC – DAS for AML/2, AML/J



TSM evolving support for LAN-FreeLAN-Free supports

BA Client, file level

BA Client, Volume level/Image BackupPrior to 5.1.5, the tape was rewound between each volume

No support for Backup Set restoreNT/W2K

Supported from 4.1.0, NT/W2K server and NT/W2K TDPs

4.2.1 for Backup Archive ClientSun Solaris and AIX

Supported from 4.2.0, TDPs and Backup Archive ClientHP-UX

Supported from 5.1.0, Full TSM device driver supportLinux (x86, zLinux and zOS)

Supported from 5.2.0

6



TSM code dependencies

TSM 4.2 and 5.1– TSM Server and Storage Agent code MUST be at the same PTF and

patch level

– Restricts ability to roll out new codeTSM 5.2

– TSM Server and Storage Agent code only dependant at version and release level

– Independent of PTF level

– Easier to deploy and install maintenance



Agenda - SAN considerationsHigh Availability considerations

– SANs offer high availability

– Tape high availability options

– Fail over and tapeHBA and tape setupHBA SharingSAN design considerationsAttaching tape drives to SANsSAN Operational considerationsDevice addressing considerations

– Managing device addressing– Persistent Naming– Static Device Naming

– Mapping device names to devices

7



SAN’s offer high availabilityWorks well for disk

– Arrays designed to have two access paths

– Device drivers exploit multiple paths– Failover of paths

– Clustering software fails over applications and disk storage

– Effective TSM support for fail-over– DB and Log Mirrors, DB page shadowing– support for MSCS and HACMP

Does not work well for tape– Tape drives usually single connection

– SCSI Controlled tape libraries usually have a single control point

– Drivers only exploit single path

– Failover issues

Clustering

Redundancy Multiple paths



Tape high availability options

3590 Dual SCSI/Fibre ports– AIX ATAPE driver can failover to second pathIf the primary path, say rmt0 is not available, the driver will use other paths/device

names via an alternative HBA/device port to access the same device. This is described in the 3590 manual, ‘IBM TotalStorage Tape Installation and User

Guide’. ftp://ftp.software.ibm.com/storage/devdrvr/Doc/

358x LTO libaries– New option to have two library controller LUNs on different paths

– Use two different HBA’s/Switches– Only supported with AIX ATAPE driver– Still single path to drives

– Divide drives between two HBA’s and switches– Still access library, reduced number of drives

Good news for AIX

8



Fail-over and TapeSCSI Reserve/Release used to serialize access to drives

– Maintains integrity if device definitions are wrongOutstanding SCSI Reserves remain if server using drives go downDrives only released if:

Failed server restarted

Drive powered off and on.

SCSI Reset issued

Potential Impact on other users

SCSI Attached Tape drives can be reset using SCSI Reset

Fibre attached tape drives do not honor SCSI Resets



Supported SCSI Fail-over configurationsTSM 4.2+ with MSCS on W2K

– Devices registered in wizard, SCSI reset on fail-overTSM 5.1.5 with HACMP on AIX

– Devices defined in ‘startserver’ script, issues verifydevice to reset device

SharedDisk

Shared Tape

Shared SCSIBus

TSM Serverin Cluster

TSM Clients

IP network

9



Supported Fibre Failover configurations

TSM Serverin HACMP cluster

TSM 5.2.0 with HACMP on AIX– Devices defined in ‘startserver’ script, issues verifyfcdevice to reset device



Fail-over and LAN-Free and Tape Library SharingOutstanding SCSI Reserves remain if storage agents using drives go downDrives become unavailable to other LAN-Free/Tape Sharing users

Only released if:Failed server restartedDrive powered off and on.

Potential drive mapping issuesFC Reset issued

Potential Impact on other users

Potentially need more tape drives

Some horrible messages if TSM server tries to dismount tapes belonging to failed storage agents.

10



Failure scenario

ANR8925W Drive DRIVE0 in library ATLP1000 has not been confirmed for use by server UKSAN1_SA for over 1200 seconds.Drive will be reclaimed for use by others.ANR8336I Verifying label of DLT volume 00157D in drive DRIVE0 (MT6.1.0.1).ANR8311E An I/O error occurred while accessing drive DRIVE0 (MT6.1.0.1) for SETMODE operation, errno = 1.ANR8355E I/O error reading label for volume 00157D in drive DRIVE0 (MT6.1.0.1).ANR8311E An I/O error occurred while accessing drive DRIVE0 (MT6.1.0.1) for OFFL operation, errno = 1.ANR8469E Dismount of DLT volume 00157D from drive DRIVE0 (MT6.1.0.1) in library ATLP1000 failed.ANR9999D mmsscsi.c(1647): ThreadId<48> Volume may still be in the drive DRIVE0 (MT6.1.0.1).ANR8446I Manual intervention required for library ATLP1000.

Server running Storage Agent goes down (hardware failureFibre loss), when using a tape device

TSM makes drive ‘unavailable’



Supporting LAN-Free fail-overNo TSM support for fail-over of Storage AgentSupporting Storage Agent Fail-over

Configure two storage agents, one on each server

Different Storage Agent NamesNo dependency between TSM clients and Storage Agent

BA client and TDP failover, Storage Agent does not

BA client and TDP automatically use new SA on fail-over server

No device issues if LAN-Free client machine fails-over when not using tape devices

Devices lost if LAN-Free client machine fails-over when using tape drives

Require more tape drives

Procedures to recover lost drives

11



HBA and tape Setup considerationsCheck configuration recommendations of tape hardware vendor

– IBM recommendations

NT/W2KMax Scatter-Gather must be set to 65 or greater

Unable to write to new tapes on Storage AgentW2K creates tapes which cannot be readTSM will check for this in 4.2.1.11 and higher levels.

Issues message, unable to use drive. W2K and LTO use 5.0.2.4 or higher level of Ultrium LTO driver.



HBA SharingSharing of disk and tape on same HBA not always supported by hardware vendors

Disk OK

Access to tape drives lost under high workload conditionsdrives timeout, go offline

IBM supports disk and 3590 on AIX with 6227/8 adapter under moderate workloads

IBM-SSG do not recommend sharing disk and tape in other configurations.i.e. LTO some evidence that it is OK in low workload environments, such as

previous NT/SQL server LAN-Free environment.

Possible solution, with new generation of adapters being certified to support dual use without restrictions

12



SAN design

Most SANs designed for disk accessdata flow is optimized for hosts <==> disks

stovepipe design, separate SAN ‘islands’

Tape backup requires flow across SANMay need additional Inter-Switch Links

More ports because of HBA Sharing issues

DiskArray

TAPE



Attaching tape drives to SANs

Most tape drives are Arbitrated Loop devicesNote some directors do not support direct attachment of FC-AL devices

Brocade switches support Public and Private Loop devices in fabric

InRange OK

McData directors do not support Arbitrated Loop Devicesrequire use of Sphereon 4300 or similar departmental switch.Or SCSI attached via Fibre-to-SCSI Gateway/Router

TAPETAPE

ED5000

4300 etc

TAPE

13



SAN Operational considerationsUsing fibre channel arbitrated loops (Hubs)

During boot up sequence LIP will interrupt tape operations for shared devices.

Rebooting servers can cause tape failures

When devices and servers are on the same loop, devices should not be rebooted while tape is in use by a TSM server or Storage Agent

Power up sequenceSAN, tape devices, then TSM Server and Storage Agents

Scheduling to avoid drive contentionStorage Agents must wait if no drives available.

Use MountWait parameter to avoid backup failuresManaging device access

Device Class, Mount Limit

Device PATHs



Device Addressing ConsiderationsAll device addresses are defined centrally on TSM server for LAN-Free

Library, Device and Path statements.Each host knows devices by different device names

Changes in device names or SCSI addresses can cause failuresRequirement to manage device addressing

All definitions must point to same physical deviceQuestion: How to map all device names to the same physical device?

Device Namelb1.0.1.3//./tape0//./tape1/dev/rmt0/dev/rmt1

/dev/rmt/0st/dev/rmt/1st

TSM DefinitionLibrary Lib1 lb1.0.1.3Drive Drive0 //./tape0Drive Drive1 //./tape1Path Drive0 /dev/rmt0Path Drive1 /dev/rmt1

Path Drive0 /dev/rmt/0stPath Drive1 /dev/rmt/1st

TAPE

TA PE

W2KTSM Server

AIX Storage Agent

Solaris Storage Agent

F C d ev ic eWWN1

WWN2

14



TapeDrives

SANGateway/Router

OS DeviceName to

TSM DeviceName

SCSI IDto

OS DeviceName

DeviceWWN

to SCSI ID

Device WWN

SCSI ID to LUN

ID1

ID2

ID3

SCSI Bus

WWN

WWN

Host

TSMOS

DeviceDriver

HBA

Managing Device Addressing in SANs - 1Host Device Names and SCSI addresses can change

devices added or removeddevices failing

A number of address mappings occur.WWN to SCSI addressSCSI address to device addressGateway/Router - device SCSI address to LUN

Described in Redpiece: Managing device addressing of SAN attached tape for use with Tivoli Storage Manager, REDP-0150-00



Managing Device Addressing in SANs - 2

Changes are more likely to happen if there are large numbers of devices and hosts in SAN.

– New devices being added, or removedSolution

– Use HBA Persistent Naming– Fixes SCSI address to device WWN

– Static device name mapping– Device Names remain unchanged– Fixed device name to SCSI address mapping

– TSM 5.2– Automatic device tracking

15



HBA persistent naming supportHBA maintains a fixed WWN to SCSI address relationshipSupport matrix

Platform Emulex Qlogic

AIX Not Applicable (use 6227/8 adapter)

Not Applicable (use 6227/8 adapter)

Windows NT/W2K

YES Yes (from 8.1.3 with

SANblade Manager) Solaris YES YES



Persistent Naming with Emulex on Windows

16



Qlogic HBA on WindowsQlogic from 8.1.3 onwards with SANBlade Manager/SANSurfer

Download management utility from Qlogic Website



Static device naming with Microsoft WindowsTSM device driver uses a naming convention which does not change as devices are added or removed

mtx.y.z.nLTO device driver uses default Windows device naming

\\.\tape0Can change if new devices added or removedWith W2K can change in-flight if devices added or removed from SANRecommendation to use mtx.y.z.n,

Use information from TSM Device Information screen to determine mtx.y.z.n device addresses

In TSM 5.1, the mtx.y.z.n name is shown.

17



Static device naming on AIX and Solaris

AIX– Device drivers automatically track devices using serial numbers

– This mapping is staticSolaris

– Static device naming convention

– Uses symbolic link to map device name to SCSI address

– Requires HBA Persistent Naming to be configured – HBA configuration file

ls –l /dev/rmt/*lrwxrwxrwx 1 root other 45 Jan 3 14:22 /dev/rmt/0mt -> ../../devices/pci@1f,0/pci@1/scsi@2/mt@5,1:mt



TSM 5.2 Automatic device trackingTSM tracks devices by serial number

– Specify S/N when defining devices– Automatic discovery of S/N when device defined

At start of each operation TSM server and SA will check the device is the one it expects it to be:

– Windows– Initiates a search for the device and changes mapping to point to

new device and then continues operation.– UNIX

– Issues message and fails operation on that device

Avoids need for persistent binding on Windows– Qlogic complex to set up. – Extra mgmt utilities. – Persistent binding options not tested by hardware vendors!

18



Mapping device names to devices -1

In a SAN all definitions for a shared device must point to the same physical device.

– Each host usually has a different device name for the device depending on the hardware and configuration.

– Only common information available on each platform is the deviceWWN and serial number.

Device Namelb1.0.1.3//./tape0//./tape1/dev/rmt0/dev/rmt1

/dev/rmt/0st/dev/rmt/1st

TSM DefinitionLibrary Lib1 lb1.0.1.3Drive Drive0 //./tape0Drive Drive1 //./tape1Path Drive0 /dev/rmt0Path Drive1 /dev/rmt1

Path Drive0 /dev/rmt/0stPath Drive1 /dev/rmt/1st

TA PE

TA PE

W2KTSM Server

AIX Storage Agent

Solaris Storage Agent

F C d ev ic eWWN1

WWN2




1. Define drives on TSM server and determine relationship between TSM device name, host device name and device WWN/Serial No.

Also determine the Element Number of the device in the library

2. For each (Storage Agent) host determine WWN/Serial # and device name for each tape device.

3. Use WWN’s/Serial # to relate device names on each host to TSM device name.

4. Define drives on TSM server using PATH statements

19



Mapping device names to devices-3TSM Device Name Drive0 Drive1 Drive…

WWN/Serial No.

Library Element Number

TSM Server

Storage Agent1

Storage Agent2

Storage Agent3

Host Device Names

……



Mapping device names to devices – Windows 1

TSM V5.1 – TSM Utilities, Device InformationShows a devices WWN and Serial Number as well as device address (mtx.y.z.n)

Also allows central discovery of Storage Agent device mappings

Requires SNIA HBA SAN Mgmt API to be installed

20



Mapping device names to devices – Windows 2 TSM Server, Device Information display with Qlogic SNIA SAN Mgmt API InstalledDevice information can also be displayed from this screen for Storage Agents in the same Windows Domain




AIX lsattr –El mtx/rmtx shows devices WWNlscfg –vl mtx/rmtx shows devices Serial Number

SolarisRelate device name to WWN using SCSI and LUN addresses

ls -l shows device name and SCSI/LUN mapping

dmesg output shows SCSI Target address to WWN mapping

ls –l /dev/rmt/*lrwxrwxrwx 1 root other 45 Jan 3 14:22 /dev/rmt/0mt -> ../../devices/pci@1f,0/pci@1/scsi@2/mt@5,1:mt

dmesg (/var/adm/messages)......qla2200-hba0-SCSI-target-id-5-fibre-channel-name="100000e00201d0d7";

21



Agenda - Going Live

Testing– testing

Diagnosing Problems– Storage agent messages

– TSM device utilities



Testing

Test after every hardware change– Changes can introduce errors which can cause failure to recover data.

Proving data integrity– TSM 5.1 includes end to end CRC checking

– use only during testing

22



Testing

Check tape hardware works reliably with TSM server in LAN configuration firstCheck TDP’s and B/A Client work on LAN firstTest each drive with each Storage Agent to check they are properly defined and accessible.

Use BA ClientANR8779E (Session: 7, Origin: UKSAN4_SA) Unable to

open drive /dev/mt1, error number=2.

Invalid device specifiederror number=16, Device Busy (SCSI Reserved to another

system)



Diagnosing problems

Storage Agent can be run in foreground session, to see all messages.

– All Storage Agent messages should be logged centrally in the server Activity Log

– Can issue commands from TSM server consolestorage_agent1: QUERY SESSION

mttest and lbtest utilities– Provided in utilities or devices directory

– test operation of tape devices or library operations.– Shows device serial numbers

23



Question and Answer

Documents

Considerations for using TSM in a SANtsm-symposium.oucs.ox.ac.uk/2003/papers/SteveStrutt.pdf · in HACMP cluster TSM 5.2.0 with HACMP on AIX – Devices defined in ‘startserver’