The Use of HDS’s and EMC’s DASD Copy Services at Gary Spencer Worldspan

The Use of HDS’s and EMC’s DASD Copy Services at

Gary Spencer

Worldspan

The Use of HDS’s and EMC’s DASD Copy Services at Worldspan

Terminology•Target volume•Source volume•Native test system•Periodic refreshed test system•Daily test system•Read/write VM test system


External Control Unit Copy•“Remote copy”

•Runs external of host •Full volume copy•“Sync’ed” up•Synchronous (semi-sync, adaptive copy) •Suspend/split•Point in time copy•Incremental refresh (copy)•Restore capabilities •Higher Impact on Host

Source Ctl Unit

100

To Hos

t

Fiber connection

Target Ctl Unit

101

May

be T

o Hos

t # 2

xxx

yyy


Internal Control Unit Copy

•“Local Copy”•Runs external of host •Very fast – low host impact•Full volume copy•“sync’ed” up•Suspend/split•Point in time copy•Incremental refresh (copy)•Restore capabilities

Ctl Unit

100

101

To Hos

t

Target volume are “generally” not addressable to the same host as the Source volumes. (IOCP’s are generally used at Worldspan to control access)

May

be T

o Hos

t # 2

xxx

yyy


Vendor Specific Comments

Product Names

•TrueCopy - (HDS’s remote copy)•Symmetrix Remote Data Facility - SRDF (EMC’s remote copy)•Shadow Image - (HDS’s local copy)•TimeFinder - (EMC’s local copy)

Target/Source pairing

• TrueCopy and Shadow Image may be defined in the host or by the CE in the control unit

•SRDF source/target pairs may be defined in the host and must be defined by the CE in the control units BIN file.

•TimeFinder pairs are call Standard volumes/BCVs (Business Continuance Volume)•Must be defined in the control unit as “Standard” volumes or as “BCV’s”•Source/Target (standard/BCV) pairs are defined in user controlled host tables•Copies may be from any “standard” volume to any “BCV”.


Worldspan’s Three Production TPF systems

•OSS - Delta Airline’s flight control and flight operations system

•Deltamatic (RES) - Delta Airline’s reservation system

•GDS - Worldspan’s agency reservation system Northwest Airline’s reservation system

Use of HDS’s Copy Services on Worldspan’s GDS

Initial Goal – speed up test system buildFirst use of copy services to build test systemsFirst use of “local” copy

Use of HDS’s Copy Services on Worldspan’s GDS The Production GDS System

•GDS is a 8 way loosely couple TPF system•GDS is Worldspan’s agency reservation system •GDS is Northwest Airline’s reservation system

Statistics•GDS has 640 prime mods (1280 active mods)•Each physical control unit has 45 or 46 active modules•The I/O rate is approximately 250,000 + I/O’s per second

Fir

eW

all

28 HDS 7700E’s3 LCU’s per CU HDS

7700E HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E HDS

7700E

HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E

HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E

HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E

Use of HDS’s Copy Services on Worldspan’s GDSPrior GDS Test Environment

“Large Test System”

Prior GDS test environment- ltst3 VM Systems

Tape Restore

Large Test (LTST)4 7700E’s

HDS 7700E HDS

7700E HDS 7700E HDS

7700E

Any GDS production processor

Large Test System runs read/write to VM during the week – this read/write system is a customer system.Down Time is considered an impact.System is refreshed each Sunday from Saturday capture tapes.Native testing was restricted to weekends before the refresh (some exceptions allowed)The LTST DASD has connectivity to each of the GDS production processors.

Use of HDS’s Copy Services on Worldspan’s GDSCurrent GDS Test Environment

Large Test

HDS LTST 7700E - 1

•4 physical CU’s•2 copies of the production database•TrueCopy’s target from the production system rotates each week.•Shadow copy the new TrueCopy’ed test system to other address range•Unique VSN for each address range

HDS 7700E

Addr range xx00-xx3f


Use of HDS’s Copy Services on Worldspan’s GDSCurrent GDS Test Environment

Large Test

Current GDS test environment - ltst

3 VM Systems

4 7700E’s 1 set of shadow volumes

HDS 7700E HDS

7700E HDS 7700E HDS

7700E

Any GDS production processorOr VM LPAR

Fir

e

Wall

HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E

HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E

14 7700E’s1 set of shadow volumes

Shadow volumes

Rotating TrueCopy Targets

Low addresses

High addresses

Prod volumes

Native or VM read/write testing use the non-ltst volumes

Step one – shadow productionStep two – TrueCopy to LTSTStep three – build LTSTStep four – IPL new databaseStep five – shadow LTST

Use of HDS’s Copy Services on Worldspan’s GDSPrior GDS Disaster Recovery

Prior GDS GDS Disaster Recovery

Tape Restore

Use of HDS’s Copy Services on Worldspan’s GDSCurrent GDS Disaster Recovery

Current GDS Disaster Recovery

Fir

eW

all

28 HDS 7700E’s3 LCU’s per CU

HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E

HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E

HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E

HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E HDS

7700E HDS 7700E

•A shadow image copy of the GDS database each night.•System may be restored from the shadow’ed volumes.•Groundwork for a no tape capture

shadow volume

Productionvolume


TPF Software Use

TPF software use

•First use of HDS’s TPF software for TrueCopy and Shadow Image•Originally, Shadow Image and TrueCopy activities were done manually by CE

•Production control improvements•TPF Coverage staff controls the activity•Automation is able to run the “copy” procedures.•The TPF software eliminated the need for logging tapes during the database split

•TPF software now controls all TrueCopy and Shadow Image activities for test system builds and DR support

•Software was new

Use of HDS’s Copy Services on Worldspan’s GDSBenefits to GDS

GDS payoffs

1. Test System benefitsa. Improved LTST down/build time (18 plus hours to less than 2 hours) b. Removing tape restore from LTST build eliminated tape prep/sort/errors time.c. Better and More Native Testing capabilities

i. No down time for native testingii. Native testing at any time – no impact on ltst databaseiii. Native test time setup takes about 30 minutes (using some of VM’s LPAR’s )

d. Read/write ltst system under VMe. Quick copies of small test systems (8 mod test system)

2. Disaster Recovery benefits.a. Laid groundwork to eliminate tape captureb. Production database can be restored (quickly) from the shadow image copy.

3. New DASD is often formatted using the control unit’s copy services.

4. Proved TrueCopy and Shadow Image technology for TPF use at Worldspan5. Proved TrueCopy and Shadow Image TPF software controls for use at Worldspan


GDS Current Development

Current development

• Working with HDS to use shadow image to split the production database on a

nightly basis • Working with HDS and with Worldspan’s coverage, production control staffs

to improve automation and production control issues• Training Worldspan Coverage Staff

Possible Future GDS Development

• Another set of shadow volumes on the large test system (allow to create a daily test system)

• More shadow volumes on the production database (allow multiple backup copies of the production database)

• Continue to work towards a capture without tapes.

Use of EMC’s Copy Services on Worldspan’s Deltamatic System

•Most ambitious goals•Most work done initially with TPF software•EMC provided an onsite consultant•First Use of Timefinder

•SRDF was already in use on OSS

Use of EMC’s Copy Services on Worldspan’s Deltamatic System The Production Deltamatic (RES) System

Fir

eW

all

•Deltamatic is a 6 way loosely coupled TPF system•Deltamatic is Delta Airline’s reservation system

EMC 8430

Statistics•Deltamatic 592 prime mods (1184 active mods)•Each EMC 8430 has 74 active modules•Each EMC 5830 has 53 or 54 active mods •The I/O rate is approximately 115,000 I/O’s per second

EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430EMC

8430

EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830

11 EMC 5830’s2 LCU per CU


Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemPrior Deltamatic Test Environment

Prior Deltamatic test environment2 VM Systems

PMR

Tape Restore

•Native testing restricted to quarterly DR test weekends•No daily test system•Intermediate step to replace PMR with 5430’s and manual SRDF•Brought in the 8730 to upgrade the testing facilities

Deltamatic Test System (RTS)

Fir

e

Wall

EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830


EMC 8730

SRDF LINK

EMC 5430

Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemThe EMC 8730 Control Unit for Three Test Systems

EMC 8730 - 1

•10 LCU’s•3 copies of the production database•Each LCU has three 64 address ranges•Unique VSN for each address range•11 production control units (22 LCUs) mapped to 10 test system LCUs.

EMC 8730



Addr range xx80-xxbf

LCU 1



Addr range xx80-xxbf

LCU 10

…

Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemThe EMC 8730 Control Unit for Three Test Systems

EMC 8730 - 2

•First address range in each LCU•The daily test system, a base for VM VPAR systems •Target volumes for SRDF from the production system (refreshed nightly)• TimeFinder’s standard volumes

•Second address range in each LCU•The periodic test system, a base for VM VPAR systems •These devices are mirrored•TimeFinder’s BCV group 1

•Third address range in each LCU•The native test system or a read/write test system for VM •TimeFinder’s BCV group 2

Addr range xx00-xx3fTimeFinder std volumes

Addr range xx40-xx7fTimeFinder BCV group 1

Addr range xx80-xxbfTimeFinder BCV group 2

8730’s LCU x

Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemCurrent Deltamatic Test Environment

Current Deltamatic test environment

2 VM Systems

SRDF linksStd to std

Fir

e

Wall

EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830

11 EMC 5830’s2 LCU per CU1 std, 2 bcv’s each volume

EMC 8730

2 Deltamatic production processors

standard

BCV 1

BCV 2

Loosely coupledNative testing

•VM VPARS Daily test system•VM VPARS Periodic test system

•No more tape restore •VM read/write test system

Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemPrior Deltamatic Disaster Recovery

Prior Deltamatic Disaster Recovery

Fir

e

Wall

EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430


…

EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430


SRDF linksContinuous, semisync

Wor

ldsp

an D

ata

Cen

ter

DR

Site •Break SRDF links for …

•DASD for quarterly DR test exercise•Quarterly native test DASD•Selective database recovery from

capture or logging tapes

Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemCurrent Deltamatic Disaster Recovery

Current Deltamatic Disaster Recovery

ProductionEMC

5830 and 8430

Standard volumes(production active volumes)

BCV 1 Copy to from standard

BCV 2 Copy to from standard

Alternate TimeFinder copy from standard volumes to BCVs

Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemCurrent Deltamatic Disaster Recovery

Current Deltamatic Disaster Recovery

Fir

e

Wall

EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430


EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430EMC

8430EMC 8430


SRDF linksContinuous, semisync

Wor

ldsp

an D

ata

Cen

ter

DR

Site

•Control DR SRDF stop/start with TPF software•In production, we can “capture” the database rotating between two BCV’s.•At the DR site, we can “capture” the database rotating between two BCV’s.•Selective database recovery from the daily test system•Production system may be restored from the BCV’s

EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830

EMC 5830EMC

5830EMC 5830EMC

5830EMC 5830EMC

5830


Each standard has 2 BCV’sEach standard has 2 BCV’s

Each standard has 2 BCV’s

Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemTPF Software Use

TPF software use

•First use of EMC’s TimeFinder software•EMC’s TPF SRDF software had been used on OSS•Similar advantages as the GDS

•TPF Coverage Staff controls the activity, software controls are automated

•No tapes (including logging tapes) required for test system refresh

•The TPF software controls all SRDF and TimeFinder activities for test system builds and DR activities

•Working to use remote TimeFinder for DR copies.

•TimeFinder must be started with host software (not a routine task for a CE)

•The software can be used to restore the production database from the BCVs..

Use of EMC’s Copy Services on Worldspan’s Deltamatic SystemBenefits to Deltamatic

Deltamatic payoffs

1. Test System benefitsa. Native Testing capabilities on request (no longer must wait for a quarterly DR test)b. Daily Test System

i. Improved problem analysis from 6 wk system c. A read/write VM system for high write testing d. Periodic Test System build much easier, faster (no tapes)

2. Disaster Recovery benefits.a. Coverage controls DR SRDF activity with TPF softwareb. Laid groundwork to eliminate tape capturec. Have multiple days of copies of the database on DASDd. Daily Test System

i. Selective database restoreii. Eliminate the need for continual logging tapes (although not done yet)

3. Proved TimeFinder and remote TimeFinder technology for TPF use at Worldspan’s 4. Proved EMC’s TPF TimeFinder software for use at Worldspan’s

Use of EMC’s Copy Services on Worldspan’s Deltamatic System

Deltamatic Current Development

Current and future development

•Completing the implementation in using BCV’s for database capture/restore•Working with EMC to complete implementation of remote TimeFinder use•Working with EMC to use TimeFinder in the production system•Working with EMC to improve automation and production control issues•Training Worldspan Coverage Staff

• Work towards a no tape capture solution• More BCV’s for more “back” days of capture

Possible Future Deltamatic Development

Use of EMC’s Copy Services on Worldspan’s OSS

•Limited Goals due to older Technology used for this system •EMC 5230’s – no TimeFinder •Primary goal was to set up a native test environment

•Initial use of EMC’s TPF software

•Work similar to the Deltamatic work

Use of EMC’s Copy Services on Worldspan’s OSS The Production OSS System

Spl

it

Site

(Disaster Recovery Location)(Worldspan Data Center)

•OSS is a Uni-Processor TPF system•OSS is Delta Airline’s flight control and flight operations system

EMC 5230

EMC 5230

EMC 5230

EMC 5230

Statistics•OSS has 72 prime mods (144 active mods)•Each physical control unit has 36 active modules•The I/O rate is approximately 3500 I/O’s per second


Use of EMC’s Copy Services on Worldspan’s OSSBenefits to OSS

OSS payoffs

Similar to the Deltamatic benefits.

By adding a new control unit for testing – and using EMC’s copy services -

1. Test system has native testing and a daily test system

2. Disaster Recovery gained from the daily test system and easier setup for DR testing. Also, the groundwork was started to help eliminate tape capture

3. Proved SRDF technology and TPF software controls for TPF use at Worldpan.

With new technology - OSS could make use of TimeFinder in similar ways as Deltamatic.

Use of EMC’s Copy Services on Worldspan’s OSSRecap of the OSS work

Spl

it Site

(Disaster R

ecovery Location)

(Wor

ldsp

an D

atac

ente

r)

recap of oss work

Production OSS Processor

EMC 5230

EMC 5230

EMC 5230

EMC 5230

EMC 5230

DR OSS Processor

RES Processor

PMR

EMC 5230

SRDF links

Tape Restore

2 VM Systems

Daily test systemNative test system

SRDF links

Manual process

AMOD UP

•DR duplicate DASD•DASD for DR test exercise•Quarterly native test DASD


Implementation Experiences

Common Experiences between Vendors or between TPF Systems

Production I/O rate impacted copy servicesMost copy service problems were found during production useStalled modsHad to switch (or will switch) to use local copy only on production. Then remote copy from that local production copy.

The TPF software commands were done manually until the concept was proven. At that time the software commands to control the copy activity were automated Many Automation/Production Control issues

Coverage education. Specifically awareness that this activity runs outside of TPF ’s control. CE’s education.

Outstanding Vendor and Vendor’s programmer support. Code was high quality - errors found were generally due to configuration issues. Vendor’s would provide off hour telephone or onsite support.

Building source/target tables required vendor help.

OCO code delayed testing, especially during production testing (examples : waiting for programmers to resolve problems, waiting for new code).


Implementation Experiences

Experiences with the Delta Systems and EMC

Need for rebuilding BIN files during implementation and testingEMC provided a full time onsite consultant for the TPF software implementation and to help set up the use of TimeFinder

Experiences with the GDS and HDS

TPF software was new


What Worldspan Gained

•Test Systems•Easier access to native testing on all three systems•A “daily” test system on both of the Delta Airline TPF systems•Faster “Periodic” Test system build for GDS and Deltamatic•Test systems built using no tapes (reduced tape activities)•Less test system down time (“or” test systems are more available)•Daily test systems, native test systems are increasing the quality of testing

•Disaster Recovery•Easier (and faster) to set up a DR test for Delta Airlines.•Quick access to a 24 hour old database for selective database restores•All three systems have a DASD copy of the database to restore from• Started Groundwork for a no tape capture

•More ?•As the programming groups become experienced using these test environments, we have seen them exploit this tool and continue to come up with new ways to use these systems. Today on GDS we are seeing extensive recoup/capture testing.

Documents

The Use of HDS’s and EMC’s DASD Copy Services at Gary Spencer Worldspan