Phone: 603-879-9022 30 Temple St, Suite 303 e-mail ...sciinc.com/remotedba/techinfo/tech_presentations/Adventures of... · IBM DS8000, Global Mirror ... 6. Operating systems reconfigured

Copyright 2015

SCI, LLC, Nashua, NH USASoftware Concepts International, LLC.

World Class Managed Services For OpenVMS

Brad McCuskerSoftware Concepts International, LLC30 Temple St, Suite 303Nashua, NH 03060, USA

Phone: 603-879-9022e-mail: [email protected]

www.sciinc.com

A World of Difference in Database Support

Copyright 2015



From external to in-house without knowing how to spell VMS

We sold off what? Need a copy – quick!

Hardware refresh and DR via virtualization

Copyright 2015


World Class Managed Services For OpenVMS 3

Copyright 2015



Managing OpenVMS systems and databases requiring the highest

levels of performance and availability – worldwide

Copyright 2015



Copyright 2015



Healthcare sector

Hosting provider exiting the business Also their application provider

Customer needed to: Host system Assume application maintenance Continue to run their business

6

Copyright 2015



ES-45, 4P, 32GB, VMS 8.3, EVA 3000s (Chicago) COBOL, MultiNet, DCPS, SSL and Ghostscript

~500GB data to be migrated

Single System supporting 8 customers Each customer had test and prod environments This customer was largest and had 5 environments

DR: Similar configuration (Denver) Home grown, host level data replication (transactional) Not complete replication

7

Copyright 2015



Prod – Denver, CO: ES80, 6P, 48GB DR – Tacoma: ES80, 6P, 48GB Dev/Test - Tacoma: DS25, 2P, 16GB

IBM DS8000, Global Mirror between sites

OS, LP – same as source

Oversized for growth

8

Copyright 2015



IT team knew nothing of VMS

Subsidiary new to corporate infrastructure No control over resources, urgency not appreciated

Data Migration – HIPAA Difficult to determine data volume

DR requirement high priority

Plenty of managers – rarely clear who was in charge

9

Copyright 2015



January – first contact Contacted by 2 organizations (HP and PWC)

May – First detailed planning session June – Hardware received July – System buildout August – Dev/Test Migration November – Production Migration

10

Copyright 2015



Network from Chicago? Struggled to understand how much data, rate &

reliability

Hand carry from Chicago? Time consuming

Hand carry across Denver? Concern over completeness of replication

11

Copyright 2015



Hand carry across Denver Rely on replication for transactional data Push non transactional data from Chicago to Denver

12

Contingency Plan: Create full backup tape in Chicago Fly tape Chicago to Denver

This contingency saved the project!

Copyright 2015



Corporate solution – CommVault No Alpha client

Solution: Backups copied to Linux, CommVault from Linux Create savesets to disk Compress savesets SFTP to Linux

13

Copyright 2015



100GB/day – time consuming

Used PIGZ to compress Parallel gzip

SFTP problems Multinet SFTP unstable, >2GB transfers didn’t work Split the .gz file into 2GB chunks prior to transfer

14

Copyright 2015



“warm standby” Global Mirror replicates LUNS to Tacoma Including system disk

ES80 booted with simple VMS system disk Replicated LUNS mounted RO on DR system DR Event: Stop replication Reboot ES80 from replicated system disk SCI added code to startup to recognize hardware – do

the right thing for network, etc.

15

Copyright 2015



We learn (late) that Global Mirror is new to their infrastructure

VMS systems first to significantly use it.

August 2014 – We learn it doesn’t work (2.5 months before “go live”) Fix targeted for late Sep/early October

Forced to re-evaluate “home grown” replication Ultimately, problem was resolved in early October

16

Copyright 2015



Storage team promised they would monitor replication and recognize failures

Didn’t trust After all – Global Mirror was also brand new to them

SCI added replication monitoring On Prod, update timestamp in a file on each disk On DR, read the file; if time is not within 5 minutes of

system time – raise alert

And…. We caught replication failures that corporate missed!

17

Copyright 2015



Copyright 2015



Two AlphaServer systems must be migrated from physical hardware in one country to emulated systems in another country within weeks

SCI engaged to perform total system migration AlphaServer 1200 / OpenVMS V7.2 AlphaServer 4000 / Digital UNIX (aka Tru64) V4.0E All data transferred unchanged to emulation

Copyright 2015



Complete systems content successfully migrated from physical hardware to CHARON-AXP emulated environment

Hardware refitted Operating system, data and application unaffected Systems IP address modified for new location Virtual disk and tape replace physical drives

Copyright 2015



We felt a little bad being involved with sending tshis old hardware out to pasture

Copyright 2015



Need to transfer +/- 200GB business sensitive data securely over international borders Two 256GB memory sticks from different vendors Bitlocker AES-256 encryption with password known

only to customer

Limited downtime available No network Sites within 1 day drive of each other

Copyright 2015



Copyright 2015



1. Full physical system disk backups performed 2. Backups copied to encrypted devices 3. Transfer devices transported via automobile4. Emulator configured; matching disk drive layout5. Full system content restored from transfer devices

Copyright 2015



6. Operating systems reconfigured for network, disk and tape drive configurations

7. Data validated to compare old and new 8. Application tested for data and functionality9. Tested automated monthly / daily backup jobs 10. Data transfer devices securely erased

Copyright 2015



During initial data copy and validation phase, SCI laptop used as emulator platform

Network backup target

Validate systems functionality

Copyright 2015



HP ProLiant DL380p Gen8 2x E5-2680 v2 (10C, 25M Cache, 2.80/3.60T GHz) 32GB

RHEL 6 Update 5 (Santiago) Linux 2.6.32

CHARON-AXP V6.1

Copyright 2015



Prior AlphaServer 1200 1 CPU 2GB memory 1 tape 2 disks 10/100mb NIC

New Emulated AlphaServer 1200 1 CPU 4GB memory 1 tape 2 disks 1gb NIC

Copyright 2015



Prior AlphaServer 4000 1 CPU 512MB memory 3 tapes 4 disks 10/100mb NIC

New Emulated AlphaServer 4000 1 CPU 4GB memory 3 tapes 4 disks 1gb NIC

Interestingly enough, had been running at 10mb (presumably forever); set to 100 to make data copies tolerable

Copyright 2015



Container files represent emulated disk and tape drives When tape image is dismounted by guest, it is disconnected from

CHARON-AXP and can be manipulated Can be replaced with different tape image retaining same name Tape images ("*.vtape") created automatically if not existing

as1200_vms /charon/vms/DKA0.*disk /charon/vms/DKA100.*disk

as4000_tru64 /charon/tru64/DKA0.*disk /charon/tru64/DKA100.*disk /charon/tru64/DKA200.*disk /charon/tru64/DKA300.*disk

Copyright 2015



Once replacement system tested and validated

1. $ SET DEFAULT SYS$MANAGER:2. $ DISMOUNT DRA0:3. $ INIT/ERASE/SYSTEM DRA0: CLEANED4. $ SHOW DEV SYS$SYSDEVICE:5. $ COPY/ALLOC={freeblocks-1} NLA0: X.X6. $ SET FILE/END X.X7. $ DELETE/ERA/EXCL=*.DIR SYS$SYSDEVICE:[*…]*.*.*

! Ignore warnings and errors8. Power off system

Copyright 2015



Copyright 2015



Replace aging Alpha hardware

Provide Disaster Recovery capability 2 data centers, same corporate campus.

Utilize corporate NetApps SAN

33

Copyright 2015



Semiconductor manufacturer – Highly available

OpenVMS V7.3-2 , Rdb, Workstream for MES

3 Node Cluster ES45, 1 GHz, 2P 1 node very lightly used

1 additional ES45 – Dev/Test

34

Copyright 2015



CHARON-AXP on VMware hosts 2 * Fujitsu RX300 S8 E5-2643v2 CPUs (6C/12T, 3.50GHz). 5 virtual cores presented to each host

128 GB RAM (32GB per Windows host) 2 * 10Gb NICS, 2 * 1Gb NICs

NetApp Metrocluster – 2 FAS3270 filers 8 * 1G Ethernet ports

Reduced cluster from 3 nodes to 2 nodes

35

Copyright 2015



No FibreChannel on NetApps. Customer standard is NFS to VMWare hosts

Introduce split site cluster for DR (1700’)

Minimize downtime Customer prefers none

Data migration – how?

No legitimate test environment

36

Copyright 2015



Shared “vdisk” container files? Performance concerns with file shares Reliability & persistence of file shares

“vidisk” containers in each location with VMS HBVS between sites? Performance concerns using MSCP

Shared VMWare multi -writer .vmdk files? Not proven in production We chose this as first option

37

Copyright 2015


World Class Managed Services For OpenVMSGB network - NFS

VMWare leverages hosted .vmdk images as a Windows local diskESX allows multi-writer sharing of .vmdk images (VMWare KB 1034165)

\\.\PhysicalDriveN

(.vmdk)

ESX

NFS

NetApps Disk VolumesDisk Volumes

Building 1/ESX Host 1 Building 2/ESX Host 2

\\.\PhysicalDrive0

(.vmdk)… Disk controller

emulation: SCSI

Charon-AXP/DS20

VMWare Guest 2Windows

OpenVMS

DKA0

DKAn

Disk controlleremulation: SCSI

Charon-AXP/DS20

VMWare Guest 1Windows

OpenVMS

DKA0

DKAn

Copyright 2015



\\.\PhysicalDriveN

(.vmdk)

ESX

NFS



\\.\PhysicalDrive0

(.vmdk)…


Charon-AXP/DS20VMWare Guest 1

OpenVMS

Production “A” DKA0

DKAn



OpenVMS

Test/Dev DKA0

DKAn



OpenVMS

Production “B” DKA0

DKAn

Normal Operations

Copyright 2015



ESX

NFS



\\.\PhysicalDriveN

(.vmdk)

\\.\PhysicalDrive0

(.vmdk)

…



OpenVMS


DKAn



OpenVMS

Development DKA0

DKAn



OpenVMS


DKAn

Disaster Strikes

Copyright 2015



ESX

NFS



\\.\PhysicalDriveN

(.vmdk)

\\.\PhysicalDrive0

(.vmdk)

…



OpenVMS


DKAn



OpenVMS

Development DKA0

DKAn



OpenVMS


DKAn



OpenVMS


DKAn

VMware Recover Guest 1 on Host 2

Copyright 2015



1. 2 CHARON emulators on 2 VMWare instances. Use multi-write .vmdk containers

2. Build “test” OpenVMS cluster Test shared .vmdk works (system and data disks) Test V -Motion between sites (for planned

maintenance) Test VMWare recovery between sites (for DR)

3. Present all LUNs to emulator4. Test environment is ready

42

Copyright 2015



Remove 1 of 3 nodes (Remove NODE_A) Some processing had to be moved Some SCSNODE dependencies

Lots of node specific queue cleanup

Plan to boot an emulator into NODE_A’s root

43

Copyright 2015



Boot 1 emulator into NODE_A’s root

Use HBVS to synch data between Alpha/EVA cluster and CHARON/NetApps

FAIL!!!

44

Despite documentation indicating Satellite boot supported by CHARON – it isn’t

Copyright 2015



Create Heterogeneous cluster: 2 Alphas booting from EVA 1 Emulator booting from NetApps

Add network infrastructure for SCS traffic HBVS transfers data to NETApps (including Alpha

system disk) Shutdown CHARON node when copies complete Boot both CHARON nodes using replicated Alpha

cluster disk

45

Copyright 2015



Add an emulator to the cluster, in place of a prod node Couldn’t boot as node B because of references to node

B’s local disk on node C Had to reboot C

46

Copyright 2015



Repeat data transfer process Shutdown Alphas Shutdown CHARON nodes Boot CHARON node off replicated Alpha cluster

system disk

47

Copyright 2015



CHARON uses physical license token

What happens in a disaster?

Network USB hub? Rejected – single point of failure

48

Copyright 2015



CHARON Support Customers receive:“An additional CHARON license key, enabling 360 hours of execution time per supported product license, valid during the duration of the support contract. “

1 permanent license connected to each ESX host 1 “spare” license connected to each ESX host Specify license preference for each config

49

set session license_key_id[0]=nnnnnnn

set session license_key_id[1]=nnnnnnn

Copyright 2015



Alphas used SLS backing up to tape

Emulators: Database backups to VMS disk Non-database disks (.vmdk files) all reside on

dedicated NetApp datastore Daily NetApp snapshot of datastore used for backups

50

Copyright 2015



Copyright 2015



Brad

Brad McCusker

[email protected]

www.sciinc.com

Copyright 2015



Located in Nashua, NH (USA) 30 years in business supporting OpenVMS!

International reputation as leading provider of Managed services for OpenVMS & databases OpenVMS performance and consulting services

Proven global track record Actively managing 100s of systems and databases At many sites 24x365 since 1995

Copyright 2015



System and database performance consulting

VAX/Alpha emulation – CHARON reseller

Migration consulting Specializing in minimal downtime migrations

Oracle’s worldwide provider of CODASYL DBMS training

54

Documents

Phone: 603-879-9022 30 Temple St, Suite 303 e-mail ...sciinc.com/remotedba/techinfo/tech_presentations/Adventures of... · IBM DS8000, Global Mirror ... 6. Operating systems reconfigured