1

Disaster Recovery “Cookbook” Guide

Using VMWARE VI3, StoreVault and Sun.

(Or how to do Disaster Recovery / Site Replication for under $50,000)

By Scott Sherman, VCP, NACE, RHCT

Systems Engineer Integrated Storage Consulting Services Inc

scott@eiscsi.com http://www.eiscsi.com

2

Table of Contents Is this you? .............................................................................................................................................................. 3 About this “Cookbook”........................................................................................................................................... 4 Primary Site Setup .................................................................................................................................................. 6 Moving your Virtual Machines from Local Storage to the iSCSI LUN using VirtualCenter .............................. 16 How to move the virtual machines if you are not using VirtualCenter. ............................................................... 19 VI3 setup for Disaster Recovery Site.................................................................................................................... 30 StoreVault setup for your Disaster Recovery / Failover site ................................................................................ 36 StoreVault Replication Setup................................................................................................................................ 44 Testing the replicated LUN................................................................................................................................... 47 Solution Pricing and Summary ............................................................................................................................. 59 Revision Notes ...................................................................................................................................................... 61

3

Is this you?

A. Your boss has read an article on his last business trip about Disaster Recovery and tasks you with the job. In the past this would be a monstrous expensive task involving redundant hardware, clustering software and a hope and a prayer that it works.

B. You have a few mission critical servers that you want to keep online without breaking the bank. C. Maybe your worried that if you lose connectivity to your primary site for a period of several days

that your business would suffer D. You have gone through the personal hell of trying to restore a mission critical server from tape, only

to find the last tape is lost, damaged, or eaten by your tape drive. Now you want it replicated. This cook book is for you. This is a step by step guide on how to setup an easy to manage and cost effective Disaster recovery solution for your environment.

4

About this “Cookbook” For the past year or so I have seen many “whitepapers” on how to setup Disaster Recovery solutions using various VMWARE and disk solutions. Most of these papers have been based on theory with little showing the how to, or they pitch expensive Professional Services offerings to set it up for you. After some thought and discussions with my peers I realized that this could be done using current offerings from vendors, with a straight forward implementation. So here it is. This is a guide designed to show you how do setup replication to a DR site in a step by step fashion. This “cookbook” may seem large but it is documented with screenshots and dialogs to get setup going very quickly with readily available technology. Introduction . This paper is broken out in sections which allow the setup of the primary site on SUN Hardware, VMWARE VI3 and StoreVault storage. Then we will take these virtual machines and replicate them via the StoreVault Replication Service to have them run in another site. By using this combination of hardware and software it gives the ability to have a Highly Available server farm without the expense of clustering software. Please remember that these configurations below are baseline configurations and that you may want to change, add, or modify the hardware depending on your Operating System, Applications, Users and Network Speed. High Level Setup Attached is an overview of the basic networking using my primary site behind router #1 with two VMware servers. I have only one VMWARE server in my DR site to reduce costs. Assuming that you are not Depending on your setup you might be able to get by with a single server at your DR site. (Depends on how many virtual machines are mission critical and the performance characteristics of your applications. More servers can be added for your own needs. Also the link between the sites is on business class internet service (8mb down / 1mb up) and not on a dedicated circuit. The sites are “tunneled” via the routers for security.

5

Assumptions

1. This document assumes that you have basic familiarity with VMWARE and STOREVAULT products

2. Those machines have already been setup on the primary and DR site using VI3. 3. That my mission critical servers are currently in VMWARE and hosted on local disk (They could be

on SAN/NAS, we are going to show how to move them to an iSCSI LUN which will be replicated) Disclaimer This paper is presented as a “cookbook” guide on how to setup a DR solution using STOREVAULT, SUN, and VMWARE products. Your environment may require specific changes that are not covered in this paper. These companies did not sponsor nor endorse this paper and are trademarks of their respective companies. Scott Sherman, the companies mentioned nor Integrated Storage assume no liability on the use or implementation of these concepts. For further information please contact us.

6

Primary Site Setup First we need to setup your primary server for access to an iSCSI LUN on your primary StoreVault. This is show on in the following section by moving your virtual machines from local disk (residing on the local server) to an iSCSI LUN on the StoreVault.

Log into the primary site (I am doing this via the VIC server) where we have 2 VMWARE servers (IP 192.168.1.150 and 192.168.1.160). Currently my virtual machines are stored on local disk (Storage1(1)) and NFS mounts. What we need to do is enable iSCSI on the systems. Starting with 192.168.1.150 goto the configuration tab, then click on networking

7

After verifying that the VMKernel is configured in the networking section of your server. (for more info about this please refer to the VI3 documentation). This is what allows VI3 to communicate via the iSCSI protocol to the StoreVault. A separate IP address is needed for the VMKernel. In our example it has an IP of 192.168.1.151 Then we have to allow iSCSI to be passed through the VI3 firewall. From the main screen click on Security Profile

Click properties

Click on Software iSCSI Client Click ok. At this point I reboot the system to ensure all changes take effect.

8

After the reboot log back in to the VI3 client, click on Storage adapters then the iSCSI Software adapter

Click on properties and the iSCSI Initiator properties will come up. Click configure then enable and ok.

Click over the Dynamic Discovery tab Click Add and enter the IP of your primary site StoreVault Click Ok. This will tell your VI3 server to send it’s initiator to the StoreVault

9

At this point log into your primary site StoreVault and create a LUN big enough for your mission critical Virtual Machines. My LUN is 200gb. (You can check the size of each machine by logging into the VI3 client, and look at the size of each of your virtual machine disks)

Click Create LUN(SAN)

Click Next

10

Add the IQNS that were in VMWARE to the LUN

11

If your IQN didn’t show up you can add it manually as follows

Right click on the LUN and choose manage LUN

Click on initiators and verify that you can see 2 entries for each machine.

12

Now that your LUN is created via the StoreVault GUI we need to go back to the VI3 client. Click on Initiator group and add the missing system into the group (My missing machine is the x345-IP 192.168.1.150). Click save, then ok at the successful prompt. Go back to the VI3 client

Choose rescan under the configuration / storage tab

13

Click on storage (SCSI, SAN and NFS)

Choose Add Storage Disk / LUN

Choose the LUN we created in the StoreVault GUI Click Next

14

Click Next

Name the datastore (Note- on the primary site the Datastore name MUST be the same to allow VMOTION if applicable.)

15

Click Next

Click Finish REPEAT THE STEPS TO ADD THE VOLUME TO OTHER MACHINES AT YOUR PRIMARY SITE

16

Moving your Virtual Machines from Local Storage to the iSCSI LUN using VirtualCenter Log onto the VMWARE VirtualCenter.

Choose your virtual machine right click and choose Migrate

Choose the same host

17

Click Next

This is the key option. Choose Move virtual machine configuration files and virtual disks. Then the datastore choices will appear. Choose the datastore we created on the StoreVault. Click Next

18

Click Finish Repeat these steps for all virtual machines you want to replicate to your DR site.

19

How to move the virtual machines if you are not using VirtualCenter. Moving the virtual machines is still fairly easy if you don’t have the VIC server. It just makes use of the command line.

1. Goto your VI3 console, log in at root and create another user with the useradd command. This is necessary as the default security profile in VMWARE does not allow users to log in remotely as root. You must login as another user then switch user to root. This option can be changed, please refer to www.vmware.com for info about this.

Then issue the passwd command with the username right after it. My example has the username of druser. You will then type the password twice then when back at the # prompt. Type exit Download a ssh client from the web. For this paper I’m using PUTTY which can be found by googleing putty.exe. When you execute putty you should see this screen

20

Enter the IP of the server we want to migrate the virtual machines from (Mine is 192.168.1.160). Click open. If this is the first time you login to this machine via putty, you will get a message about accepting the key. Choose Yes

Login as the user we created Type in su – root then the root password Directory structure of VMware / = root /vmfs = where virtual machines are located /vmfs/volumes/storage1/ = is where virtual machines are stored on a local system /vmfs/volumes/XXXXX = XXXXX is another datastore location, it can be NFS, iSCSI or Fiber Channel.

21

We need to move the files in the /vmfs/volumes/storage1 directory to the iSCSI LUN located at

This command copies the directory /vmfs/volumes/storage1/root to the iSCSI lun which is located at /vmfs/volumes/missioncritialscsi This WILL TAKE about 3 minutes per gig of data (may vary depending on your network, please be patient). Also this leaves the original files on the local machine.

22

After the copy we verify the files are in the correct directory. We now need to point the VI3 server to the virtual machines which we moved tot the iSCSI LUN. Log onto the VI3 client to the machine where the files are

Click on the machine, click on New Virtual Machine

23

Choose Custom, click next

Name the machine

24

Choose the iSCSI LUN

Choose the Operating System

25

Processors

Memory

26

Network Adapters

I/O Adapters

Choose Use existing virtual disk

27

Browse to the iSCSI LUN and the correct directory (Mine is root). Choose the filename.vmdk

Choose Next

28

Click Next

Click Finish

29

The new machine is now available the VI3 client. You can delete the old system after you have verified it works correctly.

30

VI3 setup for Disaster Recovery Site Remember to make machine HOST BASED LICENSE, unless you have a license server at the DR site unless you have a license server at your DR site.

Click on machine name Configuration Tab Add Networking Choose VMKERNEL, click next

Note – You can choose to put your VMKERNEL on another virtual switch, for this case I’m leaving it on vSwitch0

31

Click Next Then click finish on next screen You may get this

Click Yes

32

Enter your Default Gateway Click ok From the main screen click on Security Profile

Click properties

33

Click on Software iSCSI Client Click ok. At this point I reboot the system to ensure all changes take effect. Log back in and choose the configuration tab Storage adapters then the iSCSI software adapter

Click properties

34

Choose Configure

Click Enable then ok

Now click on dynamic discovery Click Add

35

Enter the IP of your StoreVault Click ok ISCSI is setup on your VI3 DR BOX. Now we need to see the StoreVault GUI

36

StoreVault setup for your Disaster Recovery / Failover site Login to the DR StoreVault box

For testing we will create a LUN

Click Create LUN (SAN)

37

Click Next

38

Map over your iSCSI initiator by highlighting and clicking the right arrow.

You can verify which is the correct iSCSI IQN by verifying in the VI3 client under storage adapters. (For our example the x235-595ae094 is our target)

39

Click Commit

Success, click finish Go back to the VI3 client

40

Configuration Rescan Click ok

Your LUN should be created and appear in the details section

41

Click storage (SCSI, SAN and NFS) and choose Add Storage Disk/LUN

Your LUN should appear, click next

42

Click Next

Name your DATASTORE

43

Click Next

Click Finish

Your test lun should appear under storage

44

StoreVault Replication Setup At this point we have both sites setup with VI3 and talking to the StoreVault at each location. Now we will setup the LUN with the virtual machines to replicate from the primary site to the DR site every hour. Login in to the StoreVault Client on the primary site

Choose “Manage Replication” from the LUN we want to replicate

Click Next

45

Enter the information for the Destination StoreVault.

Choose how often you want the data to be replicated. One of the advantages of the StoreVault is that every hour it’s not replicating the whole volume. It’s only moving the blocks that changes (in 4k increments). This will reduce the replication time drastically after the initial sync.

46

Successful You can telnet into the source box (This is not supported by SV support) to check the status of your replication with a snapmirror status command

For my 200 gig LUN it took about 3 hours (It’s better to do the initial replication in the same center then ship the DR unit to the site)

47

Testing the replicated LUN. After the replication has completed, the DR site can now be tested to ensure that your applications and servers are all at the DR site and functioning. This involves disabling the replication of data from the primary site to the DR site as follows. Goto the SV GUI on the DR site box

Click on the LUN , right click and choose recover. Lun will move to the Primary Storage section of the GUI.

Note – This breaks the replication from the primary site to the DR site You then need to setup the DR site machines to access the LUN. You can follow the steps in the previous sections to enable iSCSI on the VI3 machine. Then add the IP of the DR StoreVault.

48

Goto the StoreVault GUI and right click on manage LUN

49

Goto the Initiator Group and add in the drsite initiator. Click Save

Under the inittators tab confirm that your initator is there. (This example shows the node name drsite). It can still be there if you didn’t do the previous step. But will not work if it’s not added into the Initator Group.

50

Goto Hardware / Storage Adapters, then click rescan.

This will populate the Details field of the GUI. We now have our datastore mapped and ready to add in the machines. The last step is adding the Virtual Machines into the DR sites VI3 software. As of now there is not an automated way to do this, but when Site Recovery Manager comes out from VMWARE sometime in Q1/2008 this may be an automated task.

51

Goto the storage section of hardware, then click add storage, choose Disk / LUN. If should then find your storage on the DR site StoreVault and access the LUN.

Now to add the machines choose New Virtual Machine

52

Choose Custom, then next.

Enter what you have named the machine at the primary site.

53

Select the LUN from your DR site StoreVault, click next

Choose your operating system, click next.

54

Choose the number of processors you want, click next

Choose the amount of memory, then click next.

55

Choose the Network and quantity of NIC’s you want, click next.

Choose the SCSI Adapter you want, click next.

56

Choose Use an existing disk, click next

Select your virtual machines disk, click next.

57

Click Next

Click Finish

58

Your Machine is now added to the DR site. You can now power up the VM and ensure that everything is functional. Repeat the above steps for all machines that are in the iSCSI LUN. Notes:

1. You may need to change the IP’s of your guest operating systems, or have DHCP setup at your DR site. 2. The original LUN and site may be up and running and you can test your DR site. 3. A current limitation of the StoreVault is that you cannot resync the LUN. If you have a failover

situation and do processing at your DR site, you must then replicate the DR LUN to your primary site and reattach them there.

59

Solution Pricing and Summary Disaster Recovery should be looked at my all company’s of all sizes to ensure that they can recover from any disaster (Flood, Weather, Etc) and it can be done without breaking the bank, or the need of massive professional service engagements. I’ve chosen the hardware below as I believe they are best of breed, truly mission critical and can scale as needed. I wanted to include these prices so the reader would know a baseline of what hardware and software is needed and approximate costs. This hardware selection may not have the horsepower to keep every server and application running in a disaster, but will provide sufficient processing power for about 10-20 mission critical applications and replicate them to a DR site. All prices are list / web as of 12/20/2007 qty Sun Server #1 $ 5,460.00 Sun Server #2 $ 5,460.00 Sun Server #3 $ 5,460.00 StoreVault 500 #1 $ 9,980.00 StoreVault 500 #2 $ 9,980.00 VMWARE VI3 Standard (3 copies) $ 8,985.00 Support $ 1,797.00 $47,122.00

60

StoreVault S500

SUN X4100 SERVERS

Item Description Qty Price (U.S.) Total

Sun Fire X4100 Server 1 A86-FJZ2BH4GCBA $ 4,760.00

Sun Fire X4100 M2 Server, 2 AMD Opteron Model 2216, 2.4 GHz, 1MB, Processor, 4 GB (4 x 1 GB DIMMs) DDR2-667 Memory, 4 Disk Chassis, 2 x 73 GB Disks, DVD-ROM/CD-RW, 2 High Efficiency PSU, Service Processor, 4 x 10/100/1000 Ethernet Ports, 3 USB 2.0 Ports, 2 x 8-lane PCIe Slots, RoHS-5 Compliant

2 X311L $ 0.00Power Cord Kit, North American/Asian, RoHS Compliant

2 X4225A-Z $ 350.002 GB (2 x 1 GB DIMMs) Registered ECC PC3200 DDR2-667 Memory Kit, RoHS-5 Compliant

1 $ 5,460.00 $ 5,460.00

. Subtotal $ 5,460.00

61

Revision Notes 12/26/2007 – V1.0 Released for initial review and comment.