Red Hat Enterprise Virtualization Disaster RecoveryJohn Herr, RHCASenior Software Engineer, Red Hat

Aaron WeitekampSenior Software Engineer, Red Hat

What is this session about?

This session is about Disaster Recovery

•An overview of the failover process.

•The methods available to create a failover.

•The Red Hat Enterprise Virtualization 3.3 REST API.

•Automating the failover process.

Presenters

John Herr (joherr@redhat.com)●Senior Software Engineer, Red Hat

Aaron Weitekamp (aweiteka@redhat.com)●Senior Software Engineer, Red Hat

Both work in the Systems Engineering Team at Red Hat. This team identifies high-value solution stacks based on input from Sales, Marketing, and Engineering teams and develops reference architectures for customers.

Failover EnvironmentNo two environments are the same

Environment Description

•Required Mirrored Storage

•Red Hat Enterprise Virtualization Manager

•Data Domain

•Optional Mirrored Storage

• ISO Domain

•Export Domain

•Non-Mirrored Storage

•DNS/DHCP server

Failing OverThe Process

Failover Steps

1.Ensure storage domain data is available

2.Ensure name resolution for the RHEV Manager host name

3.Bring up RHEV Manager at second site

4.Add second sites hypervisors to the datacenter

5.Modify the storage configuration

6.Bring up VMs

Preparing

•Move storage domain data

•Mirrored data storage (Ideal Solution)

•Name resolution

Interfaces to Control Failover

Failover Methods

•Web User Interface (partial)

•REST API

•rhevm-shell

•rhevm-sdk-python

•rhevm-sdk-java

•Third Party Applications

Web User Interface (portal)

RHEVM-Shell

•Wrapper around the REST API calls

•Data is passed and returned in a human readable format

•Uses either UUIDs or Common Names to access resources

•Scriptable

RHEVM-Shell Setup

$ cat /root/.rhevmshellrc [cli]autoconnect = Trueautopage = True[ovirt-shell]username = admin@internaltimeout = -1extended_prompt = Falseurl = https://rdr-rhevm.sitea.example.org/apiinsecure = Falsefilter = Falsesession_timeout = -1ca_file = /root/ca.crtdont_validate_cert_chain = Falsekey_file = Nonepassword = passwordcert_file = None

RHEVM-Shell Example

•Interactive use of shell$ rhevm-shell[RHEVM shell (connected)]# add datacenter --name DC-iSCSI --storage_type iscsi --version-major 3 --version-minor 3

•Non-Interactive use of shell$ rhevm-shell <<< "add datacenter --name DC-iSCSI --storage_type iscsi --version-major 3 --version-minor 3 --expect '201-created'"

RHEVM-Shell Example - Output

id : ecd6d665-9488-40eb-808d-66ad12126c5e

name : DC-iSCSI

status-state : uninitialized

storage_type : iscsi

supported_versions-version-major: 3

supported_versions-version-minor: 3

version-major : 3

version-minor : 3

REST API

•Calls are made through a web portal as GET, PUT, POST, etc requests.

•Data is passed and returned in XML format

•Uses UUIDs to access resources

•Curl can be used to submit API requests

•Scriptable

REST API Example$ wget --no-check-certificate https://rdr-rhevm.sitea.example.org:/ca.crt

$ curl --cacert ca.crt --user 'admin@internal:password' \ --header "Accept: application/xml" \ --header "Content-type: application/xml" \ --request "POST" \ --data '<data_center> <name>DC-iSCSI</name> <storage_type>iscsi</storage_type> <version minor="1" major="3"/> </data_center>' \ https://rdr-rhevm.sitea.example.org/api/datacenters

REST API Example - Output<?xml version="1.0" encoding="UTF-8" standalone="yes"?><data_center href="/api/datacenters/1727e963-bd5c-4e43-99c5-d79fd1635103" id="1727e963-bd5c-4e43-99c5-d79fd1635103"> <name>DC-iSCSI</name> <storage_type>iscsi</storage_type> <version major="3" minor="1"/> <supported_versions> <version major="3" minor="2"/> <version major="3" minor="1"/> <version major="3" minor="3"/> </supported_versions> <status> <state>uninitialized</state> </status></data_center>

RHEVM software development kits

•rhevm-sdk-python, rhevm-sdk-java

•Libraries for interacting with the REST API

API Methods

Disaster Recovery APIs

• /api/storageconnections/CONNECTION_UUID ☺

•path, iqn, address, port, user, password

• /api/networks/NETWORK_UUID

•name, description, ip, vlan, stp and display

• /api/hosts

• /api/hosts/HOST_UUID/nics/NIC_UUID• network, ip and boot_protocol

Update iSCSI Storage Connection – rhevm-shell

[RHEVM shell (connected)]# update storageconnection e545772d-0e34-4db6-8e6f-db076ef8a139 --target iqn.2001-05.com.equallogic:0-1cb196-7de60c201-2e19b5f809c5310b-rdr-iscsi-b

id : e545772d-0e34-4db6-8e6f-db076ef8a139

address: 172.31.143.200

port : 3260

target : iqn.2001-05.com.equallogic:0-1cb196-7de60c201-2e19b5f809c5310b-rdr-iscsi-b

type : iscsi•

Update iSCSI Storage Connection – REST API

curl --silent --cacert rhevm.cer \

--user "admin@internal:2bor!2b" \

--header "Accept: application/xml" \

--header "Content-type: application/xml" \

--request "PUT" \

--data "<storage_connection> <target>iqn.2001-05.com.equallogic:0-1cb196-7c160c201-08f9b5f80995310b-rdr-iscsi-a</target> </storage_connection>" \

https://rdr-rhevm.sitea.example.org/api/storageconnections/e545772d-0e34-4db6-8e6f-db076ef8a139

Disaster Recovery Process

Failover Steps

•Bring up RHEV Manager

•Fix networking

•Keep original hostname• Prevents need to recreate certificates

•Add hypervisors to RHEV Manager and approve them

•Modify storage connection using API

•Start the Vms

•Test for functionality

Mirror Data from Primary Site to Backup Site

•This is hopefully already done using mirroring

• If not, the data domain needs restored to the backup site

•Could take a while

Name Resolution

•Create a dns entry for the RHEV Manager

•This entry should resolve the original FQDN to an address on the backup site's network

•Will require zone for original site

●Bring up the RHEV Manager

•Fix networking

•New IP Address (can be automatically taken care of with DHCP)

•May need to edit /etc/udev/rules.d/70-persistent-net.rules

•Keep original hostname to prevent the need to re-create certificates

Hypervisors

•Add the new hypervisors to RHEV Manager

•Add configure hypervisor nics if needed

•Power on the Vms

•Test the Vms for functionality

New Backup API Methods

Backup APIs

• /api/vms/VM_UUID/snapshots

• /api/vms/VM_UUID/snapshots/SNAP_UUID/disks

• /api/vms/VM_UUID/disks

●Backup Process

•Create a snapshot of the VMs disk

•Attach the snapshot onto a backup appliance VM

•Backup the data on the disk

•Detach the snapshot from the appliance

•With the API, all this can be controlled via the appliance VM

Resources

Summit Sessions of Interest

•Configuring a disaster-resilient Red Hat Enterprise Virtualization environment

• Wednesday, 2:30 - 3:30

•Red Hat Enterprise Virtualization: Overview & roadmap• Wednesday, 3:40 – 4:40

•Red Hat Enterprise Virtualization Hypervisor roadmap• Thursday, 9:45 – 10:45

•Red Hat Enterprise Virtualization deep dive• Wednesday, 4:50 – 5:50

Summit Demos

•Symantec

•Tintri

Reference Architectures

•Red Hat Customer Portal

•Provides tar balls of scripts used in reference architecture if available

•https://access.redhat.com• Support -> Knowledgebase -> Reference Architectures

•Red Hat Resource Library

•No scripts available

•http://www.redhat.com/resourcelibrary/reference-architectures/

ThanksPlease complete the session survey