HA for OpenStack: Connecting the dots Raghavan “Rags” Srinivas Rackspace

OpenStack Meetup, Boston on Feb. 19th 2014

Rags

•  Solutions Architect at Rackspace for OpenStack-based Rackspace Private Cloud

•  Speaker at JavaOne, RSA conferences, Sun Tech Days, JUGs and other developer conferences

•  Trying to help make OpenStack more “App Developer” friendly

Agenda What is HA?

HA of OpenStack APIs

HA of RabbitMQ

MySQL HA

Resources and Summary

A Peek into HA Methods

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

OpenStack Design Tenets •  Scalability and elasticity are our main goals •  Any feature that limits our main goals must be optional •  Everything should be asynchronous

–  a) If you can't do something asynchronously, see #2

•  All required components must be horizontally scalable •  Always use shared nothing architecture (SN) or sharding

–  a) If you can't Share nothing/shard, see #2

•  Distribute everything –  a) Especially logic. Move logic to where state naturally exists.

•  Accept eventual consistency and use it where it is appropriate. •  Test everything

4

What is HA?

HA as Nines Downtime/Year

99% (two nines) 3.65 days

99.9% 8.76 hours

99.99% 52.56 minutes

99.999% 5.26 minutes

99.9999% (six nines) 31.5 seconds

•  Minimization of system downtime •  Minimization of data/transaction loss •  In case of multiple (or interrelated)

failures, minimization of data loss is preferred over minimization of system downtime

Implementing HA

•  Elimination of Single Point of Failure (SPOFs) •  Redundancy of network components such as switchers and routers

•  Redundancy of applications and automatic service migrations

•  Redundancy of storage components

•  Redundancy of facilities services such as power, AC, etc.

Client

NODE 1 Replication Services

Health Check

Cluster Communication

NODE 2 Replication Services

Health Check

Cluster Communication

VIP

Components (High Level)

Concepts State Description Example

Stateless

• There is no dependency between requests

• No need for data replication/synchronization. Failed request may need to be restarted on a different node.

Apache web server, Nova API, Nova Scheduler, etc.

Stateful

• An action typically comprises multiple requests

• Data needs to be replicated and synchronized between redundant services (to preserve state and consistency)

MySQL, RabbitMQ, etc.

More Concepts Terminology Description

Failover Migration of a service from the “primary” to the “secondary”

Failback Migration of service back to the “primary”

Switchover Migration is initiated manually

Much more concepts

Active/Passive o  There is a single master

o  Load balance stateless services using a VIP and a load balancer such as HAProxy

o  For Stateful services a replacement resource can be brought online. A separate application monitors these services, bringing the backup online as necessary

o  After a failover the system will encounter a “speed bump” since the passive node has to notice the fault in the active node and become active

Active/Active o  Multiple masters

o  Load balance stateless services using a VIP and a load balancer such as HAProxy

o  Stateful Services are managed in such a way that services are redundant, and that all instances have an identical state

o  Updates to one instance of database would propagate to all other instances

o  After a failover the system will function in a “degraded” state

HA for OpenStack

•  OpenStack APIs (nova, cinder, etc.)

•  RabbitMQ

•  MySQL

•  Cinder, Swift, and so on

•  Heat (still Work in Progress)

•  Application running on OpenStack (Application dependent)

Agenda What is HA?

HA of OpenStack APIs

HA of RabbitMQ

MySQL HA

Resources and Summary

A Peek into HA Methods

HA on OpenStack

•  Overall Philosophy (Don’t reinvent the wheel)

•  Leverage time-tested Linux utilities such as Keepalived, HAProxy and Virtual IP (using VRRP)

•  Leverage Hardware Load Balancers

•  Leverage replication services for RabbitMQ/MySQL such as RabbitMQ Clustering, MySQL master-master replication, Corosync, Pacemaker, DRBD, Galera and so on

vrrp_script rabbitmq {!

script “usr/sbin/service rabbitmq-server status" # Check the service status!

interval 5 # check every 5 seconds!

weight -2 # adjust priority by -2 if OK!

rise 2 # required number of failures for KO switch!

fall -2 # required number of successes for OK switch!

}!

Keepalived

•  Based on Linux Virtual Server (IPVS) kernel module providing layer 4 Load Balancing

•  Implements a set of checkers to maintain health and Load Balancing

•  HA is implemented using VRRP Protocol

1 2 3 4 5 6 7

HAProxy

• Load Balancing and Proxying for HTTP and TCP Applications

• Works over multiple connections

HA with Keepalived, VRRP & HAProxy

Host1 Host2

Backup

Realserver1 Realserver2

Keepalived HAProxy Network Layer

Application Layer

VRRP

Application

HA on Rackspace Private Cloud

HAProxy

INTERNET

Compute Node 1 Compute Node 2 Compute Node N

VIP(Keepalived, VRRP)

VMs Instantiated

Redundant Active-Passive Infrastructure services

Redundant Active-Active Infrastructure services

Controller 1 Controller 2 Active-Passive Infrastructure services

(MySQL, Rabbit) Active-Active Infrastructure services

(API services)

Heartbeat

HA on Rackspace Private Cloud (switchover)

INTERNET

Compute Node 1 Compute Node 2 Compute Node N

VIP(HAProxy)

VMs Instantiated

Infrastructure services

Controller 1 Controller 2 Active-Passive Infrastructure services

(MySQL, Rabbit) Heartbeat

Agenda What is HA?

HA of OpenStack APIs

HA of RabbitMQ

MySQL HA

Resources and Summary

A Peek into HA Methods

RabbitMQ HA Options

•  Health Check without Clustering

•  Clustering without Health Check

•  Health Check and Clustering

RabbitMQ HA

Ethernet VRID 13

192.168.236.199

Controller 1 VRID 13

IP address: 192.168.236.11

Backup (Passive) Master (Active)

Controller 2 VRID 13 IP address: 192.168.236.12

RabbitMQ RabbitMQ

RabbitMQ Clustering

Agenda What is HA?

HA of OpenStack APIs

HA of RabbitMQ

MySQL HA

Resources and Summary

A Peek into HA Methods

MYSQL HA: MASTER/MASTER REPLICATION

MySQL – Master/Master Replication

Ethernet VRID 12

192.168.236.198

Controller 1 VRID 12

IP address: 192.168.236.11

Backup (Passive) Master (Active)

Controller 2 VRID 12 IP address: 192.168.236.12

MySQL MySQL

Master/Master

MySQL – Master/Master Replication simplified

MYSQL HA: COROSYNC, PACEMAKER AND DRBD

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Pacemaker, Corosync and DRBD

27

Image from: http://dev.mysql.com/doc/refman/5.0/en/ha-drbd.html"

Pacemaker Corosync DRBD

High availability and load balancing stack for the Linux

platform

Totem single-ring ordering and membership protocol

Synchronizes data at the block device

Interacts with applications through Resource Agents (RA)

UDP and InfiniBand based messaging, quorum, and

cluster membership to Pacemaker

Uses a journaling system (such as ext3 or ext4)

Pacemaker, Corosync, DRBD

DRBD Service Service

FILE SYSTEM

BUFFER CACHE

DRBD

DISK SCHED

DISK DRIVER NIC DRIVER

TCP/IP

RAW DEVICE

DISK

NIC DRIVER

TCP/IP

RAW DEVICE

FILE SYSTEM

BUFFER CACHE

DRBD

DISK SCHED

DISK DRIVER

DISK NIC NIC

MYSQL HA: GALERA

Galera

DBMS

wsrep API

DBMS

wsrep API

DBMS

wsrep API

Galera Replication

Transparent Connections

CLIENTS •  Synchronous multi-master cluster technology for MySQL/InnoDB

•  MySQL patched for wsrep (Write Set REPlication)

•  Active/active multi-master topology

•  Read and write to any cluster node

•  True parallel replication, in row level

•  No slave lag or integrity issues

Multi-master replication

•  Based on Optimistic Concurrency Control

•  In case of two transactions modifying the same row on different nodes, one of the transactions will abort

•  Victim transaction will get Deadlock Error

•  Application needs to handle this error

Multi-master cluster looks like one big database with multiple entry points

read & write read & write read & write

MySQL

Multi-master Replication

MySQL MySQL MySQL

write write

GALERA REPLICATION

Multi-master conflicts

MySQL MySQL MySQL

GALERA REPLICATION

write write

Conflict detected

Multi-master conflicts

MySQL MySQL MySQL

GALERA REPLICATION

OK write Deadlock error

Multi-master conflicts

OpenStack and Galera

Image from http://www.severalnines.com/blog/clustering-mysql-backend-openstack"

Galera on Rackspace Private Cloud/OpenStack A How To: OFFICIALLY UNSUPPORTED

1.  Install Rackspace Private Cloud on 2 controllers with HA mode (Haproxy, Keepalived and VRRP is already installed)

2.  Install Galera (with ws-rep) on 3 separate nodes

3.  Mysqldump from controller nodes to Galera node

4.  Grant privileges to OpenStack (nova, glance, etc.) and haproxy users

5.  Update keepalived and haproxy and OpenStack configuration files on controller/compute

6.  Stop/Uninstall MySQL services on controller nodes and restart controller nodes

Agenda What is HA?

HA of OpenStack APIs

HA of RabbitMQ

MySQL HA

Resources and Summary

A Peek into HA Methods

A PEEK INTO HA METHODS

Vendor Clustering/Replication Technique Characteristics

Rackspace Keepalived, HAProxy, VRRP, native clustering

•  Automatic install on 2 controller nodes via Chef recipes

Red Hat Pacemaker, Corosync, DRBD

•  Manual installation. Fewer components to install

Cisco Keepalived, HAProxy, Galera for MySQL

• Manual install, at least 3 controller nodes

HP Microsoft Windows based installation with Hyper-V

• MS SQL server and other Windows-based methods

HA methods

HA on the Public Cloud

Agenda What is HA?

HA of OpenStack APIs

HA of RabbitMQ

MySQL HA

Resources and Summary

A Peek into HA Methods

Infrastructure Clustering/Replication Technique Characteristics

OpenStack APIs None required (Stateless) •  HA also serves as scale out using HAProxy

RabbitMQ RabbitMQ Clustering •  RabbitMQ Clustering is setup for single/multiple nodes

Heat TBD • Application Dependent (No standard methods yet).

MySQL Many • Discussed later slide

HA methods

Clustering Method Replication Technique Characteristics

Pacemaker/Corosync/DRBD Mirroring on Block Devices •  Well tested, more complex to setup. •  Split brain possibility

Keepalived/HAProxy/VRRP Works on MySQL master-master replication

•  Simple to implement and understand. •  Works for any storage system. •  Master-master replication does not work

beyond 2 nodes.

Galera Based on write-set Replication (wsrep)

• No Slave lag • Needs at least 3 nodes • “Deadlock erros” on hotspot rows. • Relatively new

Others MySQL Cluster, RHCS with DAS/SAN Storage

• Some relatively new (GTID) • Some well test • More complex setup

HA methods for MySQL

Resources •  OpenStack HA guide

•  http://docs.openstack.org/high-availability-guide/content/ch-intro.html

•  https://wiki.ubuntu.com/ServerTeam/OpenStackHA

•  Other Resources

•  http://www.rackspace.com/blog/implementing-high-availability-ha-for-rackspace-private-cloud/

•  http://www.rackspace.com/blog/high-availability-ha-with-galera-for-rackspace-private-cloud/https://www.hastexo.com/

•  http://www.mysql.com/why-mysql/white-papers/mysql-high-availability-drbd-configuration-deployment-guide/

•  http://docwiki.cisco.com/wiki/OpenStack_Havana_Release:_High-Availability_Manual_Deployment_Guide

•  http://www.drbd.org/

•  http://www.codership.com/

•  http://www.severalnines.com/blog/clustering-mysql-backend-openstack

•  https://wiki.openstack.org/wiki/BasicDesignTenets

•  http://db.cs.berkeley.edu/papers/hpts85-nothing.pdf

Book

Summary

•  In general leverage existing methods of HA

•  There are several time-tested and more recent methods for implementing MySQL HA.

•  Rackspace Private Cloud provides Chef cookbooks and recipes for implementing HA via Keepalived,

HAProxy and VRRP.

•  Galera is gaining more popularity. Since it’s Active/Active it does scale out and is HA.

•  Few steps to get from Rackspace Private Cloud to MySQL with Galera (officially unsupported).

•  Corosync/Pacemaker/DRBD is recommended by Oracle/MySQL.

•  OpenStack HA guide goes through all these options in more detail.

Thank you! Raghavan “Rags” Srinivas Solutions Architect Rackspace