Disaster Recovery on Demand on the Cloud

Protect your app from OutagesNati Shalom CTO GigaSpaces@natishalom

May 2013

2

AWS and outages Outage impact Disaster Recovery – it’s all about redundancy! Cloudify as a solution for redundancy Demo with Cloudify on EC2

® Copyright 2013 GigaSpaces Ltd. All Rights Reserved

AGENDA

3

AWS USAGE

Managing Big Data on the Cloud

• AWS – around 0.5M servers• Facebook – less than 0.1M servers• Google – around 1M servers

4

THE OUTAGE PROBLEM

® Copyright 2012 GigaSpaces Ltd. All Rights Reserved5

OUTAGE – APRIL 21, 2011

® Copyright 2012 GigaSpaces Ltd. All Rights Reserved6

OUTAGE - JUNE 29, 2012

® Copyright 2012 GigaSpaces Ltd. All Rights Reserved7

OUTAGE - OCTOBER 22, 2012

® Copyright 2012 GigaSpaces Ltd. All Rights Reserved8

OUTAGE - CHRISTMAS EVE 2012

® Copyright 2012 GigaSpaces Ltd. All Rights Reserved9

NOT ONLY AMAZON

28 December 2012 - some owners of Microsoft's XBox 360 gaming console were unable to access some of their cloud-based storage files.

26 July 2012 - Service for Microsoft’s Windows Azure Europe region went down for more than two hours

29 February 2012 - The ultimate result was service impacts of 8-10 hours for users of Azure data centers in Dublin, Ireland, Chicago, and San Antonio.

10

THAT’S WHAT YOU EXPECT?

Managing Big Data on the Cloud

99% - 3.65 days downtime99.9% - 8.76 hours downtime99.99% - 53 minutes downtime99.999% - 5.26 minutes downtime

® Copyright 2012 GigaSpaces Ltd. All Rights Reserved11

OUTAGE IMPACT – DESIGN FOR FAILURES

Outage could cost…$89K per hour for Amadeus$225K per hour for PayPal!

12

DISASTER RECOVERY

13

MULTI CLOUD

Managing Big Data on the Cloud

14

PREPARE FOR DISASTER RECOVERY

Managing Big Data on the Cloud

•Dedicated expert for DR architecture•Define target recovery time & point•Assume every tier can fail•Use monitoring and alerts•Document your operational processes

15

CHAOS MONKEY

Managing Big Data on the Cloud

16

It’s all about REDUNDANCY!

17

CLONE YOUR ENVIORMENT

Managing Big Data on the Cloud

18

CLONE YOUR DATA

•RDS Read Replica•More to come…

19

Automating your DR

Processes

Leverage Existing Automation Frameworks

Configuration Centric APP Centric (PaaS)

CLONE YOUR ENV - HOW DOES IT WORK?

BUILT IN SUPPORT FOR MANAGING DATA IN THE CLOUD

Real Time Relational DB Clusters

NoSQL Clusters Hadoop

Storm MySQL MongoDB Hadoop (Hive, Pig,..)

Elastic Caching XAP Postgress Cassandra ZooKeeper

Couchbase

ElasticSearch

23

Real Life Scenario

24

Technology-based concrete process control and information service

Deployments across North America, Latin America, Asia, and Europe for nearly a decade

Part of W.R. Grace & Co , $6.3 B Company.

The problem: On-Demand HA/DR over multiple Cloud regions.

CASE STUDY: VERIFI

High Availability

Data Replication

Disaster Recovery

ELASTIC ON-DEMAND DISASTER RECOVERY

25

Problem Can we eliminate the

RTO vs. Cost trade-off in the cloud?

Solution (Elastic DR) A hybrid between Hot

and Warm DR Switch to Active site

in matter of seconds through cloud-agnostic lifecycle automation recipes

VERIFI (INITIAL) ARCHITECTURE

26

Availability region (US-West: Oregon)

Data VolumeInternet EC2 Instance

mod_cluster

EC2 Instance

JBoss

Data Volume

EC2 Instance

EC2 Instance

PostgresSQL

Cassandra

4 recipes

ELASTIC DR ON-DEMAND: FAILOVER SCENARIO

27

Region (US-West Oregon)

App ServersPostgresSQL

Region (US-East Virginia)

PostgresSQL

Cloud #1 Cloud #2

Region (US-East Virginia )

PostgresSQL

Cloud #1 Cloud #2

XApp Servers

Region (US-West California)

PostgresSQL

Cloud #3

Region failure occurs

* Initially, all those actions may be done manually by Verifi’s Ops team (e.g.: via recipe commands in CLI)

Bootstrap another cloud in a different region using the same application recipe used to bootstrap cloud #2 above*

1 2 3

Liveness poll

Liveness poll

0 Upon initial deployment, the primary deplyoment of the application “verifi” will be bootstrapped onto cloud #1, another slightly modified application recipe “verifi_dr” will be bootstrapped as cloud #2, polling cloud #1 for failure, and acting as a PostgresSQL db slave.

Turn Postgres slave into master, Start app server instances*

FAILOVER SCENARIO

28

Region (US-West Oregon)

App ServersPostgresSQL

Region (US-East Virginia)

PostgresSQL

Cloud #1 Cloud #2

Region (US-East Virginia )

PostgresSQL

Cloud #1 Cloud #2

XApp Servers

Region (US-West California)

PostgresSQL

Cloud #3

Region failure occurs

Bootstrap another cloud in a different region using the same application recipe used to bootstrap cloud #2 above*

1 2 3

Liveness poll

Liveness poll

0 Upon initial deployment, the primary deployment of the application will be bootstrapped onto cloud #1, another slightly modified application recipe will be bootstrapped as cloud #2, polling cloud #1 for failure, and acting as a PostgresSQL db slave.

Turn Postgres slave into master, Start app server instances*

29 Copyright 2012 Gigaspaces. All Rights Reserved

NEXT STEPS

Across clouds(AWS, Rackspace, Azure…etc)

Across AWS regions

Across AWS zones

1 application + overrides

Several cloud drivers

1 application + overrides1 cloud driver

1 application + overrides 1 cloud driver

Avai

labi

lity

Supported byVerifi phase #1

30 Copyright 2013 Gigaspaces. All Rights Reserved

ELASTIC ON-DEMAND DR: COSTS

Main Site (US-West) Warm DR Site (US-East) Hot DR Site

Cost $82,068 $12,625 $82,068

Main Site 1 Load balancer, 2 JBoss instances, 1 PostgreSQL master, 3 Cassandra

DR Site 1 PostgreSQL slave – All other instance start on demand upon failover

31 Copyright 2013 Gigaspaces. All Rights Reserved

ELASTIC DR: WARM DR COST, CLOUD PORTABILITY

4 recipes

DR Site$12k

Sam

e Re

cipe

$14k

$6k

$5k

$9k

32 Copyright 2013 Gigaspaces. All Rights Reserved

ELASTIC DR: HOT DR COST

4 recipes

DR Site$82k

Sam

e Re

cipe

$79k

$115k

$68k

$91k

33

Disaster Recovery – it’s all about redundancy! Cloning your environment – app stack Cloning your Data – DB Replication

Automation makes DR processes simple Use recipes to clone your app stack consistently Use replication to clone your data

Leverage cloud economics to reduce the cost DR on Demand Multi Cloud

® Copyright 2013 GigaSpaces Ltd. All Rights Reserved

SUMMARY

34

Thank You!@natishalom

® Copyright 2013 GigaSpaces Ltd. All Rights Reserved

QUESTIONS & ANSWERS