(ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014

November 13, 2014 | Las Vegas, NV

“Everything fails, all the time.”

500

Black Friday balance: Submarino down, Americanas down, Walmart

down, Magazine Luiza with problems, FNAC down. #BlackFridayFail

Magazine Luiza is doing a clearance sale for New Year. It’s a nice sale,

but so good that their website is down

–120 orders / minute

XA

Going beyond the traditional high availability concept

eXtreme availability

XA

global infrastructure

beginning

processes

developers

architects

any failure

Graceful

Micro services m-SOA

availability needs

AWS

Amazon EC2 Amazon EC2

Elastic Load Balancing

Availability

Zone A

Availability

Zone B

Amazon S3

Static

Website

Amazon

CloudFront

User

Amazon

Route 53

Secondary

Sta

tic

Assets

Primary + Health Checks

Auto

Scaling

Group

us-east

Amazon S3

Static

Website

Amazon

CloudFront

User

Amazon

Route 53

SecondaryS

tatic

Assets

Active/Active with Latency Based Routing + Health Checks

EC2 EC2

Elastic Load Balancing

Availability

Zone A

Availability

Zone B

us-east-1

EC2 EC2

Elastic Load Balancing

Availability

Zone A

Availability

Zone B

us-west-1

Amazon S3

Static

Website

Amazon

CloudFront

User

Amazon

Route 53

SecondaryS

tatic

Assets

Active/Active with Latency Based Routing + Health Checks

EC2 EC2

Elastic Load Balancing

Availability

Zone A

Availability

Zone B

us-east-1

EC2 EC2

Elastic Load Balancing

Availability

Zone A

Availability

Zone B

us-west-1

EC2

us-east-2

EC2

us-west-2

Amazon

CloudFront

User

Amazon

Route 53

Sta

tic

Assets

Active/Active with Latency Based Routing + Health Checks

EC2 EC2

Elastic Load Balancing

Amazon

S3

Static

Content

EC2 EC2

Elastic Load Balancing

CA

CH

EW

EB

SE

RV

ER

Availability

Zone A

Availability

Zone B

Availability

Zone A

Availability

Zone B

sa-east-1 us-east-1

CA

CH

EW

EB

SE

RV

ER

CA

CH

EW

EB

SE

RV

ER

CA

CH

EW

EB

SE

RV

ER

Amazon RDS Multi-AZ

(with read replicas)

Amazon DynamoDB and

Amazon S3 have built-in

HA

Availability Zone A

Availability Zone B

Amazon RDS read replicas rock!

Availability Zone A

Availability Zone B

Amazon RDS

Read Replica

Amazon RDS

Read Replica

Availability Zone A

Availability Zone B

Amazon RDS Master

Amazon RDS

Read Replica Amazon RDS

Read Replica

Amazon RDS Standby

Availability Zone C

Amazon RDS

Read Replica

us-east-1 us-west-1

Amazon RDS read replicas rock!

Availability Zone A

Availability Zone B

Amazon RDS

Read Replica

Amazon RDS

Read Replica

Availability Zone A

Availability Zone B

Amazon RDS Master

Amazon RDS

Read Replica Amazon RDS

Read Replica

Amazon RDS Standby

Availability Zone C

Amazon RDS

Read Replica

us-east-1 us-west-1

Amazon RDS read replicas rock!

Availability Zone A

Availability Zone B

Amazon RDS

Read Replica

Amazon RDS

Read Replica

Availability Zone A

Availability Zone B

Amazon RDS Master

Amazon RDS

Read Replica Amazon RDS

Read Replica

Amazon RDS Standby

Availability Zone C

Amazon RDS

Read Replica

us-east-1 us-west-1

Amazon RDS Master

(promoted)

Availability Zone

A

Amazon RDS

us-east-1

Amazon

EC2 as

GoldenGate

Hub for

Source DB

Availability Zone

A

Amazon RDS

Amazon

EC2 as

GoldenGate

Hub for

Source DB

us-west-1

GET

us-east-1 us-west-1

GET

us-east-1 us-west-1

Let’s look

inside here!

Consuming AWSDynamoDBstreamsAdapterClient adapterClient =new AWSDynamoDBstreamsAdapterClient(updateStreamsCredentials, .. );

..

AmazonDynamoDBClient dynamoDBClient =new AmazonDynamoDBClient(dynamoDBCredentials, ..);

..

KinesisClientLibConfiguration workerConfig =new KinesisClientLibConfiguration (.., streamId,

updateStreamsCredentials, ..)

.withMaxRecords(100)

.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON);

Worker worker =new Worker(recordProcessorFactory, workerConfig, adapterClient,

dynamoDBClient, ..);

Thread t = new Thread(worker);t.start();

Consuming AWSDynamoDBstreamsAdapterClient adapterClient =new AWSDynamoDBstreamsAdapterClient(updateStreamsCredentials, .. );

..

AmazonDynamoDBClient dynamoDBClient =new AmazonDynamoDBClient(dynamoDBCredentials, ..);

..

KinesisClientLibConfiguration workerConfig =new KinesisClientLibConfiguration (.., streamId,

updateStreamsCredentials, ..)

.withMaxRecords(100)

.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON);

Worker worker =new Worker(recordProcessorFactory, workerConfig, adapterClient,

dynamoDBClient, ..);

Thread t = new Thread(worker);t.start();

Consuming AWSDynamoDBstreamsAdapterClient adapterClient =new AWSDynamoDBstreamsAdapterClient(updateStreamsCredentials, .. );

..

AmazonDynamoDBClient dynamoDBClient =new AmazonDynamoDBClient(dynamoDBCredentials, ..);

..

KinesisClientLibConfiguration workerConfig =new KinesisClientLibConfiguration (.., streamId,

updateStreamsCredentials, ..)

.withMaxRecords(100)

.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON);

Worker worker =new Worker(recordProcessorFactory, workerConfig, adapterClient,

dynamoDBClient, ..);

Thread t = new Thread(worker);t.start();

Consuming AWSDynamoDBstreamsAdapterClient adapterClient =new AWSDynamoDBstreamsAdapterClient(updateStreamsCredentials, .. );

..

AmazonDynamoDBClient dynamoDBClient =new AmazonDynamoDBClient(dynamoDBCredentials, ..);

..

KinesisClientLibConfiguration workerConfig =new KinesisClientLibConfiguration (.., streamId,

updateStreamsCredentials, ..)

.withMaxRecords(100)

.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON);

Worker worker =new Worker(recordProcessorFactory, workerConfig, adapterClient,

dynamoDBClient, ..);

Thread t = new Thread(worker);t.start();

Processingpublic class StreamsRecordProcessor implements IRecordProcessor {

..

@Override

public void processRecords(List<Record> records,.. ) {

for(Record record : records) {

if (record instanceof RecordAdapter) {

Record ddbStreamRecord = ((RecordAdapter)

record).getInternalObject();

switch(ddbStreamRecord.getEventName()) {

case "INSERT" : case "MODIFY" :

DemoHelper.putItem(dynamoDBClient, tableName,

ddbStreamRecord.getDynamodb().getNewImage());

break;

...

Processingpublic class StreamsRecordProcessor implements IRecordProcessor {

..

@Override

public void processRecords(List<Record> records,.. ) {

for(Record record : records) {

if (record instanceof RecordAdapter) {

Record ddbStreamRecord = ((RecordAdapter)

record).getInternalObject();

switch(ddbStreamRecord.getEventName()) {

case "INSERT" : case "MODIFY" :

DemoHelper.putItem(dynamoDBClient, tableName,

ddbStreamRecord.getDynamodb().getNewImage());

break;

...

Processingpublic class StreamsRecordProcessor implements IRecordProcessor {

..

@Override

public void processRecords(List<Record> records,.. ) {

for(Record record : records) {

if (record instanceof RecordAdapter) {

Record ddbStreamRecord = ((RecordAdapter)

record).getInternalObject();

switch(ddbStreamRecord.getEventName()) {

case "INSERT" : case "MODIFY" :

DemoHelper.putItem(dynamoDBClient, tableName,

ddbStreamRecord.getDynamodb().getNewImage());

break;

...

Processingpublic class StreamsRecordProcessor implements IRecordProcessor {

..

@Override

public void processRecords(List<Record> records,.. ) {

for(Record record : records) {

if (record instanceof RecordAdapter) {

Record ddbStreamRecord = ((RecordAdapter)

record).getInternalObject();

switch(ddbStreamRecord.getEventName()) {

case "INSERT" : case "MODIFY" :

DemoHelper.putItem(dynamoDBClient, tableName,

ddbStreamRecord.getDynamodb().getNewImage());

break;

...

ERP

Database

Replica

ERP

Database

Replica

Availability

Zone AAvailability

Zone B

sa-east-1

EC2 EC2

WORKERS

COLO

ERP

Database

AWS Direct

Connect

NoSQL NoSQL

WORKERSAmazon Simple

Queue Service

us-east-1 Internet

Same infrastructure with constant

data replication

Please give us your feedback on this session.

Complete session evaluations and earn re:Invent swag.

http://bit.ly/awsevals