Masterclass Webinar: Application Services and Dynamic Dashboard

Masterclass

Application Services and Dynamic Dashboard

Ryan Shuttleworth – Technical Evangelist @ryanAWS

A technical deep dive beyond the basics Help educate you on how to get the best from AWS technologies

Show you how things work and how to get things done Broaden your knowledge in ~45 mins

Masterclass

A grand title for a demonstration system that ties together services Show you how to use services like SNS and SQS to carry AWS events

Use S3 as a web server to host dynamically generated content Why? Show you some tips and tricks you can use in your projects

Application Services & Dynamic Dashboard

So what are we going to run through?

Services & topics

EC2 Instances to run our application code

SNS To publish events from our instances

Autoscaling To generate events as our application scales up & down

SQS To persist the event messages for processing

S3 To store and serve content we create

CloudFormation To build our system as a managed stack

DynamoDB To store all events from application

IAM To control the creation and management of resources

To do what? Mimic an application that implements auto-

scaling

Trap, transport and store the scaling events produced

Use a simple technique to produce pseudo-dynamic content from S3

To do what? Mimic an application that implements auto-

scaling

Trap, transport and store the scaling events produced

Use a simple technique to produce pseudo-dynamic content from S3

An exercise beyond compute and storage!

There’s a movie you can view:

http://youtu.be/lb9qPhxIVNI I’ll show this link again at the end

http://youtu.be/lb9qPhxIVNI


This demo is just an illustration of what you can do with these services

Built in this way…


Auto scaling Group

An arbitrary application that we

can scale up and down

Auto scaling Group

SNS notification from auto scaling

group SNS event body as JSON

SQS queue to persist event


Messages produced when instances are

started or terminated

Auto scaling Group




DynamoDB table holding instance details

Monitoring instance

Auto scaling Group


Auto scaling Group





S3 bucket holding dashboard web content

Auto scaling Group

Monitoring instance


EC2 Instance Contents


Monitoring instance

Python script Read SQS queue and generate data for S3

Static site HTML, Javascript, css

reading data file


SNS & SQS Essential glue between applications

Simple Notification Service

Reliable Redundant storage

Scalable Unlimited number of messages

Simple CreateTopic, Subscribe, Publish

Flexible HTTP, Email, SQS Secure

Topic policies

Integrated EC2, CloudWatch, Auto Scaling

A subscription by an SQS queue for messages

published on this topic

A CloudWatch alarm that will publish to

an SNS topic

Simple Queue Service

Reliable Queues store messages across

availability zones

Scalable Designed for unlimited services

reading unlimited number of messages

Simple CreateQueue, SendMessage,

ReceiveMessage, DeleteMessage

Inexpensive Low per request fees Secure

Authentication

Performance Excellent throughput

Python Application A


>>> import boto

>>> conn = boto.connect_sqs()

>>> q = conn.create_queue('myqueue')


>>> import boto



>>> from boto.sqs.message import Message

>>> m = Message()

>>> m.set_body('This is my first message.')

>>> status = q.write(m)


Python Application B

>>> import boto




>>> m = Message()



>>> m = q.read(60)

>>> m.get_body()

Message not visible to other applications for 60 seconds


Python Application B

X

>>> import boto




>>> m = Message()



>>> m = q.read(60)

>>> m.get_body()

>>> q.delete_message(m)

Message not visible to other applications for 60 seconds

The core trick…

Auto-scaling SNS SQS

Event notifications

Persistence of event data

The core trick…

Auto-scaling SNS SQS

Event notifications

Persistence of event data

You can do this from anything to anything

as-create-launch-config

--image-id <ami id>

--instance-type t1.micro

--group <security group>

--launch-config my-launch-cfg

Create a Auto Scaling launch configuration:

as-create-auto-scaling-group my-as-group

--availability-zones <az list>

--launch-configuration my-launch-cfg

--max-size 20

--min-size 1

Create a Auto Scaling group:

Amazon Resource Name

as-put-notification-configuration my-as-group

--topic-arn <arn-from-SNS-topic>

--notification-types

autoscaling:EC2_INSTANCE_LAUNCH,

autoscaling:EC2_INSTANCE_TERMINATE

as-put-notification-configuration my-as-group

--topic-arn <arn-from-SNS-topic>

--notification-types

autoscaling:EC2_INSTANCE_LAUNCH,

autoscaling:EC2_INSTANCE_TERMINATE

autoscaling:EC2_INSTANCE_LAUNCH autoscaling:EC2_INSTANCE_LAUNCH_ERROR autoscaling:EC2_INSTANCE_TERMINATE autoscaling:EC2_INSTANCE_TERMINATE_ERROR

A subscription by an SQS queue for messages

published on this topic

{

"Type" : "Notification",

"MessageId" : <message id>,

"TopicArn" : <arn>,

"Subject" : "Auto Scaling: termination for group \"SNS-Dashboard-ASG\"",

"Message" : ”…",

"Timestamp" : "2013-05-21T09:13:09.555Z",

"SignatureVersion" : "1",

"Signature" : ”…",

"SigningCertURL" : "https://sns.us-east-

1.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem",

"UnsubscribeURL" : "https://sns.us-east-

1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-

1:241861486983:SNS-Dashboard-ASNotifications-7GU41DCQW8HC:ed30bf6e-582c-4fd2-8e07-

28f7d1ac6278"

}

{

"Type" : "Notification",

"MessageId" : <message id>,

"TopicArn" : <arn>,

"Subject" : "Auto Scaling: termination for group \"SNS-Dashboard-ASG\"",

"Message" : ”…",

"Timestamp" : "2013-05-21T09:13:09.555Z",

"SignatureVersion" : "1",

"Signature" : ”…",

"SigningCertURL" : "https://sns.us-east-

1.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem",

"UnsubscribeURL" : "https://sns.us-east-

1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-

1:241861486983:SNS-Dashboard-ASNotifications-7GU41DCQW8HC:ed30bf6e-582c-4fd2-8e07-

28f7d1ac6278"

}

{\"StatusCode\":\"InProgress\",\"Service\":\"AWS Auto Scaling\",\"AutoScalingGroupName\":\"SNS-Dashboard-ApplicationServerGroup-K61R5797WCMA\",\"Description\":\"Terminating EC2 instance: i-8bb679eb\",\"ActivityId\":\"dfc1181b-0df8-47dc-aa8d-79e13b8a33d1\",\"Event\":\"autoscaling:EC2_INSTANCE_TERMINATE\",\"Details\":{},\"AutoScalingGroupARN\":\"arn:aws:autoscaling:us-east-1:241861486983:autoScalingGroup:77ef2778-ded1-451a-a630-6a35c8e67916:autoScalingGroupName/SNS-Dashboard-ApplicationServerGroup-K61R5797WCMA\",\"Progress\":50,\"Time\":\"2013-05-21T09:13:09.442Z\",\"AccountId\":\"241861486983\",\"RequestId\":\"dfc1181b-0df8-47dc-aa8d-79e13b8a33d1\",\"StatusMessage\":\"\",\"EndTime\":\"2013-05-21T09:13:09.442Z\",\"EC2InstanceId\":\"i-8bb679eb\",\"StartTime\":\"2013-05-21T09:12:20.323Z\",\"Cause\":\"At 2013-05-21T09:12:02Z a user request explicitly set group desired capacity changing the desired capacity from 5 to 1. At 2013-05-21T09:12:19Z an instance was taken out of service in response to a difference between desired and actual capacity, shrinking the capacity from 5 to 1. At 2013-05-21T09:12:19Z instance i-8fdbafed was selected for termination. At 2013-05-21T09:12:19Z instance i-8ddbafef was selected for termination. At 2013-05-21T09:12:20Z instance i-8bb679eb was selected for termination. At 2013-05-21T09:12:20Z instance i-85b778e5 was selected for termination.\"}

http://bootstrapping-

assets.s3.amazonaws.com/as-register-

instances.template

Template available:

Subscription

Notification

Topic

SQS Queue

Auto Scaling

Group

Subscription

Notification

Topic

SQS Queue

Auto Scaling

Group

"NotificationConfiguration" : {

"TopicARN" : { …"ASNotifications"

},

"NotificationTypes”

[ "autoscaling:EC2_INSTANCE_LAUNCH”,

"autoscaling:EC2_INSTANCE_TERMINATE" ]

},

Adding a notification to a topic for Auto Scaling group

events

Subscription

Notification

Topic

SQS Queue

Auto Scaling

Group

"ASNotifications" : {

"Type" : "AWS::SNS::Topic",

"Properties" : {

"Subscription" : [ {

"Endpoint" : { … ASNotificationsQueue …},

"Protocol" : "sqs"

} ]

}

},

"NotificationConfiguration" : {

"TopicARN" : { …"ASNotifications"

},

"NotificationTypes”

[ "autoscaling:EC2_INSTANCE_LAUNCH”,

"autoscaling:EC2_INSTANCE_TERMINATE" ]

},

Subscription to topic from an SQS queue

We now have events in SQS Let’s do something with them…

Auto scaling Group





Messages produced when instances are

started or terminated

Auto scaling Group





Monitoring instance

Auto scaling Group


Auto scaling Group





S3 bucket holding dashboard web content

Auto scaling Group

Monitoring instance


EC2 Instance Contents


Monitoring instance

Python script Read SQS queue and generate data for S3

Static site HTML, Javascript, css

reading data file


Read messages

from SQS queue

Write data to

DynamoDB table

Form JSON file

from updated

results

Write file to S3

for javascript to

interpret

http://bootstrapping-

assets.s3.amazonaws.com/as-node-manager.py

Script available

1. Read SQS Queue

# Connect to SQS and open queue

sqs = boto.connect_sqs()

queue = sqs.get_queue(sqs_queue_name)

queue.set_message_class(RawMessage)

while True:

rs = queue.get_messages(num_messages=10)

for raw_message in rs:

# Parse JSON message

envelope = json.loads(raw_message.get_body())

message = json.loads(envelope['Message'])

# Trap the EC2_INSTANCE_LAUNCH event

if message['Event'] == 'autoscaling:EC2_INSTANCE_LAUNCH':

save_instance(message['EC2InstanceId'], ddb_table_name)

# Trap the EC2_INSTANCE_TERMINATE event

elif message['Event'] == 'autoscaling:EC2_INSTANCE_TERMINATE':

delete_instance(message['EC2InstanceId'], ddb_table_name)

# Delete the message from the queue and continue polling

queue.delete_message(raw_message)





while True:


















while True:


















while True:














2. Write to DynamoDB

def save_instance(instance_id, ddb_table_name):

instance = get_instance(instance_id)

# Connect to DynamodB (using key from env) and get table

ddb = boto.connect_dynamodb()

table = ddb.get_table(ddb_table_name)

# Create a new record for this instance

item = table.new_item(

hash_key=instance.id,

attrs = {

'pub_hostname': instance.public_dns_name,

'pub_ip': instance.ip_address,

'priv_hostname': instance.private_dns_name,

'priv_ip': instance.private_ip_address,

'ami_id': instance.image_id,

'region': instance.region.name,

'availability_zone' : instance.placement,

'terminated': 'false'

}

)

# Save the item to DynamoDB

item.put()









attrs = {









}

)


item.put()









attrs = {









}

)


item.put()

Item ‘fields’

Item key









attrs = {









}

)


item.put()

63

def delete_instance(instance_id, ddb_table_name):

# Connect to DynamodB and get table



# Get the item to soft delete

item = table.get_item(instance_id)

# Update the terminated flag

item['terminated'] = 'true'


item.put()










item.put()










item.put()

67

3. Write to S3

def write_instances_to_s3(instances_json, s3_output_bucket, s3_output_key):

# Connect to S3 and get the output bucket

s3 = boto.connect_s3()

output_bucket = s3.get_bucket(s3_output_bucket)

# Create a key to store the instances_json text

k = Key(output_bucket)

k.key = s3_output_key

k.set_metadata("Content-Type", "text/plain")

k.set_contents_from_string(instances_json)

def write_instances_to_s3(instances_json, s3_output_bucket, s3_output_key):

# Connect to S3 and get the output bucket

s3 = boto.connect_s3()

output_bucket = s3.get_bucket(s3_output_bucket)

# Create a key to store the instances_json text

k = Key(output_bucket)

k.key = s3_output_key

k.set_metadata("Content-Type", "text/plain")

k.set_contents_from_string(instances_json)

{

"instances":[

{

"id":"i-5525c932",

"terminated":"true",

"ami_id":"ami-7341831a",

"availability_zone":"us-east-1d",

"region":"RegionInfo:us-east-1",

"pub_ip":"107.21.167.7",

"pub_hostname":"ec2-107-21-167-7.compute-1.amazonaws.com",

"priv_ip":"10.201.2.233",

"priv_hostname":"domU-12-31-39-13-01-1B.compute-1.internal"

},

{

"id":"i-bb6a86dc",

"terminated":"false",

"ami_id":"ami-7341831a",

"availability_zone":"us-east-1a",

"region":"RegionInfo:us-east-1",

"pub_ip":"174.129.82.128",

"pub_hostname":"ec2-174-129-82-128.compute-1.amazonaws.com",

"priv_ip":"10.242.211.185",

"priv_hostname":"ip-10-242-211-185.ec2.internal"

}

]

}

…and now we have data in S3 we can build a web view…

Instances.txt

S3 bucket

From

monitoring

app

HTML/CSS/JS

Instances.txt

S3 bucket

From

monitoring

app

HTML/CSS/JS

Web page

load from

S3

Periodic refresh

Instances.txt

S3 bucket

From

monitoring

app

HTML/CSS/JS

Web page

load from

S3

jQuery get

instance

data from

S3

Periodic refresh

S3 Bucket

css

img

js

index.html

instances.txt

S3 Bucket

css

img

js

index.html

instances.txt JSON data

jQuery.getJSON()

jQuery.getJSON()

Javascript functions in index page hosted in S3

Built around an AJAX ‘get’ of instances JSON in S3

some jQuery table

selectors/modifiers

+

Want to try this yourself? View the video tutorial here:


And grab the CloudFormation template here: http://bootstrapping-assets.s3.amazonaws.com/as-

register-instances.template



1. Create security groups 2. Create a notification of type SQS 3. Create SQS queue 4. Create auto-scaling launch configs & groups 5. Add auto-scaling notifications to SQS SNS notification 6. Create S3 bucket 7. Create DynamoDB table 8. Start instances 9. Bootstrap monitoring application

Summary

Pub/Sub models with SNS S3 & pseudo dynamic content

More than compute & storage

Reliable delivery with SQS DynamoDB for high performance

Summary

Pub/Sub models with SNS S3 & pseudo dynamic content

More than compute & storage

Reliable delivery with SQS

Given you some ideas? Introduced you to some handy services? Helped you with some CloudFormation?

DynamoDB for high performance

Summary

View the video: http://youtu.be/lb9qPhxIVNI

Find out more: aws.amazon.com






Technology

Masterclass Webinar: Application Services and Dynamic Dashboard