Rackspace & Akamai vs. Amazon & CloudFront for a Django site

CAPABILITIES

Sep Dehpourlinkedin.com/in/sepehrgithub.com/erasmose

IntroTradi1onal vs. CloudRackspace vs. Amazon

Akamais3 objectsToolsEnd LA Django

Nov 19 2013Sep Dehpour

www.linkedin.com/in/sepehrwww.github.com/erasmose

Moving to the Clouds: Amazon S3 & Cloud Front

vs. Rackspace Cloud Files & Akamai

Wednesday, November 20, 13

http://www.linkedin.com/in/sepehr


http://www.github.com/erasmose


CAPABILITIES



Akamais3 objectsToolsEnd

Sep Dehpour is the primary engineer on a Django site that has over 1.5 million page views per day.

His responsibilities range from systems administration, Chef DevOps, performance optimizations to new feature development.

Sep is an avid skier and likes to think he is a good surfer too, but that shit ain't easy.

About the presenter


CAPABILITIES




In this talk: Comparison of the 2 major cloud providers for storing and serving sta1c files and the challenges for a Django site.

Rackspace & Akamai vs. Amazon & CloudFront


CAPABILITIES




Serving Sta1c filesTradi1onal

Nginx/Apache

Cloud

CDN

OriginWednesday, November 20, 13

CAPABILITIES




Storing & Serving Sta1c files

CDN

Origin

User


CAPABILITIES




Comparison in glance

Pay as you grow from 10¢/GB/Mo

Akamai CDN at 12¢/GB at 213 PoP

Store files and media of any size

Highly scalable, redundant

Pay as you grow from 9.5¢/GB/Mo

Cloudfront CDN at 12¢/GB

Store files and media of any size

Highly scalable, redundant


CAPABILITIES




CDN Comparison in glance

Akamai is delivering 20-30% of web

Apple, Netflix, Amazon.com

1,200 points of presence (PoP)

But Rackspace plan uses 213 PoP

Akamai NetSession Interface peer-to-peer

Nasa, PBS

40 super PoP

no peer-to-peer

Tip: Use hQp://www.cdnplanet.com/tools/cdnfinder to find the CDN used to serve sta1c content.Put the full path to an image or CSS file to see what CDN is serving it.


http://www.cdnplanet.com/tools/cdnfinder

http://www.cdnplanet.com/tools/cdnfinder

CAPABILITIES




CDN Speed Comparison 1Mb object


CAPABILITIES




CDN Speed Comparison 12Kb object


CAPABILITIES




hQp://www.akamai.com/html/technology/visualizing_akamai.htmlAkamai Visualizing


http://www.akamai.com/html/technology/visualizing_akamai.html

http://www.akamai.com/html/technology/visualizing_akamai.html

CAPABILITIES




So what’s exactly on S3/Cloud Files?Container{ Containers at the root level holding objects

Object {

“key”:”path to file to simulate a nested folder structure”,

“etag”:MD5 checksum,

“content_type”:Mime Type,

“ttl”: time to expire on CDN

}, ...

}Wednesday, November 20, 13

CAPABILITIES




Toolsets/libraries

Rackspace control panel AWS control panelWeb Toolkit

Swift Client (written in Python) S3cmd (written in Python)Command line

CloudFuse S3fsVirtual File System

Pyrax BotoPython

Django Cumulus Django StoragesDjango

Go s3Django (other)


CAPABILITIES




Toolsets/libraries




Pyrax BotoPython


Go s3Django (other)


CAPABILITIES




Toolsets/libraries




Pyrax BotoPython


Go s3Django (other)


CAPABILITIES





Unlike tradi1onal file systems that essen1ally save data to and retrieve data from disk, virtual filesystems do not actually store data themselves. They act as a view or transla1on of an exis1ng file system or storage device.


CAPABILITIES




Buggy: CloudFuse S3fs (?)Virtual File System

Unlike tradi1onal file systems that essen1ally save data to and retrieve data from disk, virtual filesystems do not actually store data themselves. They act as a view or transla1on of an exis1ng file system or storage device.


CAPABILITIES




Toolsets/libraries




Pyrax BotoPython


Go s3Django (other)


CAPABILITIES




http://s3tools.org/s3cmdRun s3cmd --configureRun s3cmd ls to list all your buckets. Make a bucket withs3cmd mb s3://my-new-bucket-name List the contents of the buckets3cmd ls s3://bucket_name Upload a file into the buckets3cmd put addressbook.xml s3://logix.cz-test/addrbook.xml

Command lineS3cmd


http://s3tools.org/s3cmd

http://s3tools.org/s3cmd

CAPABILITIES




Toolsets/libraries




Pyrax BotoPython


Go s3Django (other)


CAPABILITIES




import pyraxpyrax.set_credential_file("/home/user/.pyrax.cfg")cf = pyrax.cloudfilesprint "list of containers:", cf.list_containers()

cont = cf.get_container("my_container")

#uploading a folderlocal = "path to a local folder"cf.sync_folder_to_container(folder_path=local, container=cont, include_hidden=False)

Command linePyrax


CAPABILITIES




#uploading filepth = "path to main.css on local"file_name_on_cloud = "static/assets/stylesheets/main.css"chksum = pyrax.utils.get_checksum(pth)obj = cont.upload_file(pth,obj_name= file_name_on_cloud, etag=chksum)

print "Calculated checksum:", chksumprint "Stored object etag:", obj.etag

Command linePyrax


CAPABILITIES




Toolsets/libraries




Pyrax BotoPython


Go s3Django (other)


CAPABILITIES




from boto.s3.connection import S3Connection

from boto.s3.key import Key

import sys

conn = S3Connection('AWS credentials', 'AWS

credentials')

bucket = conn.get_bucket('bucketname')

rs = bucket.list("Keys starting with this string")

for key in rs:

print key.name

Command lineBoto


CAPABILITIES




Toolsets/libraries




Pyrax BotoPython


Go s3Django (other)


CAPABILITIES




Toolsets/libraries




Pyrax BotoPython


Go s3Django (other)


CAPABILITIES




#s3utils.pyfrom storages.backends.s3boto import S3BotoStorage

class StaticRootS3BotoStorage(S3BotoStorage): location = 'static'

class MediaRootS3BotoStorage(S3BotoStorage): location = 'media'-----------------------------------------------------------------------#settings.pyDEFAULT_FILE_STORAGE = "my_project.s3utils.MediaRootS3BotoStorage"STATICFILES_STORAGE = "my_project.s3utils. StaticRootS3BotoStorage"AWS_STORAGE_BUCKET_NAME = storage_bucket_nameAWS_BUCKET_NAME = storage_bucket_nameAWS_S3_CUSTOM_DOMAIN = custom_domain

DjangoDjango Storages


CAPABILITIES




Always use Django default_storage so you can easily switch the back-end storage

from django.core.files.storage import default_storage

from django.core.files.base import ContentFile

path = default_storage.save('/path/to/file', ContentFile('new content'))

path u'/path/to/file'

default_storage.size(path)

11

default_storage.open(path).read()

'new content'

default_storage.delete(path)

default_storage.exists(path)

False



CAPABILITIES




But the 3rd party app that we were using did custom file writing instead of using Django default_storage



CAPABILITIES




And we had a few days till deploying



CAPABILITIES




Toolsets/libraries




Pyrax BotoPython


Go s3Python (other)


CAPABILITIES




Go S3: A light wrapper around Python Botohttps://github.com/erasmose/go_s3

When to use Go_s3?You have custom file read/write in your apps or 3rd party apps that are not compatible with “Django Storages” and you don’t have time to go through making it use “Django default storage” so it gets compatible with “Django Storages”

or

You want to have full control over what goes on S3 and its access control (ACL)

PythonGo S3


https://github.com/erasmose/go_s3


CAPABILITIES




Go S3: A light wrapper around Python Botohttps://github.com/erasmose/go_s3

#in your django settings:AWS_ACCESS_KEY_ID = "your AWS_ACCESS_KEY_ID"AWS_SECRET_ACCESS_KEY = "your AWS_SECRET_ACCESS_KEY"AWS_STORAGE_BUCKET_NAME = "your AWS_STORAGE_BUCKET_NAME"-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐#inside a custom save() for a Django model:

go_s3.connect_to_s3()try: go_s3.upload_file(file_full_path, full_path_s3, acl='private', del_after_upload=True) print go_s3.get_grants(full_path_on_s3)except: logger.error("Unexpected error dealing with s3 upload: ", exc_info=True) finally: go_s3.close_connection()

PythonGo S3




CAPABILITIES




ACL canned access control policy

a. private: Owner gets FULL_CONTROL. No one else has any access rights.

b. public-‐read: Owners gets FULL_CONTROL and the anonymous principal is granted READ access.

c. public-‐read-‐write: Owner gets FULL_CONTROL and the anonymous principal is granted READ and WRITE access.

d. authenticated-‐read: Owner gets FULL_CONTROL and any principal authenticated as a registered Amazon S3 user is granted READ access

PythonGo S3


CAPABILITIES




LA DjangoNov 19 2013Sep Dehpour

www.linkedin.com/in/sepehrwww.github.com/erasmose

Thanks to TrueCar for hos1ng






Technology

Rackspace & Akamai vs. Amazon & CloudFront for a Django site