1. Amazon Web Services Introduction & Discussion

2. Amazon Web Services

Overview of Amazon's Services

• Explore each type of service

Not implementation specific [1]

Open Discussion

[1] http://highscalability.com/amazon-architecture 3. Amazon Web Services The Picture Associates Alexa Marketplace DevPay WebStore Prime Grocery Historical Pricing [1] http://highscalability.com/paper-dynamo-amazon-s-highly-available-key-value-store 4.

Small Pieces Loosely Joined

...There are intriguing assertions: thatthe Web is "broken on purpose"and that its many pockets oferroneous information and its available forums [...] letpeople feel more comfortable with their own inherent imperfections . [...] Weinberger's analysis, thoughoccasionally facile and too relentlessly optimistic and overstated , is surely destined to be the subject of furious debate in chat rooms the cyber-world over. --H. O'Billovich [1]

[1] http://www.amazon.com/Small-Pieces-Loosely-Joined-Unified/dp/B00008NRGI/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1204731761&sr=8-1 Amazon Web Services Defined 5.

Small Pieces Loosely Joined

Think Unix Model

• Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. -- Doug McIlroy [1]

[1] [Salus] Peter H. Salus. A Quarter-Century of Unix. Addison-Wesley. 1994. ISBN 0-201-54777-5 Amazon Web Services Defined 6.

Small Pieces Loosely Joined

Think Unix Model

Single Deployment

Embrace the Power of Simplicity

Open Collaboration

Leverages Momentum Makers

Force Multiplication

• Estimates of 40% of sales from affiliates [1]

Amazon Web Services Defined [1] http://en.wikipedia.org/wiki/Amazon.com 7. Amazon Web Services From the Ground Up

Provides a Storage Service Since April 2006 [1]

Geographically distributed content network

• U.S. and European Union

• REST and SOAP interfaces over HTTP

• A BitTorrent(TM) protocol interface is provided

Does have a Service Level Agreement [2]

• 25% for < 99%

• 10% for > 99% and < 99.9%

• 3 nines implies >> from boto.sqs.connection import SQSConnection, Message

• • >>> conn = SQSConnection('', '')

• • >>> q = conn.create_queue('myqueue', 60)

• • >>> m = Message()

• • >>> m.set_body('This is my first message.')

• • >>> rs = q.write(m)

• • >>> rs = q.get_messages()

• • >>> rs[0].get_body()

• • >>> q.delete_message(rs[0])

[1] http://boto.googlecode.com/svn/trunk/doc/sqs_tut.txt [2] http://aws.typepad.com/aws/2007/05/sqs_super_queue.html Amazon Web Services Ground Control to Major Tom... 15. Amazon Web Services Show me the Money

Supports multiple forms of payment

• Permits true micro-payment [1]

• Costs vary based on Amazon's costs

Amazon manages security and fraud responsibility

GateKeeper enforces policies for senders and receivers

Explicit Payment Instructions language

• only five transactions or $50 per month" or "refuse payment after August 1"

• Three token exchange

Integrates with Amazon's existing customer experience

• Tens of Millions of existing customers

• Pay Now Widget

• Promotes market places for intermediaries

[1] http://paulstamatiou.com/2007/08/04/why-you-shouldnt-ignore-amazons-new-fps [2] http://www.amazon.com/gp/browse.html?node=342430011 [3] http://docs.amazonwebservices.com/AmazonFPS/2007-01-08/FPSGettingStartedGuide/ 16. Amazon Web Services The Sky's the Limit [1] http://www.amazon.com/b/?node=201590011

Provides programmatic compute capacity

• Including pre-configured appliance images

... allows you to obtain and configure capacity with minimal friction [1]

The Basic Process

• Build or Identify an appropriate AMI

• Configure Security and Access

• Start, Monitor and Stop Instances

• Pay

17. Amazon Web Services The Sky's the Limit

What's an AMI ? -Amazon Machine Instance

• Golden Master of the image you want to run

• Linux only, Windows is a hack [1]

• AMI's can be private [2] Vendor tools like rPath [3]

• Leverage an Open Operating Systems bazaar

• • Some AMI's can have associate costs per instance

Multiple Instances can run from a single AMI

• By default, limited to 20, simple request for more

Getting started with EC2 [4]

Mike Culver - Introductory video [5]

[1] http://www.enomaly.net/wiki.1137+M5c1693b2f05.0.html [2]http://docs.amazonwebservices.com/AmazonEC2/dg/2006-06-26/creating-an-ami.html [3] http://highscalability.com/product-rpath-creating-and-managing-virtual-appliances [4] http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/ [5] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=583&categoryID=100 18. Amazon Web Services The Sky's the Limit

Example:

• > ec2-describe-images -x all

• • IMAGE ami-23b6534a ec2-public-images/fedora-7-apache-manafest.xml

• > ec2-add-keypair j-keypair

• • (generates ssh keypairs)

• > ec2-run-instance ami-23b6534a -k j-keypair

• • (could take up to a few minutes)

• > ec2-authorize default -p 22

• • (Authorize port 22 for the default security group)

• > ec2-describe-instances

• • (Shows instance info including DNS name)

• > ec2-terminate-instances

• > ec2run ami-e3a5408a -n 20 -g appserver

19. Amazon Web Services The Sky's the Limit [1] http://www.amazon.com/b/?node=201590011 [2] http://www.amazon.com/b/?node=201590011#measure [3] http://www.amazon.com/b/ref=sc_fe_c_0_201590011_2?ie=UTF8&node=370375011&no=201590011&me=A36L942TSJ2AJA

Small Instance - $.10 / hour

• 1.7 GB of memory,

• 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit)

• 160 GB of instance storage, 32-bit platform

Large Instance - $0.40 / hour

• 7.5 GB of memory

• 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)

• 850 GB of instance storage, 64-bit platform

Extra Large Instance - $0.80 / hour

• 15 GB of memory

• 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)

• 1690 GB of instance storage, 64-bit platform

1 EC2 Compute Unit is ~ 1.0-1.2 GHz 2007 Opteron or 2007 Xeon

Bandwidth costs of still apply at 256Mb / sec

20. Amazon Web Services The Sky's the Limit

Uses x509 certificates for Developer Auth.

• SSH keypairs for instance authenticaion

Instances can belong to security groups

Network is dynamic via NAT

• IP's and hostnames change

Momentum

• RedHat is deploying a cloud on EC2 [1]

• Facebook [2]

• Hadoop [3]

• EC2 for Python Programmers [4]

[1] http://www.redhat.com/solutions/cloud/ [2] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1044&ref=featured http://www.amazon.com/gp/browse.html?node=391557011 [3] http://blog.awswebshop.com/2008/02/27/taking-massive-distributed-computing-to-the-common-man-hadoop-on-amazon-ec2s3/ [4] http://jimmyg.org/2007/09/01/amazon-ec2-for-people-who-prefer-debian-and-python-over-fedora-and-java/ 21. Amazon Web Services From the Ground Up

Amazon Mechanical Turk

• Launched in late 2005 [1]

[1] http://www.egge.net/~savory/blog_mar_04.htm 22. Amazon Web Services From the Ground Up

Amazon Mechanical Turk

• Launched in late 2005 [1]

Artificial Artificial Intelligence - Jeff Bezos

[1] http://en.wikipedia.org/wiki/Amazon.com 23. Amazon Web Services From the Ground Up [1] http://www.amazon.com/gp/browse.html?node=15879911 [2] http://www.techcrunch.com/2007/09/08/search-for-steve-fossett-expands-to-amazons-mechanical-turk/

Humans augment computational requirements

Let someone else solve complex AI problems

• Get along with the business of making money

Built around Human Intelligence Tasks (HITs)

Examples;

• Transcribe audio data

• Are these paragraphs synonymous?

• Find the business in this photo

• Write a review...

• Search for survivors

• • So far, searchers have found half a dozen previously unknown crash sites, but no sign of Fossetts plane. [2]

24. Amazon Web Services The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100

Mitch Garnaat

• Author of Bono

DVS transcoding

Tutorial [1]

25.

Read a message from the input queue -SQS

Based on message, retrieve an input file -S3

Perform our video conversion processing -EC2

Store the generated files -S3

Write a message the the output queue -SQS

Delete the input message from the input queue-SQS

Amazon Web Services The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100 26. Amazon Web Services The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100

Input Message

• Bucket: garnaat_fileit

• InputKey: f84e4a20b571abc69baf2277d193e596

• Date: Tue, 20 Feb 2007 17:21:21 GMT

• OriginalFileName: MVI_3113.AVI

• Size: 1126472

Output Message

• Bucket: garnaat_fileit

• InputKey: f84e4a20b571abc69baf2277d193e596

• Date: Tue, 20 Feb 2007 17:21:21 GMT

• OriginalFileName: MVI_3113.AVI

• Size: 1126472

• OutputKey: e69e376be5af6f88f81d3e31adf27988;type=video/quicktime

• Server: ConvertVideo

• Host: domU-12-31-34-00-02-82

• Service-Read: Wed, 21 Feb 2007 01:28:14 GMT

• Service-Write: Wed, 21 Feb 2007 01:28:27 GMT

27. Amazon Web Services The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100

Define the Service Instance

• • from boto.services.service import Service

• • import os

• • class ConvertVideo(Service):

• • ProcessingTime = 30

• • Command = """ffmpeg -y -i %s -f mov -r 29.97 -b 1200kb -mbd 2 -flags

• • +4mv+trell -aic 2 -cmp 2 -subcmp 2 -ar 48000 -ab 192 -s 320x240

• • -vcodec mpeg4 -acodec aac %s"""

• • def process_file(self, in_file_name, msg):

• • out_file_name = os.path.join(self.working_dir, 'out.mov')

• • command = self.Command % (in_file_name, out_file_name)

• • os.system(command)

• • return [(out_file_name, 'video/quicktime')]

28. Amazon Web Services The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100

$ boto/services/submit_files.py -b myvideos -q vc-input ~/movies

...

50 files successfully submitted.

$ boto/services/start_service.py -r -m boto.services.convertvideo

-c ConvertVideo -a ami-2eba5f47 -i vc-input -o vc-output

-e mitch@garnaat.com -n 1

python boto/services/get_results.py -q vc-status ~/movies

retrieving file: MVI_3110.mov

...

50 results successfully retrieved.

Minimum Processing Time: 2

Maximum Processing Time: 58

Average Processing Time: 17.820000

Elapsed Time: 896

Throughput: 3.348214 transactions / minute

$ python boto/services/get_results.py -q test-status ~/movies

retrieving file: MVI_3110.mov

500 results successfully retrieved.

...

Minimum Processing Time: 2

Maximum Processing Time: 60

Average Processing Time: 17.794000

Elapsed Time: 928

Throughput: 32.327586 transactions / minute

Submit Files and Setup Queue Startup EC2 worker(s) 1 Instance 10 Instances, 500 files 29. Amazon Web Services The Example [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=691&categoryID=100

Okay... so what ?

$.004 per movie, $8k server => 2,000,000 movies

24 hrs * 60 min = 1440 movies / day * 365 days

525600 movies / year * 4 cameras = 2.1M

30.

This matters to Amazon !

• For the period of 4Q07, bandwidth utilized for S3 & EC2 was greater than the combined bandwidth of Amazon.coms global websites. [1]

• • In Dec '07 Amazon was the #7 website in the US [2]

• Same period 330,000+ new developers registered

• • Up 30,000+ from the previous last quarter.

Success Stories [3]

Startup Challenge [4]

Amazon Web Services So Who, What, Why [1] http://biz.yahoo.com/bw/080130/20080130006013.html [2] http://www.comscore.com/press/release.asp?press=2000 [3] http://www.amazon.com/gp/browse.html?node=182241011 [4] http://www.amazon.com/gp/browse.html?node=377634011 31.

It's not just Amazon

• Google [1]

• Microsoft! [2]

• Yahoo

IBM [3][4][5][6][7]

Additional Examples

• BigTable, GFS, MapReduce

• CouchDB

• Hadoop

Amazon Web Services So Who, What, Why [1] http://feedblog.org/2007/12/14/google-vs-amazon-in-open-infrastructure/ [2]http://www.microsoft.com/sql/dataservices/default.mspx [3] http://redmonk.com/jgovernor/2007/04/19/why-ibm-should-acquire-amazon/ [4] http://www.ogf.org/OGF22/materials/1137/Irving+Wladawsky-Berger+Keynote.pdf [5] http://www.news.com/8301-10784_3-9817905-7.html [6] http://www-03.ibm.com/press/us/en/pressrelease/22613.wss [7] http://blogs.zdnet.com/SAAS/?p=415 32. Amazon Web Services Take Away

Customers can leverage the cloud !

• Energy getting invested up the stack

The Industry is embracing simplicity

• Functional programming renaissance

• • Analogs to the resurgence of the mainframe

• Application Trend - trust (but verify)

• • Intuition over specificity

Amazon doesn't re-instantiate components

• Insanity - doing the same thing over and over and expecting different results.

Define Commodity - ...

33. Amazon Web Services Thinkovers [1] http://www.kitchensoap.com/2008/02/27/when-do-you-get-too-big-to-use-cloudy-stuff/ [2] http://www.readwriteweb.com/archives/amazon_haas_hardware_as_a_service.php

[1]

[2]