JamesRH2007.06.11

http://msblogs/jamesrh

7 major AWS Services (http://aws.amazon.com) Amazon E-Commerce Service (ECS) Amazon Elastic Compute Cloud Beta (EC2) Amazon Historical Pricing Amazon Mechanical Turk Beta Amazon Simple Storage Service (S3) Amazon Simple Queue Service (SQS) Alexa Web Services

Originally planning to write against both EC2 & S3 but EC2 beta was closed until recently There are EC2 scaling issues (confirmed by Bezos at Web2.0) S3 Scaling well:

5B objects Peak transaction rate:

Day: 923M Second: 16,67

This demo app was written to learn the S3 interfaces, their strengths, and weaknesses

04/18/23 2

Went to aws.amazon.com Signed into Amazon account

Same one used for Amazon shopping Fill in additional contact information

agreed to terms and conditions AWS provides Access Key & Secret Key Tested using www.awszone.com

Simple web app supporting most AWS operations via SOAP Failed with: “Client.NotSignedUp”

Email showed up later complaining about credit card out of date

I corrected the payment issue and was up and running User provisioning is fast and efficient

Overall, quick and easy process

04/18/23 3

Blob store with per blob metadata Blobs supported up to 5GB

~36.4 hour upload via Comcast “broadband” Metadata up to 4k in name/value pairs (2k of user

metadata) Up to 100 ACLs on each object (part of system metadata)

Read, write, read_ACP, write_ACP, and full_control Owner has full control and is not changeable

Each blob stored in a bucket Bucket name space shared by ALL S3 users All buckets have a single unchanging owner

Each blob has a URI identifier in a flat per-bucket namespace Directory hierarchy simulated

Get(key=“C:/X/Y/Z.txt”) gets the file Get(key=“C:/X”) gets all objects with leading “C:/X/” in URI Get(key=“C:/X/”, separator=“/”) gets all objects with leading “C:/X/” but

no subsequent “/”

04/18/23 4

Client libraries offered as samples for: Java SOAP, Java REST, C# SOAP, C# REST,

Ruby REST, Python REST, & Perl REST BitTorent also supported Docs at protocol layer rather than APIs API implementations uneven without

docs C# SOAP has max send of 1M (can use

DIME attachment for larger) C# REST doesn’t support direct streaming

Source provided so quality & functionality issues all have solution

04/18/23 5

conn = new AWSAuthConnection(key, secretKey);res = conn.delete(Bucket, key, null);Soap request Headers:

- <SOAP-ENV:Body>- <m:DeleteObject xmlns:m="http://s3.amazonaws.com/doc/2006-03-01/">  <m:AWSAccessKeyId>10T01CX2MA2M9WD25002</m:AWSAccessKeyId>   <m:Signature>NbT52AjYRzhHdA490B+T20KBYYg=</m:Signature>   <m:Timestamp>2007-03-14T20:10:28.000Z</m:Timestamp>   <m:Bucket>10T01CX2MA2M9WD25002-Archiver</m:Bucket>   <m:Key>C:/JamesRH/dev/AWS/tst/test.tst*1</m:Key>   </m:DeleteObject>  </SOAP-ENV:Body>

SOAP response Headers- <soapenv:Body>- <DeleteObjectResponse xmlns="http://s3.amazonaws.com/doc/2006-03-01/">- <DeleteObjectResponse>  <Code>204</Code>   <Description>No Content</Description>   </DeleteObjectResponse>  </DeleteObjectResponse>  </soapenv:Body>

04/18/23 6

REST Access Key:AWS : Base64(hmac-sha1(VERB, Content-MD5, Content type, Date, AMZ headers, resourceKey ))

SOAP Model Doesn’t sign

S3 Archiver is a version store Registered directories & sub-directories contents are stored in S3 Each new file version is stored separately Any version of any file can be restored in place Multiple versions of a file can be restored with version suffixes on

filename Any file version can be deleted from S3

Roughly 3,500 lines of C# written against C# REST interface

04/18/23 7

Performance: 12,220 files, 12,500 versions totaling

5.1 GB synchronized in under 30 seconds (local-only operation)

Actual transfers very slow ranging from roughly 1GB/hour to as low as 100 MB/hour (28 KBs)

Issues mostly network bandwidth-related on the perf side and standard internet flakiness on the reliability side

Files stored in S3 with version numbers C:\JamesRH\OpsFriendlyApps.docsx*10

Metadata as name/value pairs with each file: Version Size Hash Last Write Create Time Access Time Archive Time

Fetching metadata for 1,000s of files slow so have a cache stored in S3 S3_Archiver.xml

Large scale testing required real data so implemented encryption Single file operations slow and very uneven in performance so need

to run all non-metadata-only operations asynchronously On scale-up to 1000s of files S3_Archiver.xml gets large and slow so

implemented compression On scale-up to 10s of 1000s of files S3_Archiver.xml still slow to

transfer so check hash and avoid transfers when possible

04/18/23 8

• C:/user/file1*1• C:/user/file1*2• C:/user/file2*1• C:/file3*1• cache• cache*C:/usr/

libary1• cache*C:/usr/

library2

04/18/23 9

Error rate ~4 in 1,000 and similar across different networks: Comcast:

Recv: 43 to 567.19 KBs Send: 5 to 52 KBs

Microsoft: Recv : 317 to 10,935 KBs Send: 437 to 5,556 KBs

Home Large File: Send Fast/Slow ratio: 9.8x Recv Fast/Slow ratio: 12.9x Recv/Send ratio: 10.8x

Work Large File: Send Fast/Slow ratio: 12.7x Recv Fast/Slow ratio: 34.4x Recv/Send ratio: 1.9x

Conclusion: Send to receive difference ranges from a factor of 2 to an order

of magnitude 2 orders of magnitude spread in throughput Internet error rate high 5+ orders of magnitude higher latency than local disk

04/18/23 10Source: one week hourly perf & reliability probe of S3 at 1k, 10k, 100k, 1m, & 25m

04/18/23 11

Kbs

4 errors

Errors show as negative throughput

04/18/23 12

Kbs

Errors show as negative throughput

1 error

04/18/23 13

Kbs

Errors show as negative throughput

2 errors

2 errors3 errors

04/18/23 14

Kbs

Errors show as negative throughput

1 error

Summary of This Month's Activity as of March 14, 2007 Billing Cycle for this Report: March 1 - March 31, 2007

  Usage Charges Rate   Usage     Totals                   Amazon Simple Storage Service                $0.20 per GB of data transferred (details)    15.174 GB      $3.04         $0.15 per GB-Month of storage used (details)    0.214 GB-Mo      $0.04                 Total: $3.08            

All web services are sold by Amazon Digital Services, Inc.

Summary of Last Month's Statement: February 1 - February 28, 2007                    Total Charges for February 1 - February 28, 2007: $0.07 

Amazingly inexpensive: Developed a 3,000 line interactive system storing

1,000s of files and several GB stored with many test cycles at $3.11

04/18/23 15

Current bandwidth price (through May 31, 2007) $0.20 / GB – uploaded $0.20 / GB - downloaded

New bandwidth price (effective June 1, 2007) $0.10 per GB - all data uploaded $0.18 per GB - first 10 TB / month data downloaded $0.16 per GB - next 40 TB / month data downloaded $0.13 per GB - data downloaded / month over 50 TB Data transferred between Amazon S3 and Amazon EC2 will remain free

New request-based price (effective June 1, 2007) $0.01 per 1,000 PUT or LIST requests $0.01 per 10,000 GET and all other requests No charge for delete requests

Storage will continue to be charged at $0.15 / GB-month used

04/18/23 16