Upload
kwittkopf
View
455
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
Amazon Web ServicesIntroduction to Cloud Storage
Kevin WittkopfEnterprise Solutions Architect…
Cloud StorageAWS offers multiple cloud-based storage options.
• Amazon S3 (Simple Storage Service)
• Amazon EBS (Elastic Block Storage)
• Amazon EC2 Local Instance Store (Ephemeral volumes)
• Amazon Cloudfront
Each has a unique combination of performance, durability, cost, and interfaceOften used to form a Storage Hierarchy
Storage by the Numbers
Cloudfront S3 EBS Local Disk
Scale Unlimited Temporary Storage
Unlimited Storage5TB objects100 name spaces (buckets)
1TB per volume~44TB per server
147 – 1690GB1-5 volumes
Durability N/A 99.999999999% (standard)99.99%(RRS)
.1-.5% AFR Ephemeral design requires application-level redundancy
Latency Low High Low Very Low
Management Zero Maintenance
Zero Maintenance Snapshots, RAID, Consistency
Manual Redundancy
Locality Global Region-Centric AZ-Specific In-machine
Interface REST or SOAPWeb Services
REST or SOAPWeb Services
Block Device Block Device
S3 Amazon Simple Storage Service
©2011 Amazon Web Services May not be reused or redistributed without permission
STORAGE
Amazon
EBSAmazon
SimpleDB
Amazon
S3
COMPUTE
Amazon
EC2
Amazon
RDS
©2011 Amazon Web Services May not be reused or redistributed without permission
Amazon Simple Storage Service
Amazon S3: Simple Storage Service
Highly scalable data storage in-the-cloud
Programmatic access via web services API
Is a Web Store, not a file system Optimized for WORM
Eventually consistent
Fast, highly available and durable
Economical
©2011 Amazon Web Services May not be reused or redistributed without permission
Region
Datacenter
Datacenter
Is Your Data Storage This Resilient?Note: Conceptual drawing only. Actual number of nodes & datacenters may vary
Datacenter
...Node
1Node
n
©2011 Amazon Web Services May not be reused or redistributed without permission
Amazon S3 Adoption Rate: Billions of Objects Stored
©2011 Amazon Web Services May not be reused or redistributed without permission
Q4 2006 Q4 2007 Q4 2008 Q4 2009 Q4 2010 Q2 20110
100
200
300
400
500
600
2.914
40
102
262
490
6 months!
Peak Requests:290,000+
per second
Amazon S3 Namespace
Amazon S3
bucket bucket
object object objectobject
bucket
object object
©2011 Amazon Web Services May not be reused or redistributed without permission
Amazon S3 Namespace
Amazon S3
mculver-images media.mydomain.com
Beach.jpg img1.jpg img2.jpg2009/maui/sun.jpg
public.blueorigin.com
Installer.msi img/pic1.jpg
©2011 Amazon Web Services May not be reused or redistributed without permission
Reduced Redundancy Storage Saves $$
Designed to provide 99.99% durability and 99.99% availability of objects over a given yearThis durability level corresponds to an average annual expected loss of 0.01% of objects Designed to sustain the loss of data in a single facility
©2011 Amazon Web Services May not be reused or redistributed without permission
S3 Pricing has Three Dimensions
Storage Data Transfer Requests
Tier Normal
RRS Tier Price Type Price
Frist 1 TB/mo
$0.14/GB
$0.093/GB All transfer in Free PUT, COPY, POST, or LIST
$0.01 per
1,000
Next 49 TB $0.125/GB
$0.083/GB First 1 GB out Free GET and all other requests
$0.01 per
10,000
Next 450 TB
$0.11/GB $0.073/GB <= 10 TB out $0.12/GB
Next 500 TB
$0.095/GB
$0.063/GB Next 40 TB out
$0.09/GB
Next 4000 TB
$0.08/GB
$0.053/GB Next 100 TB out
$0.07/GB
> 5000 TB $0.055/GB
$0.037/GB > 150 TB out $0.05/GB
©2011 Amazon Web Services May not be reused or redistributed without permission
Access Control Lists• Control access to bucket or to object• No inheritance from bucket to object• Set at create, write, or any time• List of up to 100 grants by:
– Owner– Email– Any AWS account– Anyone
• Permissions:– READ – Object or bucket– WRITE – Create, overwrite, delete objects in
bucket– READ_ACP – Read ACL for bucket or object– WRITE_ACP – Overwrite ACL for bucket or object– FULL_CONTROL – All of the above
©2011 Amazon Web Services May not be reused or redistributed without permission
Bucket Policy
Allow Jason or Mark to do anything in my S3 bucket when coming from the Office internal network
{"Statement":[{ "Effect":"Allow", "Principal":{"AWS":["4649-6425", "5243-0045"]}, "Action":"*", "Resource":"/mybucket/*", "Condition":{ "IpAddress":{"AWS:SourceIp":"176.13.0.0/12"}}}]}
©2011 Amazon Web Services May not be reused or redistributed without permission
S3 Multipart Uploads
Improved throughput—upload parts in parallel
Quick recovery from any network issues—Smaller part size minimizes the impact of restarting a failed upload due to a network error.
Pause and resume object uploads— upload object parts over time Once you initiate a multipart upload there is no expiry must explicitly complete or abort the multipart upload
Begin an upload before you know the final object size— can upload as you are create.
©2011 Amazon Web Services May not be reused or redistributed without permission
Even More Amazon S3 (not so simple, is it?)Website supportVersioningBrowser Upload to S3 LoggingRequestor PaysSigned URLsBittorrent supportIAMMeta-data
©2011 Amazon Web Services May not be reused or redistributed without permission
AWS Import/Export
Get your data into AWS faster - load it onto a portable storage device and ship it to an Amazon data centerFaster than Internet transfer and more cost effective than upgrading your connectivityUse cases: data migration, offsite backup, direct data interchange, disaster recovery
©2011 Amazon Web Services May not be reused or redistributed without permission
AWS Import/Export vs. S3 upload costs
100Gb 500Gb 1 Tb 2 Tb 3 Tb 5 Tb * 10 Tb **$0
$200
$400
$600
$800
$1,000
$1,200
Import/ExportS3 upload
AWS Import/Export vs. S3 upload time savings
100 Gb 500 Gb 1 Tb 2 Tb 3 Tb 5 Tb0
10
20
30
40
50
60
Import/ExportS3 uploadD
ays
STORAGE
Amazon
EBSAmazon
SimpleDB
Amazon
S3
COMPUTE
Amazon
EC2
Amazon
RDS
©2011 Amazon Web Services May not be reused or redistributed without permission
EBS Features
Persistent storageVolume lifetime is independent of any particular EC2 instance.
General purposeRaw, unformatted, block device. Use from Linux, Solaris or Windows.
PerformanceEqual to or better than local EC2 drive.
High reliabilityBuilt-in redundancy within availability zone.AFR (Annual Failure Rate) between 0.1% and 1%.
ScalableVolume sizes ranging from 1 GB to 1 TB.
EasyEasy to create, attach, back up, restore, and delete volumes.
©2011 Amazon Web Services May not be reused or redistributed without permission
EU West Region
Availability Zone A
Availability Zone B
Availability Zone C
Japan
Availability Zone A
Availability Zone B
Amazon Web Services: Durable & Available
US East Region
Availability Zone A
Availability Zone C
Availability Zone B
US West Region
Availability Zone A
Availability Zone B
Note: Conceptual drawing only. The number of Availability Zones may vary
Singapore
Availability Zone A
Availability Zone B
©2011 Amazon Web Services May not be reused or redistributed without written permission
EBS Volumes are in a Single Availability Zone
Note: Conceptual diagram only
Availability Zone A
EBS Volume 1
Availability Zone B
EBS Volume 2
©2011 Amazon Web Services May not be reused or redistributed without permission
Internet
AWS Cloud
How EBS Interacts With EC2 and S3
EC2 Availability Zone
EC2
S3 Storage
EBS
EC2 EC2
EBS EBS EBS EBS EBSEBS Snapshot
EBS Snapshot
EBS Snapshot
EBS Snapshot
EBS Snapshot
Create Snapshot
Clone From Snapshot
©2011 Amazon Web Services May not be reused or redistributed without permission
What’s the Difference BetweenAmazon S3 and Amazon EBS?
Amazon S3 EBS
Paradigm Object store File system
Performance Very fast Very very fast
Redundancy Across data centers Within data center
Security Public Key / Private Key
Visible only to your EC2
Pricing $0.14/GB/Mo. stored $0.10/GB/Mo. allocated
Access from the Net? Yes(1) No(2)
Typical use case Write once, read many It’s a disk drive
(1) Only with proper credentials, unless ACLs are world readable(2) Accessible from Net if mounted to server and set up as FTP, etc.
©2011 Amazon Web Services May not be reused or redistributed without permission
EBS Pricing
EBS Volumes $0.10 per GB-month
of provisioned storage $0.10 per 1 million I/O requests No charge for mounting/unmounting
volume
EBS Snapshots to Amazon S3 $0.14 per GB-month of data stored $0.01 per 1,000 PUT requests
(when saving a snapshot) $0.01 per 10,000 GET requests
(when loading a snapshot)
©2011 Amazon Web Services May not be reused or redistributed without permission
EBS Volume Lifecycle
Create
Vast amounts of unused space
Call CreateVolume1 GB to 1 TB
Attach
Deleted
Call AttachVolume to affiliate with one EC2 instance
Attached &
In Use
• Format from EC2 instance OS
• Mount formatted drive
CreateSnapshot
Snapshot to Amazon S3
Detach
Call DetachVolume
Call DeleteVolume
©2011 Amazon Web Services May not be reused or redistributed without permission
How Do Snapshots Work?
Time
Snapshot 1
Snapshot 2 Snapshot 3
EBS Volume
S3Block 1
Block 2
Block 3
Block 4
Chunk 1
Chunk 2
Chunk 3
Chunk 4
X©2011 Amazon Web Services May not be reused or redistributed without permission
Things You Should Know… (Part I)
EBS is a network resource Networks have variable performance
Striping will improve performance, although it’s usually fine to start with 4-8 drives per logical volume is observed sweet spot Great approach for high-performance database workloads
EBS is already redundant behind the scenes. It makes no sense to implement RAID on top of EBS
©2011 Amazon Web Services May not be reused or redistributed without permission
Things You Should Know… (Part II)
Lazy loads from snapshots When you create a volume, space is allocated immediately Data from snapshot written to disk on 1st request for maximum
performance For maximum IO performance, run dd to touch all blocks
Like all of AWS EBS is constantly innovating.These tips may not be appropriate in a few months
©2011 Amazon Web Services May not be reused or redistributed without permission
Get your data into AWS faster - load it onto a portable storage device and ship it to an Amazon data center
Faster than Internet transfer and more cost effective than upgrading your connectivity
Use cases: data migration, offsite backup, direct data interchange, disaster recovery
AWS Import/Export
Easy-to-use content delivery networkSame pay-as-you-go pricing as all of AWS– No up-front contracts– No long-term commitments– Self-service sign-up
Uses Amazon S3 as the origin storeWorldwide network of edge locations– Seattle, Palo Alto, Los Angeles, St. Louis, Dallas,
Ashburn, Newark, Miami, Amsterdam, Dublin, Frankfurt, London, Hong Kong, Tokyo
Amazon CloudFront