Windows Azure StorageCloud Computing Soup to Nuts
Mike BenkovichMicrosoft Corporationwww.benkoTips.com - @mbenko
btlod-72
Agenda
Storage OverviewGetting startedWorking with BlobsAdding Tables and
QueuesWorker Roles
Core Services
Caching CDN
Service Bus Reporting
Data Sync Azure Connect
Identity HPC
Additional Services
Windows Azure
Compute Storage Database
Windows Azure StorageStorage in the CloudScalable, durable, and availableAnywhere at anytime accessOnly pay for what the service usesInstant Concurrency
Exposed via RESTful Web ServicesUse from Windows Azure ComputeUse from anywhere on the internet
Wrapped in API by SDKMicrosoft.WindowsAzure.StorageClient
Windows Azure Storage Abstractions
TablesStructured storage. A table is a set of entities; an entity is a set of properties.
QueuesReliable storage and delivery of messages for an application.
BlobsSimple named files along with metadata for the file.
DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.
Windows Azure Storage Account
Create account in Management Portalhttp://windows.azure.com
Geographically located with affinityYou get 2 keys…API based on REST… http://btlod.blob.core.windows.net/public/helloworld.txt
http://<account>.<BLOB|TABLE|QUEUE>.core.windows.net/...
Tools to Explore Storage Many tools are available some for free, some from 3rd parties, including:
Visual Studiohttp://myAzureStorage.cloudapp.nethttp://clumsyleaf.comhttp://cerebrata.com
These give access to our storage account
Storage Accounts
Demo 1
Windows Azure Storage AccountCan CDN Enable AccountBlobs delivered via 24 global CDN nodes
Can co-locate storage account with compute accountExplicitly or using affinity groups
Accounts have two independent 512 bit shared secret keys
100 TBs per account
Storage in the Development FabricDeveopment Storage Emulator provides a local “Mock” storageEmulates storage in cloudAllows offline developmentRequires SQL Express 2005/2008 or aboveSSMS Provides local view of storage items
http://msdn.microsoft.com/en-us/gg433135
Scalability TargetsStorage Account• SLA – 99.9% Availability• Capacity – Up to 100 TBs• Transactions – Up to 5000 requests per second• Bandwidth – Up to a few hundred megabytes per second
Single Queue/Table Partition• Up to 500 transactions (entities or messages) per second
Single Blob Partition• Throughput up to 60 MB/s
Scale Above the limits• Partition between multiple storage accounts and partitions• When limit is hit, app may see ‘503 server busy’: applications
should implement exponential back-off
Working with BlobsBinary Large Objects include resources we use in our applications like pictures, videos, html, css, etc. that may not fit best in a relational schema
Choice of API’s – REST vs. SDK wrapper To use SDK add references to Microsoft.WindowsAzure.StorageClient Create Storage Account Object Add client for blobs Work with container references and blobs
Our example…WPF Cloud Explorer Basic WPF Application Enabled drop events – iterate thru collection of
files Create a blob container and upload blobs
Cloud Explorer
Uploading Blobs
Demo 2
Table Storage ConceptsEntityTableAccount
contoso
Name =…Email = …
Name =…EMailAdd=
customers
Photo ID =…Date =…
photos
Photo ID =…Date =…
Table Details
InsertUpdate Merge – Partial updateReplace – Update entire entityUpsertDeleteQueryEntity Group TransactionsMultiple CUD Operations in a single atomic transaction
Create, Query, DeleteTables can have metadata
Not an RDBMS! Table
Entities
Entity PropertiesEntity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort order
Timestamp Optimistic ConcurrencyExposed as an HTTP Etag
No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double
No Fixed Schema
FIRST LAST BIRTHDATE
Wade Wegner 2/2/1981
Nathan Totten 3/15/1965
Nick Harris May 1, 1976
FAV SPORT
Canoeing
Querying
FIRST LAST BIRTHDATE
Wade Wegner 2/2/1981
Nathan Totten 3/15/1965
Nick Harris May 1, 1976
?$filter=Last eq ‘Wegner’
Purpose of the Partition KeyEntity LocalityEntities in the same partition will be stored togetherEfficient querying and cache localityEndeavour to include partition key in all queries
Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction
Table ScalabilityTarget throughput – 500 tps/partition, several thousand tps/accountWindows Azure monitors the usage patterns of partitionsAutomatically load balance partitionsEach partition can be served by a different storage nodeScale to meet the traffic needs of your table
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE) TIMESTAMP MODELYEAR
Bikes Super Duper Cycle … 2009
Bikes Quick Cycle 200 Deluxe … 2007
… … … …Canoes Whitewater … 2009Canoes Flatwater … 2006
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE) TIMESTAMP MODELYEAR
Rafts 14ft Super Tourer … 1999
… … … …
Skis Fabrikam Back Trackers … 2009
… … … …Tents Super Palace … 2008
PARTITIONKEY(CATEGORY)
ROWKEY(TITLE) TIMESTAMP MODELYEAR
Bikes Super Duper Cycle … 2009
Bikes Quick Cycle 200 Deluxe … 2007
… … … …Canoes Whitewater … 2009Canoes Flatwater … 2006
Rafts 14ft Super Tourer … 1999
… … … …
Skis Fabrikam Back Trackers … 2009
… … … …Tents Super Palace … 2008
Partitions and Partition Ranges
Server A
Table = ProductsServer
BTable = Products[Canoes - MaxKey)
Server A
Table = Products[MinKey - Canoes)
Table PropertiesEntity can have up to 255 propertiesUp to 1MB per entity
Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)
Uniquely identifies an entityDefines the sort order
Timestamp Optimistic ConcurrencyExposed as an HTTP ETag
No fixed schema for other propertiesEach property is stored as a <name, typed value> pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double
Loosely Coupled Workflow with QueuesEnables workflow between rolesLoad work in a queueProducer can forget about message once it is in queueMany workers consume the queueFor extreme throughput (>500 tps) Use multiple queuesRead messages in batchesMultiple work items per message
Queue
Input Queue (Work Items)Web Role
Web Role
Web Role
Worker Role
Worker Role
Worker Role
Worker Role
Queue DetailsSimple asynchronous dispatch queue No limit to queue length subject to storage limit 64kb per message ListQueues - List queues in accountQueue operations CreateQueue DeleteQueue Get/Set Metadata Clear MessagesMessage operations PutMessage– Reads message and hides for time period GetMessages – Reads one or more messages and hides them PeekMessages – Reads one or more messages w/o hiding them DeleteMessage – Permanently deletes messages from queue UpdateMessage – Clients renew the lease and contents
Queue’s Reliable Delivery
Guarantee delivery/processing of messages (two-step consumption)
Worker dequeues message and it is marked as Invisible for a specified “Invisibility Time”
Worker deletes message when finished processing
If Worker role crashes, message becomes visible for another Worker to process
Message lifecycle
Queue
Msg 1
Msg 2
Msg 3
Msg 4
Worker Role
Worker Role
PutMessage
Web Role
GetMessage (Timeout)
RemoveMessage
Msg 2Msg 1
Worker Role
Msg 2
Creating ThumbnailsWPF Uploader Blob uploads are good Can leverage in web applications Nice to have thumbnails created
Next steps…catalog of uploads Create model for table storage Add class library to be shared between
projects Define queue for work to generate thumbs Add queue message for image files
Catalog
Adding table and queue messages…
Demo 3
Worker ProcessIdeally suited for Worker Role Read queue message with work Process thumbnail Update catalog table Sleep for predetermined time
Worker Role
Process thumbnails
Demo 4
Windows Azure Storage SummaryFundamental data abstractions to build your applicationsBlobs: Files and large objectsDrives: NTFS APIs for migrating applicationsTables: Massively scalable structured storageQueues: Reliable delivery of messages
Easy to use via the Storage Client LibraryHands on Labs
Where can I get more info?• Visit my site http://www.benkotips.com
• Resources from today’s talk• Webcasts• Downloads
• Check out the rest of this series!• http://bit.ly/s2nCloud
• Ask questions on Windows Azure Office Hours• http://bit.ly/bqtWazOH
• Get the Tools & the Trial • http://aka.ms/AzureMB• http://aka.ms/AzureTrialMB
© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after
the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.