Windows Azure Storage Cloud Computing Soup to Nuts

  • View

  • Download

Embed Size (px)


Windows Azure Storage Cloud Computing Soup to Nuts. Mike Benkovich Microsoft Corporation - @ mbenko. btlod-72. Agenda. Storage Overview Getting started Working with Blobs Adding Tables and Queues Worker Roles. Windows Azure. Core Services. Additional Services. - PowerPoint PPT Presentation


Windows Azure StorageCloud Computing Soup to NutsMike BenkovichMicrosoft - @mbenkobtlod-72

AgendaStorage OverviewGetting startedWorking with BlobsAdding Tables and QueuesWorker RolesCore ServicesCachingCDNService BusReportingData SyncAzure ConnectIdentityHPCAdditional ServicesWindows AzureComputeStorageDatabase3Windows Azure StorageStorage in the CloudScalable, durable, and availableAnywhere at anytime accessOnly pay for what the service usesInstant Concurrency

Exposed via RESTful Web ServicesUse from Windows Azure ComputeUse from anywhere on the internet

Wrapped in API by SDKMicrosoft.WindowsAzure.StorageClient

4Windows Azure Storage AbstractionsTablesStructured storage. A table is a set of entities; an entity is a set of properties.QueuesReliable storage and delivery of messages for an application.BlobsSimple named files along with metadata for the file. DrivesDurable NTFS volumes for Windows Azure applications to use. Based on Blobs.5Windows Azure Storage AccountCreate account in Management Portal located with affinityYou get 2 keysAPI based on REST to Explore Storage Many tools are available some for free, some from 3rd parties, including:Visual Studiohttp://myAzureStorage.cloudapp.nethttp://clumsyleaf.com

These give access to our storage accountStorage AccountsDemo 1Windows Azure Storage Account

Can CDN Enable AccountBlobs delivered via 24 global CDN nodes

Can co-locate storage account with compute accountExplicitly or using affinity groups

Accounts have two independent 512 bit shared secret keys

100 TBs per account9Storage in the Development FabricDeveopment Storage Emulator provides a local Mock storageEmulates storage in cloudAllows offline developmentRequires SQL Express 2005/2008 or aboveSSMS Provides local view of storage items

There are some differences between Cloud and Dev Storage:

A good approach for developers:

To test pre-deployment, push storage to the cloud firstUse Dev Fabric for compute connect to cloud hosted storageFinally, move compute to the cloud10Scalability TargetsStorage AccountSLA 99.9% AvailabilityCapacity Up to 100 TBsTransactions Up to 5000 requests per secondBandwidth Up to a few hundred megabytes per secondSingle Queue/Table PartitionUp to 500 transactions (entities or messages) per secondSingle Blob PartitionThroughput up to 60 MB/sScale Above the limitsPartition between multiple storage accounts and partitionsWhen limit is hit, app may see 503 server busy: applications should implement exponential back-off

Working with BlobsBinary Large Objects include resources we use in our applications like pictures, videos, html, css, etc. that may not fit best in a relational schema

Choice of APIs REST vs. SDK wrapperTo use SDK add references to Microsoft.WindowsAzure.StorageClientCreate Storage Account ObjectAdd client for blobsWork with container references and blobs

Our exampleWPF Cloud ExplorerBasic WPF ApplicationEnabled drop events iterate thru collection of filesCreate a blob container and upload blobs

Cloud ExplorerUploading BlobsDemo 2Table Storage Concepts

EntityTableAccountcontosoName =Email = Name =EMailAdd= customersPhoto ID =Date =photosPhoto ID =Date =15Table Details

InsertUpdate Merge Partial updateReplace Update entire entityUpsertDeleteQueryEntity Group TransactionsMultiple CUD Operations in a single atomic transactionCreate, Query, DeleteTables can have metadataNot an RDBMS! TableEntities16Entity PropertiesEntity can have up to 255 propertiesUp to 1MB per entity

Mandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic ConcurrencyExposed as an HTTP Etag

No fixed schema for other propertiesEach property is stored as a pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double17No Fixed SchemaFIRSTLASTBIRTHDATEWadeWegner2/2/1981NathanTotten3/15/1965NickHarrisMay 1, 1976FAV SPORTCanoeing18QueryingFIRSTLASTBIRTHDATEWadeWegner2/2/1981NathanTotten3/15/1965NickHarrisMay 1, 1976?$filter=Last eq Wegner19Purpose of the Partition KeyEntity LocalityEntities in the same partition will be stored togetherEfficient querying and cache localityEndeavour to include partition key in all queries

Entity Group TransactionsAtomic multiple Insert/Update/Delete in same partition in a single transaction

Table ScalabilityTarget throughput 500 tps/partition, several thousand tps/accountWindows Azure monitors the usage patterns of partitionsAutomatically load balance partitionsEach partition can be served by a different storage nodeScale to meet the traffic needs of your table20PartitionKey(Category)RowKey(Title)TimestampMODELYEARBikesSuper Duper Cycle2009BikesQuick Cycle 200 Deluxe2007CanoesWhitewater2009CanoesFlatwater2006PartitionKey(Category)RowKey(Title)TimestampMODELYEARRafts14ft Super Tourer1999SkisFabrikam Back Trackers2009TentsSuper Palace2008PartitionKey(Category)RowKey(Title)TimestampMODELYEARBikesSuper Duper Cycle2009BikesQuick Cycle 200 Deluxe2007CanoesWhitewater2009CanoesFlatwater2006Rafts14ft Super Tourer1999SkisFabrikam Back Trackers2009TentsSuper Palace2008Partitions and Partition RangesServer ATable = ProductsServer BTable = Products[Canoes - MaxKey)Server ATable = Products[MinKey - Canoes)21Table PropertiesEntity can have up to 255 propertiesUp to 1MB per entityMandatory Properties for every entityPartitionKey & RowKey (only indexed properties)Uniquely identifies an entityDefines the sort orderTimestamp Optimistic ConcurrencyExposed as an HTTP ETagNo fixed schema for other propertiesEach property is stored as a pairNo schema stored for a tableProperties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double22Loosely Coupled Workflow with QueuesEnables workflow between rolesLoad work in a queueProducer can forget about message once it is in queueMany workers consume the queueFor extreme throughput (>500 tps) Use multiple queuesRead messages in batchesMultiple work items per messageQueueInput Queue (Work Items)Web RoleWeb RoleWeb RoleWorker RoleWorker RoleWorker RoleWorker Role23Queue DetailsSimple asynchronous dispatch queueNo limit to queue length subject to storage limit64kb per messageListQueues - List queues in accountQueue operations CreateQueue DeleteQueueGet/Set MetadataClear MessagesMessage operationsPutMessage Reads message and hides for time periodGetMessages Reads one or more messages and hides themPeekMessages Reads one or more messages w/o hiding themDeleteMessage Permanently deletes messages from queueUpdateMessage Clients renew the lease and contents24Queues Reliable Delivery

Guarantee delivery/processing of messages (two-step consumption)

Worker dequeues message and it is marked as Invisible for a specified Invisibility TimeWorker deletes message when finished processingIf Worker role crashes, message becomes visible for another Worker to process

25Message lifecycleQueueMsg 1Msg 2Msg 3Msg 4Worker Role

Worker Role

PutMessageWeb Role

GetMessage (Timeout)RemoveMessageMsg 2Msg 1Worker Role

Msg 2

Creating ThumbnailsWPF UploaderBlob uploads are goodCan leverage in web applicationsNice to have thumbnails created

Next stepscatalog of uploadsCreate model for table storageAdd class library to be shared between projectsDefine queue for work to generate thumbsAdd queue message for image filesCatalogAdding table and queue messagesDemo 3Worker ProcessIdeally suited for Worker RoleRead queue message with workProcess thumbnailUpdate catalog tableSleep for predetermined timeWorker RoleProcess thumbnailsDemo 4Windows Azure Storage SummaryFundamental data abstractions to build your applicationsBlobs: Files and large objectsDrives: NTFS APIs for migrating applicationsTables: Massively scalable structured storageQueues: Reliable delivery of messages

Easy to use via the Storage Client LibraryHands on Labs31Where can I get more info?Visit my site Resources from todays talkWebcastsDownloadsCheck out the rest of this series! Ask questions on Windows Azure Office Hours Get the Tools & the Trial 32 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.