Windows Azure Tables: Programming Cloud Table Storage

Niranjan NilakantanDevelopment LeadMicrosoft Corporation

Pablo CastroSoftware ArchitectMicrosoft Corporation

ES07

Windows Azure is the foundation of Microsoft’s Cloud Platform

It is an “Operating System in the Cloud” and provides Essential Services for the cloud Virtualized Computation Scalable Storage Automatic Management Developer SDK

Windows Azure

Fundamental Data Abstractions

Blobs – Provide a simple interface for storing named files along with metadata for the file

Tables – Provide structured storage; A Table is a set of entities, which contain a set of properties

Queues – Provide reliable storage and delivery of messages for an application

Windows Azure Tables

Provides Structured Storage Massively Scalable Tables

Billions of entities (rows) and TBs of data Can use thousands of servers as traffic grows

Highly Available Can always access your data

Durable Data is replicated several times

Familiar and Easy to use API ADO.NET Data Services – .NET 3.5 SP1

.NET classes and LINQ REST – with any platform or language

Agenda

Data model Demo – creating tables Basic CRUD using .NET and REST Demo – a sample application Advanced features

Data Model

Table A Storage Account can create many tables Table name is scoped by Account

Data is stored in Tables A Table is a set of Entities (rows) An Entity is a set of Properties (columns)

Entity Two “key” properties that together

are the unique ID of the entity PartitionKey – enables scalability RowKey – uniquely identifies

the entity within the partition

Table Partition – all entities in table with same partition key value

Application controls granularity of partition

Partitioning

Partition KeyDocument

Row KeyVersion

Property 3ChangedOn

Property 4Description

Examples Doc V1.0 8/2/2007 Committed version

Examples Doc V2.0.1 9/28/2007 Alice’s working version

FAQ Doc V1.0 5/2/2007 Committed version

FAQ Doc V1.0.1 7/6/2007 Alice’s working version

FAQ Doc V1.0.2 8/1/2007 Sally’s working version

Partition 1

Partition 2

Partitioning Guidelines

Performance Use a PartitionKey that is

common in your queries Entities with same partition key value are clustered

Scalability We monitor partition traffic Automatically load balance partitions

Each partition can potentially be served by a different storage node

Scale to meet the traffic needs of your application More partitions – makes it easier

to balance load

Partition KeyDocument

Row KeyVersion

Property 3ChangedOn

Property 4Description

Examples Doc V1.0 8/2/2007 Committed version

Examples Doc V2.0.1 9/28/2007 Alice’s working version

FAQ Doc V1.0 5/2/2007 Committed version

FAQ Doc V1.0.1 7/6/2007 Alice’s working version

FAQ Doc V1.0.2 8/1/2007 Sally’s working version

Partition 1

Partition 2

Getting all versions of FAQ is fastAccess single partition

Get all documents before 5/30/2008 takes longerHave to traverse all partitions

Entity Locality

Entities and Properties

Each Entity can have up to 255 properties Every Entity has fixed key properties

Partition key Row key

No fixed schema for rest of properties 2 entities in the same table

can have different properties Properties stored as <Name, TypedValue> pairs

Each entity has a system maintained version

Property Types

Partition key and Row key String (up to 64KB)

Other properties String (up to 64KB) Binary (up to 64KB) Bool DateTime GUID Int Int64 Double

Sample Scenario

Micro-blogging application Post messages

to channels Subscribe to channels Retrieve channel

messages

Scalability challenges Large volume of

new messages daily Need to retrieve recent

messages quickly Messages permanently

addressable

Name – PKEmail

EmailUpdatesChannels

Users

Channel – PKPostedDate – RK

TextRating

Messages

Name – PKAuthor

CreatedDate

Channels

MicroBlogging Application

demo

Define the schema as a .NET class

Table Schema

[DataServiceKey("PartitionKey", "RowKey")]public class Message{ // ChannelName public string PartitionKey { get; set; } // PostedDate public string RowKey { get; set; }

// User defined properties public string Text { get; set; } public int Rating { get; set; }}

Insert Entity using .NET

Create a new Message and Insert into TableMessage message = new Message { PartitionKey = "Channel9", // ChannelName RowKey = DateTime.UtcNow.ToString(), // PostedDate Text = "Hello PDC", Rating = 3};

serviceUri = new Uri("http://<account>.table.core.windows.net");var context = new DataServiceContext(serviceUri);

context.AddObject("Messages", message);

DataServiceContext response = context.SaveChanges();

Insert Entity using REST

Create an Atom XML payload and POST itPOST http://<Account>.table.core.windows.net/Messages

... <!– Atom envelope --><m:properties> <d:PartitionKey>Channel9</d:PartitionKey> <!-- ChannelName --> <d:RowKey>Oct-29</d:RowKey> <!-- PostedDate -->

<d:Text>Hello PDC</d:Text> <d:Rating>3</d:Rating></m:properties>

Create Table

Every Account has a table called “Tables” To use a table it has to be inserted into “Tables”

[DataServiceKey("TableName")]public class TableStorageTable{ public string TableName { get; set; }}

TableStorageTable table = new TableStorageTable("Messages"); context.AddObject("Tables", table);

DataServiceResponse response = context.SaveChanges();

// Service Uri is “http://<Account>.table.core.windows.net/”DataServiceContext context = new DataServiceContext(serviceUri);

Query Entities

.NET – LINQserviceUri = new Uri("http://<account>.table.core.windows.net");DataServiceContext context = new DataServiceContext(serviceUri);

var messages = from message in context.CreateQuery<Message>("Messages") where message.Rating == 3 select message;

foreach (Message message in messages) { }

GET http://<serviceUri>/Messages?$filter= Rating eq 3

REST

Update a property in an entity

Using the .NET APIMessage message = (from message in context.CreateQuery<Message>("Messages") where message.PartitionKey == "Channel9"   && message.RowKey == "Oct-29" select message).FirstOrDefault();message.Text = "Hi there";

context.UpdateObject(message);

DataServiceResponse response = context.SaveChanges();

PUT http://<serviceUri>/Messages(‘Channel9’, ‘Oct-29’)<m:properties> <d:Text>Hi there</d:Text> <!-- Other properties are the same --></m:properties>

Using the REST API

Delete an Entity

Using the .NET API// Get the Message object for ("Channel9", "Oct-29")

context.DeleteObject(message);

DataServiceResponse response = context.SaveChanges();

DELETE http://.../Messages(‘Channel9’, ‘Oct-29’)

Using the REST API

.NET vs. REST

ADO.NET Data Services

Client included in .NET Framework 3.5 SP1

Data represented as .NET objects

DataServiceContext methods for updates

Use LINQ to define queries

REST Interface

Use any HTTP stack

Data represented in Atom (XML)

Standard HTTP verbs for updates

Use URLs to define queries

MicroBlogging Application

demo

Windows Platform Integration

Reuse .NET skills Fully compatible with

ADO.NET data services .NET client included in .NET 3.5 SP1 LINQ support

ASP.NET integration for website authoring Sample data source control for data binding ASP.NET dynamic data for instant front-ends ASP.NET providers for membership, roles, etc.

Advanced Topics

Concurrent updates Pagination Table consistency

Client A ClientB

5 : Ch9, Jan-1, 3

1 : Ch9, Jan-2, 21 : Ch9, Jan-2, 21 : Ch9, Jan-2, 22: Ch9, Jan-2, 5

Use standard HTTP mechanisms – Etag and If-MatchGet entity – get system maintained version as ETagUpdate Entities Locally – change ratingSend Update with version check - IF-Match with EtagSuccess if version matches, and update version on Client-APrecondition failed (412) if version does not match

Concurrent Updates

9 : Ch9, Jan-3, 6

If-Match: 1 Ch9, Jan-2, 5If-Match: 1 Ch9, Jan-2, 4

Version Rating 1: Ch9, Jan-2, 41: Ch9, Jan-2, 5Error: 412

2: Ch9, Jan-2, 5

.NET: LINQ Take(N) function

Getting the Top N entities

serviceUri = new Uri("http://<account>.table.core.windows.net");DataServiceContext context = new DataServiceContext(serviceUri);

var allMessages = context.CreateQuery<Message>("Messages");foreach (Message message in allMessages.Take(100)){ Console.WriteLine(message.Name);}

GET http://<serviceUri>/Messages?$top=100

REST: $top=N query string option

GET http://<Uri>/Messages?$filter=...&$top=100

&NextPartitionKey=Channel9 &NextRowKey=Date101

Pagination – Continuation Tokens

Send a requestGET http://<serviceUri>/Messages?$filter=...&$top=100

Set HTTP query parameters

Get continuation token in response headerx-ms-continuation-NextPartitionKey: Channel9x-ms-continuation-NextRowKey: Date101

Messages

Ch9, Date1,

Ch9, Date2,

Ch9, …

Ch9,Date100,

Ch9,Date101,

Ch9, …

100

Single Table Consistency

ACID transactions for single entity CUD Insert/update/delete

Snapshot isolation for query within a single partition Consistent view from start time of the query No dirty (uncommitted) reads Does not block concurrent updates

No snapshot isolation across partitions No snapshot isolation across

different continuations of a query

Cross Table Consistency

Application is responsible for maintaining consistency Example

When a channel is deleted, delete all the messages for that channel

Failures can occur in the middle Example - Application fails

after deleting some messages Use Windows Azure Queues to help

ensure completion of operation

1. Dequeue DelCh12. Delete Ch1 from Channels3. Delete from Messages4. Delete queue entry

Messages

Channels

QueueWorker

Cross Table Consistency

Del

Ch1

2

Delete channel Delete messages workerFront end

Front End

Del

Ch5

Del

Ch

11Ch1, Msg1

Ch1, Msg2

Ch1, Msg3

Ch2, Msg1

Ch2, Msg2

Ch3, Msg1Ch1,…

Ch2,…

Del

Ch1

1. Dequeue DelCh1 and start delete2. Fails after deleting Ch1 and Msg13. DelCh1 is visible again4. Dequeue DelCh1 again5. Repeat delete operations

Messages

Channels

QueueWorker 1

Resuming After Failure

Del

Ch1

2

Delete channel Delete messages workerFront end

Front End

Del

Ch5

Del

Ch

11Ch1, Msg1

Ch1, Msg2

Ch1, Msg3

Ch2, Msg1

Ch2, Msg2

Ch3, Msg1Ch1,…

Ch2,…

Worker2

Del

Ch1

Del

Ch

1h

Future Windows Azure Table Support

At PDC Single Index

Query and retrieve results sorted by PartitionKey and RowKey

Future Support for Secondary Indexes

Query and retrieve results sorted by other properties

Single Entity Transactions Single Entity Insert,

Update, or Delete Atomically

Entity Groups Atomic transactions

across multiple entities within same partition

Summary

Azure tables are Massively Scalable Highly Available

Simple familiar API Use .NET –- ADO.NET Data Services and LINQ Or use REST Leverage your .NET expertise

ASP .NET integration for instant front ends

Resources

Links Windows Azure

http://www.azure.com/windows

ADO.NET Data Services http://blogs.msdn.com/astoriateam

Evals & Recordings

Please fill

out your

evaluation for

this session at:

This session will be available as a recording at:

www.microsoftpdc.com

Please use the microphones provided

Q&A

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market

conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.