SQL Services: Tips and Tricks for High-Throughput Data-Driven Applications

David RobinsonSenior Program ManagerMicrosoft Corporation

Extremely scalable High performing Manage large volumes of data Low CapEx/OpEx Silverlight 2.0

Our MissionAnd yes, we need to accept it…

Develop a news posting application that has the following characteristics

And…You have no hardware budget

We Can’t Let Our Fearless Leader Down

We Need Data modeling and partitioning Parallel query processing Concurrency Utilizing a middle tier Support for batch processing

AgendaHow will we accomplish our mission?

Azure Services Platform

Microsoft Dynamics CRM Services

Microsoft SharePoint Services

Data Model And ACE Concepts

Unit ofgeo-location and billing

Tied toDNS name

Collectionof Containers

Authority Container Entity

Unit of Consistency

Scope for Query and Update

Collectionof Entities

Unit of Storage Property Bag

of Name/Value pairs

No Schema Required

Data Modeling and PartitioningIt’s not much different than what you are used to Static System Data (low # of deltas)

LookupsStatic ContentTags

User Data (medium # of deltas)

Application Data (high # of deltas)Content PostingsWikisCommentsUser to User Interactions

User ProfilesUser Preferences

Data Modeling and Partitioning Entities and kinds

Our application requires more RDBMS features like JOINS, TAKE, and ORDER BY

Comment

RatingTags

Posting

BONUS!!!All fields are automatically indexed

Join in SQL

select p.*

from Posting p,PostingTag t,Tag m

where p.Id = t.PostingId andt.TagId = m.Id andm.Tag = "STRATA"

Join in SLINQ

from p in entities.OfKind(“Posting”)from t in entities.OfKind(“Tag”)from m in entities.OfKind(“PostingTag”)

where

p.Id == t.Properties["PostingId"] &&t.Properties["TagId"] == m.Id &&

m.Properties["Tag"] == “AZURE"

select p

Data Modeling and PartitioningHow does SQL Data Services store our data?

Containers are scoped to individual backend nodes

General Rule of ThumbSpread your data across

multiple containers

for best performance

Data Modeling and PartitioningHow does SQL Data Services store our data?

Application Data – Remaining Containers

Static System Data (1 Container)

User Data (36 Containers)

Containers are scoped to nodes in the SDS backend

Spread your data out amongst many containers for best performance

Modeling your data via Entities is not much different than modeling with a traditional RDBMS

Data Modeling and PartitioningTakeaways

Data modeling and partitioning Parallel Query Processing

AgendaHow will we accomplish our mission?

ü

Parallel Query Processing How does data access in SQL Data Services work?

All queries are processed

via the SDS Front Ends

General Rule of ThumbProcess your

queries in parallel for best

performance

Parallel Query Processing

demo

SQL Data Services operations are processed by the front end servers

For best performance multithread your calls to take advantage of the many front end servers

Parallel Query ProcessingTakeaways

Data modeling and partitioning Parallel query processing Concurrency

AgendaHow will we accomplish our mission?

ü ü

ConcurrencyHow do we handle multiple updates?

SQL Data Services provides immediate consistency

Lets consider when two users attempt to update the same record…

Posting 123ViewCount = 134

ViewCount++ ViewCount++

This is easily handled in SQL Data Services

Concurrent Operations in SDS

SOAPScope myEntityScope = new Scope();myEntityScope.EntityId = "AwesomePosting";myEntityScope.VersionMatch.Version = 134;myEntityScope.VersionMatch.MatchType =

VersionMatchType.Match;

RESTWebRequest request = HttpWebRequest.Create(entityUri);request.Method = "POST";request.Headers

[HttpRequestHeader.IfMatch] = "134";

SDS provides an immediate consistency model

Use appropriate versioning semantics to prevent unwanted data overwrites

MatchType property in SOAP IfMatch header in REST

Concurrency Key takeaways

Data modeling and partitioning Parallel query processing Concurrency Caching in our middle tier

AgendaHow will we accomplish our mission?

ü ü

ü

Utilizing a Middle Tier

Using a middle tier is nothing newbut wait…

Don’t forget you have no hardware

budget…

Windows Azure Service Hosting

demo

Nothing new to learn to host your services in Windows Azure

By using Windows Azure, your middle tier is located closer to your data

Caching helps combat the latency associated with cloud based data services

Utilizing a Middle TierKey takeaways

Data modeling and partitioning Parallel query processing Concurrency Caching in our middle tier Support for batch processing

AgendaHow will we accomplish our mission?

ü ü

ü ü

Support for Batch ProcessingHow do we queue items to be processed?

After someone creates certain types of postings, additional processing needs to be done before posting is made live.

This is easily handled with .NET Services

.NET Service Bus

demo

Large, complex applications can benefit from offline batch processing

.NET Service Bus is an ideal way of enabling queuing and offline processing

Support for Batch Processing Key takeaways

Data modeling and partitioning Parallel query processing Concurrency Caching in our middle tier Support for batch processing

AgendaHow will we accomplish our mission?

ü ü

ü ü

ü

Mission Accomplished !!!

With all these tips and tricks, what can we build?

Cloud 9

demo

Related SessionsCode Title Room Time

BB23 SQL Data Services : A Lap Around 502A 10/28/2008 3:30PM-4:45PM

BB03 SQL Data Services : Under the Hood 404A 10/30/2008 8:30AM-9:45AM

BB14 SQL Data Services: Futures 408B 10/29/2008 10:30AM-11:45AM

BB52 SQL Data Services: Tips and Tricks for High-Throughput Data-Driven Applications 411 10/28/2008 12:45PM-1:30PM

BB40 Sync Framework: Enterprise Data in the Cloud and on Devices 408A 10/28/2008 5:15PM-6:30PM

TL30 Microsoft Sync Framework Advances 515B 10/27/2008 1:45PM-3:00PM

BB16 SQL Server 2008: Beyond Relational 406A 10/28/2008 1:45PM-3:00PM

BB26 SQL Server 2008: Business Intelligence and Data Visualization 515A 10/28/2008 1:45PM-3:00PM

BB24 SQL Server 2008: Deep Dive into Spatial Data 404A 10/29/2008 3:00PM-4:15PM

BB07 SQL Server 2008: Developing Large Scale Web Applications and Services 411 10/28/2008 1:45PM-3:00PM

BB37 SQL Server 2008: Developing Secure Applications 515A 10/29/2008 12:00PM-12:45PM

BB25 SQL Server 2008: New and Future T-SQL Programmability 515A 10/29/2008 1:15PM-2:30PM

PC40 SQL Server Compact: Embedding in Desktop and Device Applications 402A 10/29/2008 3:00PM-4:15PM

TL42 Microsoft SQL Server 2008: Powering MSDN 411 10/29/2008 12:00PM-12:45PM

TL14 Project "Velocity": A First Look 403AB 10/28/2008 1:45PM-3:00PM

TL56 Project "Velocity": Under the Hood 403AB 10/28/2008 3:30PM-4:45PM

PC44 Windows 7: Programming Sync Providers That Work Great with Windows 408B 10/28/2008 12:45PM-1:30PM

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market

conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.