Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Art by Andrew Fryer
Dan Cohen Microsoft Consulting Services Microsoft Israel email: [email protected] Team blog: http://blogs.microsoft.co.il/blogs/mcs My blog: http://dannycohen.info
Agenda
Developer tools
Windows Azure
Compute
Storage
SQL Azure
Windows Azure platform AppFabric
Windows Azure Interoperability
Monitoring and Debugging
Microsoft Codename “Dallas”
Operating System
App/Web Server
Frameworks
Your Application
OS Services
Virtualized Instance
Hardware
Windows Azure Platform
.NET based
Application
Platform as a
Service
(PaaS) Windows Azure OS
IIS / WAS
.NET Framework
Lots of stuff…
Custom Hyper-V
Hardware
hardware
software
facilities
power/cooling
IT labor
support
network
security
maintenance
management tools
disaster recovery
backup
Acquisition cost is 10% of IT Spend
Operating cost is 90% of IT Spend
Source: IDC
Windows Azure Platform
Windows Azure Compute
Demo: Create an account / project
Windows Azure, On One Slide
The “cloud” is
1000s of geo-distributed servers & networking
Windows Azure is analogous to
An operating system for the cloud
The “operating system by analogy” provides
Application hosting
Automated service management
Upload your code and “service model”; press START
Durable storage at massive scale
Windows Azure, In One Picture
Desktop
VS Tools
WA SDK
Sto
rage
Clu
ste
r
MSFT Datacenters
Business Portal
Developer Portal
Com
pu
te C
luste
r
… …
Service Management Service
REST
REST …
Cloud VM
Runtime
API
VM Setup
User
Code
Azure Service Architectures
A service consists of
An isolation boundary
A set of component roles each with endpoints
Numbered, identical instances of each role
Azure Service Architectures A role instance is a set of:
Code
Configuration
local data
deployed in a dedicated VM
Web Role: The role is hosted on IIS
Worker Role: The role is an executable create your own web server
host a database
host Java VM, Ruby etc.)
Worker
Role
Worker
Role
Web
Role
Azure Service Architectures
A service is reachable externally via a single VIP for load-balanced traffic to multiple endpoints
Endpoints are reachable internally over TCP, HTTP Runtime API exposes & updates IP/Port values
LB
Windows Azure Compute
Demo: “Hello Cloud” with Visual Studio 2008/ 2010
Sample: Notification Of New Instances
Sample: Inter-role communication
2 VM Roles (currently) available: Web role
External synchronous HTTP/S communication
Worker role External HTTP/S / TCP communication
Mental model: “it‟s like a Windows Service”
Windows Azure Operating System Automatic /manual upgrades
See: http://msdn.microsoft.com/en-us/library/ee924680.aspx
Storage Local storage is volatile (!)
4 standard sizes Small -> Extra Large
See: http://msdn.microsoft.com/en-us/library/ee814754.aspx
Blobs
Drives
Tables Provide structured storage. A Table is a set of entities, which contain a set of properties
Queues
Blob Container Account
cohowinery
images
PIC01.JPG
PIC02.JPG
videos VID1.AVI
http://<account>.blob.core.windows.net/<container>/<blobname>
PutBlob
GetBlob
DeleteBlob CopyBlob SnapshotBlob LeaseBlob
Cache-Control Content-Encoding Content-Type
CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudBlobClient blobClient = new CloudBlobClient( account.BlobEndpoint, account.Credentials); // Create Container CloudBlobContainer cloudContainer = blobClient.GetContainerReference(containerName); bool hasCreated = cloudContainer.CreateIfNotExist(); // Access Blob in the Container CloudBlob cloudBlob = cloudContainer.GetBlobReference(blobName); //BlobRequestOptions has retry policy, timeout etc. BlobRequestOptions options = new BlobRequestOptions(); //Upload the local file to Blob service cloudBlob.UploadFile(uploadFileName, options); //Download to local file name cloudBlob.DownloadToFile(downloadFileName, options);
blocks
pages
10 GB Movie
Windows
Azure Storage
Blo
ck
Id
1
Blo
ck
Id
2
Blo
ck Id
3
Blo
ck
Id
N
blobName = “blob.wmv”; PutBlock(blobName, blockId1, block1Bits); PutBlock(blobName, blockId2, block2Bits); ………… PutBlock(blobName, blockIdN, blockNBits); PutBlockList(blobName, blockId1, blockId2…,blockIdN);
blob.wmv blob.wmv
PutPage 512, 2048PutPage 0, 1024ClearPage 512, 1536PutPage 2048,2560
GetPageRange 0, 4096
0,512 1536,2560
GetBlob 1000, 2048
1536,2048
0
10 GB
10 G
B A
dd
ress S
pace
512
1024
1536
2048
2560
ServicePointManager.DefaultConnectionLimit
ParallelOperationThreadCount CloudBlobClient
BlobRequestOptions
Timeout
Blob Operations
Copy, Snapshot, and Lease work for both types
Ways of Accessing and Serving Blob Content
Content Delivery Network access
Shared Access Signatures (Signed URLs)
Custom Domain Names
Blobs
Drives
Tables Provide structured storage. A Table is a set of entities, which contain a set of properties
Queues
Windows Azure
Blob Service
DemoBlob
Local Cache
Application
Lease
Drive X:
//Create Local Storage resource and initialize the local cache for drives CloudDrive.InitializeCache(localCacheDir, cacheSizeInMB); CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); //Create a cloud drive (PageBlob) CloudDrive drive = account.CreateCloudDrive(pageBlobUri); drive.Create(1000 /* sizeInMB */); //Mount the network attached drive on the local file system string pathOnLocalFS = drive.Mount(cacheSizeInMB, DriveMountOptions.None); //Use NTFS APIs to Read/Write files to drive … //Snapshot drive while mounted to create backups Uri snapshotUri = drive.Snapshot(); //Unmount the drive drive.Unmount();
Blobs
Drives
Tables Provide structured storage. A Table is a set of entities, which contain a set of properties
Queues
Entity Table Account
cohowinery
customers
Name =…
Email = …
Name =…
Email = …
winephotos
Photo ID =…
Date =…
Photo ID =…
Date =…
PartitionKey
(Category)
RowKey
(Title)
Timestamp ReleaseDate
Action Fast & Furious … 2009
Action The Bourne Ultimatum … 2007
… … … …
Animation Open Season 2 … 2009
Animation The Ant Bully … 2006
PartitionKey
(Category)
RowKey
(Title)
Timestamp ReleaseDate
Comedy Office Space … 1999
… … … …
SciFi X-Men Origins: Wolverine … 2009
… … … …
War Defiance … 2008
PartitionKey
(Category)
RowKey
(Title)
Timestamp ReleaseDate
Action Fast & Furious … 2009
Action The Bourne Ultimatum … 2007
… … … …
Animation Open Season 2 … 2009
Animation The Ant Bully … 2006
Comedy Office Space … 1999
… … … …
SciFi X-Men Origins: Wolverine … 2009
… … … …
War Defiance … 2008
Server B Table = Movies
[Comedy - MaxKey)
Server A Table = Movies
[MinKey - Comedy)
Server A Table = Movies
[DataServiceKey("PartitionKey", "RowKey")] public class Movie { /// Movie Category is the partition key public string PartitionKey { get; set; } /// Movie Title is the row key public string RowKey { get; set; } public DateTime Timestamp { get; set; } public int ReleaseYear { get; set; } public double Rating { get; set; } public string Language { get; set; } public bool Favorite { get; set; } }
CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudTableClient tableClient = new CloudTableClient( account.TableEndpoint, account.Credentials); // Create Movie Table string tableName = “Movies“; tableClient.CreateTableIfNotExist(tableName);
TableServiceContext context = tableClient.GetDataServiceContext(); // Add movie context.AddObject(tableName, new Movie("Action", “White Water Rapids Survival")); context.SaveChangesWithRetries(); // Query movie var q = (from movie in context.CreateQuery<Movie>(tableName) where movie.PartitionKey == "Action" && movie.Rating > 4.0 select movie).AsTableServiceQuery<Movie>(); foreach (Movie movieToUpdate in q) { movieToUpdate.Favorite = true; context.UpdateObject(movieToUpdate); } context.SaveChangesWithRetries( ); SaveChangesOptions.Batch
Scale
Queries
Entity Group Transactions
Avoid “Append only” write patterns based on PartitionKey values
Avoid using monotonically increasing suffix with a constant prefix
Example: using only the current timestamp as PartitionKey
If needed, add varying prefix to PartitionKey
Restructure your schema if required
Concatenate different keys to form appropriate index
Most Optimal: PartitionKey == “SciFi” and RowKey == “Star Wars”
Scans: Expect continuation tokens PartitionKey == “SciFi” and “Sphere” ≤ RowKey ≤ “Star Wars”
“Action” ≤ PartitionKey ≤ “Thriller”
PartitionKey == “Action” || PartitionKey == “Thriller” - currently scans entire table
“Cars” ≤ RowKey ≤ “Star Wars” - scans entire table
ServicePointManager.DefaultConnectionLimit = X;
SaveChangesWithRetries
AsTableServiceQuery
Conflict NotFound
http://dannycohen.info/2010/04/21/azure-storage-asynchronous-delete
Blobs
Drives
Tables Provide structured storage. A Table is a set of entities, which contain a set of properties
Queues
Message Queue Account
order processing
customer ID order ID http://…
customer ID order ID http://…
cohowinery
Azure Queue
Input Queue
(Work Items)
Azure Queue
Input Queue
(Work Items)
CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudQueueClient queueClient = new CloudQueueClient( account.QueueEndpoint, account.Credentials); //Create Queue CloudQueue queue = queueClient.GetQueueReference(queueName); queue.CreateIfNotExist(); //Add Message CloudQueueMessage message = new CloudQueueMessage(“some content"); queue.AddMessage(message); //Get Message message = queue.GetMessage(TimeSpan.FromMinutes(3) /*Invisibility timeout*/); // Process Message within the Invisibility Timeout
//Delete Message queue.DeleteMessage(message);
DequeueCount
Windows Azure Storage Explorers (April 2010)
Windows Azure Storage Explorer Block Blob Page Blob Tables Queues Free
Azure Blob Client X Y
Azure Blob Compressor Enables compressing blobs for upload and download
X Y
Azure Blob Explorer X Y
Azure Storage Explorer X X X Y
Azure Storage Simple Viewer X X X Y
Cerebrata Cloud Storage Studio X X X X Y/N
Cloud Berry Explorer X X Y
Clumsy Leaf Azure Explorer
Visual studio plug-in
X X X X Y
Factonomy Azure Utility X Y
Gladinet Cloud Desktop X N
MyAzureStorage.com A portal to access blobs, tables and queues
X X X X Y
Space Block X Y
Windows Azure Management Tool X X X X Y
http://blogs.msdn.com/b/windowsazurestorage/archive/2010/04/17/windows-azure-storage-explorers.aspx
Azure MMC Snap-in
http://code.msdn.microsoft.com/windowsazuremmc
Cerebrata – Azure Diagnostics Manager http://www.cerebrata.com/Products/AzureDiagnosticsManager/Default.aspx
Business Analytics
Reporting
Windows Azure Compute
Demo: Create a SQL Azure database
SQL Azure Network Topology Application
Internet
Azure
Cloud
LB
TDS (tcp)
TDS (tcp)
TDS (tcp)
Applications use standard SQL
client libraries: ODBC,
ADO.Net, PHP, …
Load balancer forwards „sticky‟
sessions to TDS protocol tier
Security Boundary
SQL SQL SQL SQL SQL SQL
Gateway Gateway Gateway Gateway Gateway Gateway
Scalability and Availability: Fabric, Failover, Replication, and Load balancing
Gateway: TDS protocol gateway, enforces AUTHN/AUTHZ policy; proxy to backend
SQL Azure Network Topology Application
Internet
Azure
Cloud
LB
TDS (tcp)
TDS (tcp)
TDS (tcp)
Applications use standard SQL
client libraries: ODBC,
ADO.Net, PHP, …
Load balancer forwards „sticky‟
sessions to TDS protocol tier
Security Boundary
SQL SQL SQL SQL SQL SQL
Gateway Gateway Gateway Gateway Gateway Gateway
Scalability and Availability: Fabric, Failover, Replication, and Load balancing
Gateway: TDS protocol gateway, enforces AUTHN/AUTHZ policy; proxy to backend
Application Topologies SQL Azure access from within
MS Datacenter (Azure compute) SQL Azure Access from outside
MS Datacenter (On-premises)
SQL Azure Access from within and outside MS
Datacenter (On-premises & Azure Compute)
Application/
Browser
SOAP/REST
HTTP/S
Astoria/REST - EDM
HTTP/S
App Code (ASP.NET)
App Code (ASP.NET)
T-SQL (TDS)
SQL Azure Windows
Azure
Code Near
App code/ Tools
T-S
QL
(T
DS
)
SQL Azure
Windows
Azure
Code Far Hybrid
SQ
L A
zu
re
Da
ta S
yn
c
Windows
Azure
SQL Azure
App code/ Tools
App Code (ASP.NET)
App Code (ASP.NET)
T-SQL (TDS)
SQL Azure Cluster
Node 55
DB3
Node 14 Node 19 Node 21 Node 99 Node 2 Node 76
Shards: Managing Lots of Data
App
Server
Master
DB DB1 DB2
Node 33
DB3 DB4 DB5 DB6
SQL Azure Cluster
Node 14 Node 19 Node 21 Node 99 Node 2 Node 76
Copies: Managing Lots of Reads
App
Server
Master
DB
DB
C1
DB
C2
Node 33
DB
C3
DB
C4
DB
C5
DB
C6
LB
What are your application‟s requirements? Storage and Transactional throughput
Storage Requirements Low High
Tra
nsa
ction
al R
eq
uire
me
nts
L
ow
H
igh
• Single Database
• No Partitioning
• Partitioned Data
• Partitioning Based on
Application
Requirements (Storage)
• Partitioned Data
• Partitioning based on
Application Requirements
(IOPS)
• Partitioned Data
• Partitioning based on
Application Requirements
(IOPS, Storage or both)
SQL Azure Tips
Developing on a local SQL Express instance has some advantages
Easy to get started, you already know how to do it!
Full fidelity with the designer and debugging tools
Reduces latency when using local Azure development tools
Reduces bandwidth and databases costs for development
Some caveats
Remember to alter your VS build settings to switch the connection string when deploying
Use tools (like SQLAzureMW) to keep you within the supported SQL Azure features
Always test in SQL Azure before deploying to production
SQL Azure Migration Wizard
http://sqlazuremw.codeplex.com
SQL Azure Tips SQL Azure connection strings follow normal SQL syntax
Applications connect directly to a database “Initial Catalog = <db>” in connection string
No support for context switching (no USE <db>)
Some commands must be in their own batch Create/Alter/Drop Database & Create/Alter/Drop Login, & Create/Alter USER with FOR/FROM LOGIN
Encryption security Set Encrypt = True, only SSL connections are supported
TrustServerCertificate = False, avoid Man-In-The-Middle-Attack!
Format of username for authentication: ADO.Net: Data Source=server.database.windows.net; User ID=user@server;Password=password;...
Setup your firewall rules first!
SQL Azure Tips
SQL Connections: Retry on failure
Connections can drop for variety of reasons Idleness
Transient (network) errors
Intentional throttling
First step: reconnect immediately Handles idleness- and transient-disconnects
Gateway handles connection retry for app Connections attempted for ~30s before failure
What to do on connection failure? Wait (10 seconds), then retry
Change your workload if throttled
SQL Connection Retry Pattern
while (true)
{
using (SqlConnection connection = new SqlConnection(connStr))
{
try
{
connection.Open();
using (SqlCommand cmd = connection.CreateCommand())
{
cmd.CommandText = @"SetBCPJobStartTime";
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add(new SqlParameter(@"@BCPJobId", BCPJobId));
cmd.ExecuteNonQuery();
}
}
catch (Exception exc)
{
// deal with error
}
}
// more stuff
// …..
}
SQL Azure Labs Data Sync
SQL Azure Labs Data Sync
SQL Azure Labs Data Sync
Windows Azure Platform
Service Remoting
Control Web services through the Internet using Service Bus
Sender solicits information from listeners
Service Bus
Access Control
Eventing
Notify remote parties of events
Sender transmits information to listeners
Listeners may handle events in different ways
Service Bus
Access Control
Tunneling
Traverse network borders without opening firewalls
Use an HTTP channel to mimic needed protocol
Service Bus
Access Control
Azure AppFabric scenario #1
Scenario: How do we connect Org. A to Org. B ?
Employees of Org. B wish to be informed of events from org. A.
Organization A
Internal
Application
Organization B
Internal
Application
Azure AppFabric scenario
Scenario: How do we connect Org. A to Org. B ?
Option #1 – On-premise integration
BizTalk + SQL + Windows + firewall licenses ?
Hardware, electricity, hosting ?
Maintenance, personnel, support services ?
Organization A
Internal
Application
Organization B
Internal
Application
Fire
wall
Fire
wall
BizTalk
+
SQL
Server
+
Windows
Servers
Azure motivation scenario
Scenario: How do we connect Org. A to Org. B ?
Option #2 – Windows Azure AppFabric integration
Cost: 3.99$ per connection / month (volume discount)
Data transfers: $0.10 in / $0.15 out per GB
Organization A
Internal
Application
Organization B
Internal
Application
http://www.microsoft.com/windowsazure/pricing
Service Bus
Access
Control
Question:
Is BizTalk Server Dead ?
In a word: No.
In two words: Hell, no!
Business Partner
CRM
HR
E-Commerce
ERP
Business
Partner
Integration Server
(EAI/B2B)
Enterprise Service Bus
(ESB)
Windows Azure
Platform
Cloud-enabled SOA platform
AppFabric
Service Bus
AppFabric
Service Bus
Device
On-Premise
Cloud
Enabling hybrid applications with AppFabric
Request
processing
and
scheduling
Website
Front End
Real
Estate
agent
laptop/
device
CRM System
Device
On-Premise
Cloud
Enabling hybrid applications with AppFabric
Request
processing
and
scheduling
Website
Front End
Real
Estate
agent
laptop/
device CRM System
Cloud
On-Premise
Enabling hybrid applications with AppFabric
Request
processing
and
scheduling
Website
Front End
Service Bus
Access Control
Cloud
On-Premise
Enabling hybrid applications with AppFabric
Request
processing
and
scheduling
Website
Front End
Service Bus
Access Control
Cloud
On-Premise
Enabling hybrid applications with AppFabric
Request
processing
and
scheduling
Website
Front
End
Service Bus
Access
Control
Rules OUT claims
IN claims
Cloud
On-Premise
Enabling hybrid applications with AppFabric
Website
Front
End
Service Bus
Access
Control
Request
processing
and
scheduling
two-way
one-way
multicast
multiplex
pub/sub
buffered
Communication
Patterns
Demo: Simple Publish/Subscribe & Multicast
Enabling hybrid applications with AppFabric
http://idesign.net
On-Premise
Site B
On-Premise
Site A
Service Bus
Access Control
Demo: Distributed Cloud based app with Windows Azure & AppFabric
Enabling hybrid applications with AppFabric
Windows Azure
LB
Web
Role
Worker
Role Azure
Queue
Windows Azure Interoperability
http://www.microsoft.com/WindowsAzure/interop
Hosting non-MS applications Mental Model:
Windows OS
Non-admin user
USB drive with “runme.bat”
Can your app run that way?
Use Worker Role
Configure Endpoints as necessary
Call Process.Start(…)
Continue monitoring from the Worker Role
Windows Azure Storage 4 Java
Windows Azure Storage
Blobs, tables, queues, drives
Highly scalable
REST interface
Java storage library (windowsazure4j.org)
Built by Soyatec
Resources
Building Java Applications with Windows Azure http://microsoftpdc.com/Sessions/SVC50
Developing PHP & MySQL Applications with Windows Azure http://microsoftpdc.com/Sessions/SVC51
Ruby on Rails on Windows Azure: http://blogs.msdn.com/b/simondavies/archive/2009/11/25/running-ruby-on-rails-on-windows-azure.aspx
Sample App: http://rubyonrails.cloudapp.net/
Downloads: http://windowsazure.com/interoperability
Thank you!