10 minutes on 10 tools and techniques

Becoming an advanced Windows Azure developer through power tools

Introduction

AgendaWhy tools are important?Sending an EmailFault Handling with TOPAZAutoscaling with WASABiWIF and Custom ClaimsGrid on Windows AzureREST Clients on Windows AzureUseful toolsDiagnostics PowershellSQL Migration and Data Sync

WhoRichard Conway and Andy Cross, co-founders UK Windows Azure User Group and founders of Windows Azure consultancy Elastacloud.

Condensed from Elastacloud’s advanced Windows Azure developer course

Advanced development requires advanced tooling

The power of tooling

Success

Productivity

Investment

50%20%

80%Tooling

An advanced developer understands a platform. They solve problems with tools where possible, focusing on the underlying business challenges. This talk will highlight advanced Windows Azure features and equip you with the power tools you need to meet your challenges head on.

Core Tooling Areas

Storage

Security

Compute

ConnectivityTooling aspects

Talk Format

• Each topic covers why each tool is required• Gives examples of its usage• Covers where you can learn more

A little about next steps

• Come to the UKWAUG http://www.ukwaug.net • If you’re new to Azure come to our Techdays-sponsored bootcamp this

Friday• These slides breeze through a lot of complicated subject matter we

may decide to put on a Techdays sponsored advanced day bootcamp – contact us on info@elastacloud.com if you’re interested in attending

• All code examples will be posted on our blog this weekend:• http://blog.elastacloud.com

Sending an EmailTopic 1

Faults and Services

Azure provides no inherent emailing capability or service

By default instances have open SMTP ports unless the firewall rules have been tightened

Cloud applications, by default are set on full trust but SMTP can be sent using partial trust

SMTP outgoing ports can change depending on the SMTP server/relay service being used

My application completes a setup or purchase process and needs to send an acknowledgment email to a user

My application completes a registration process and needs to send an email with a transient link to the new user

My application needs to sends a mass mailout to a group of people within a local database

My application needs to use an external database via an email service

Application Scenarios

Affected Services

Configuration and SmtpClientConfiguration

• Host address:• e.g. mail.azurecoder.com

• Port used:• 25, 587

• Credentials:• username, password

• Security and Integrity:• Using SSL

SmtpClient

• Traditional use of email sending within ASP.NET applications• Use of SmtpClient class to send messages from an email to an email • Use of NetworkCredentials to encapsulate usernames and passwords

Mass EmailingDetails

• The traditional former-IIS model was built using relay services• This queued messages in folders and attempted to relay to an smtp server• The use of this service proxy pattern is now a common web pattern but using external services• Scenarios include sending mass marketing mailouts from an internal database • In this instance Windows Azure can be used as a relay node to an external service from Dynamics or other applications

Using SendGridDetails

• Provision an account with 25K free emails/month @ http://sendgrid.com/azure.html• Follow account setup instructions• Use Nuget Package Manager to install SendGrid• Install-Package SendGrid• You can now use SendGrid via REST or SMTP!• SendGrid client components can use any REST/SMTP endpoint not just SendGrid

REST with SendGridDetails

• SendGrid supports a REST-based transport • A SendGrid object encapsulates a mail message• System.Net.NetworkCredential should contain the SendGrid username and password• A REST transport object should be created – this automatically defines the endpoint and the message can be sent

Mailchimp on AzureDetails

• Used when email campaigns need to be put together• A Mailchimp account with up to 12,000 emails/month is free to setup• The database containing names of recipients is within Mailchimp external to the application• By default Mailchimp uses a web-based API called MCAPI• Credentials, tokens and authorization is managed securely through API keys via the oAuth2 protocol• Codeplex hosts an MCAPI.NET project wrapper @ http://mcapinet.codeplex.com

MCAPI.NETDetails

• Download and reference MCAPI.NET• Create a Mailchimp account• Create a Mailchimp API key• Create a campaign using the Campaign Wizard in Mailchimp Administration• Import users • Create an MCApi object passing the API key• Call CampaignSendNow(CampaignId)

Configuring the DemoDetails

• Navigate to Emailer.cs• Relevant parts are interface: IEmailer and classes: EmailerSimple, EmailerCampaign, EmailerRest, EmailerSmtp• In EmailerSimple, EmailerSmtp and EmailerRest replace username and password in NetworkCredential with real values• Also replace valid ReplyTo, From and other details with real values• In EmailerCampaign replace myapikey with a valid API key from Mailchimp and a valid CampaignId

Emailing DemoMCAPI.NET, SendGrid and the SmtpClient

Further ReadingReferences• SendGrid is available at http://sendgrid.net – there is a free offer if you have an Azure application• Mailchimp is available free at http://www.mailchimp.com

Fault Handling with TOPAZTopic 2

Faults and Services

SQL Azure

Storage Services

Service Bus

Caching Service

Usage of cloud-based services can be disrupted due to:

Networking issues

Infrastructure faults

Intermittent conditions

Transient faults are smaller in duration than persistent faults

There is no intrinsic way to determine whether a fault is persistent or transient

The service developer may create a subset of faults for such transient faults

Transient Faults

Affected Services

Mitigation using TOPAZMitigation

• Retries, retries, retries!• Transient faults are recoverable fairly quickly• Design your application to work within the parameters of the services they use:• SQL Azure can drop connections and refuse new connections if too many client connections and resources are being used • Azure Storage can refuse access if too many transactions are currently taking place outside of the limits of the intended usage

What is TOPAZ

• TOPAZ: The Transient Fault Handling Application Block• Part of the Enterprise Application Blocks Library• Wraps up connection oriented APIs to the aforementioned Azure services • Detects transient failures in these services as opposed to persistent failures • Has the ability to define connection retry logic using the following parameters:• Fixed interval retries (10 secs, 10 secs, 10 secs) • Incremental interval retries (10 secs, 20 secs, ..)• Random exponential (or back-off) interval retries (5 secs, 12 secs, 20 sec, 30 secs ..)

Installing and Configuring TOPAZ

Can configure TOPAZ to use the following retry strategies:

Fixed

Incremental

Exponential backoff

Custom

Each has configuration parameters:

Maximum retries

Interval (fixed or incremental)

First fast retry

Delta Backoff

To add TOPAZ to VS.NET use the following:

Install-Package EnterpriseLibrary.WindowsAzure.TransientFaultHandling

In order to add configuration support use the Extensions Manager to add the Enterprise Library Configuration extension

Using the Enterprise Library Configuration Editor right-click a web.config or app.config and begin to configure TOPAZ

Installation Configuration

Developing with TOPAZCode

• Import two namespaces:• Microsoft.Practices.TransientFaultHandling• Microsoft.Practices.EnterpriseLibrary.WindowsAzure.TransientFaultHandling.SqlAzure …

• Define the connection type SB/SQL/Storage etc.• Define the RetryPolicy, subscribe to event Retrying and determine the type of policy (Incremental, FixedInterval, ExponentialBackoff)• Use the specific generic – In the example SqlAzureTransientErrorDetectionStrategy

Using Async Best PracticeCode

• Use ExecuteAction is an asynchronous callback with four Actions methods:• BeginAsync• EndAsync• Success• Failure

Custom TOPAZCode

• It’s possible to implement:• A custom transient error detection strategy• A custom retry strategy

• Implement ITransientErrorDetectionStrategy and the IsTransient(Exception) method to determine whether the connection-oriented service is transient• Inherit from RetryStrategy, pass in a name and “first fast retry” flag • Override GetShouldRetry to return a ShouldRetry method• To get design time support for the EnterpriseLibrary Configuration Editor decorate the RetryStrategy class with:• [ConfigurationElementType(typeof (CustomRetryStrategyData))]

TOPAZ DemoTransient Connectivity

Configuring the DemoDetails

• The example contains the following:• Using a ADO.NET SqlConnection to connect to SQL Azure • Get Properties of a Blob Container using connection string• Custom transient fault checker for SQL Azure• Custom Random retry strategy for SQL Azure

• Update appSettings section of web.config for TopazStorageConnection and TopazSqlConnection• Create Blob Container named “topaz” in storage account using Cloud Storage Studio

TOPAZ Further ReadingReferences• Download TOPAZ: use nuget• Download Enterprise Library configuration editor: use VS.NET extension manager• Topaz reference guide:

• http://msdn.microsoft.com/en-us/library/hh680901(v=pandp.50).aspx

Autoscaling with WASABiTopic 3

Scenarios and needs?

Autoscaling enables us to take advantage of the elastacity of the cloud

Autoscaling allows us to truly pay for what we use rather than overpay

Autoscaling allows us to deliver a service to our users which has got enough resources for a good user experience

Autoscaling allows us to respond to our application design through it’s operating parameters and performance

Our application needs to increase or decrease in response to greater or lesser load

Our application is busier during the week than over the weekend

We may not know how much load to expect and whether there is a backlog of work to perform

We don’t have the time to be reactive or monitor our application to changes in load

Scenarios Why we need it

WASABi; What it is?!WASABi

• Windows Azure AutoScaling Application Block• Microsoft’s answer to autoscaling• An Enterprise Application Block• A rules based engine• An assembly dll and a set of xsd schema files• A configurable on-premise or cloud hosted service allowing complex rulesets to monitor and respond to 1 … n services• A framework to build complex rules and application monitoring to enable automated elastic activity • A UI-less version of more complicated rules engines such as AzureWatch

WASABi Installation and ConfigurationInstallation

• Install the Enterprise Library Configuration Editor extension in Visual Studio• Use the NuGet package manager console:• Install-Package EnterpriseLibrary.WindowsAzure.Autoscaling

• Direct the Console to a particular project in your solution• Optionally install

Configuration

• Three configuration files needed:• App.config, Rules.xml, Services.xml

• A Storage account with credentials• The following things needs configuring:• Rules, Store, Logging, A hosted service

Using WASABiDetails

• Configuration should be complete first• WASABi can be hosted on-premise in a .exe/windows service• WASABi can be hosted in the cloud within a worker role• Config files can be stored in Blob storage or Local storage• To initiate WASABi the following can be used:

WASABi RulesConstraints• Rules underpin the use of WASABi• Two types of rules:

• Constraints• Reactive

• Example constraint is:• Scaling up or down with the number of role instances on weekdays or special times of the day

Reactive• Reactive rules test a measure to determine• Reactive rules use a condition and a test against a constant to determine whether to scale up or down a role• Reactive rules can be customised against unique application measures called operands• Example reactive rules are:• Scaling up or down in response to a performance counter• Scaling up or down in response to a storage queue

Custom WASABiDetails• WASABi can be customised in the following ways:

• Store• Rules• Operands• Actions

• Custom actions need to have a custom tag referred to in rules.xml • Build a class which uses [XmlRoot] declaratively and inherits from ReactiveRuleActionElement• The implemented CreateAction() method returns a RectiveRuleAction type• The GetResults() method of this type returns a RuleEvaluationResult collection

• Custom operands also need to have a custom tag referred to in rules.xml• Inherit from DataPointsParameterElement overriding properties with custom values and override GetCollectorsFactory() method• Return an implementation of IDataPointsCollector which enables a DataPoint to be assessed against a value in rules.xml and conditional rule logic applied

WASABi DemoAutoscaling an application

WASABi DemoConfiguration• The example contains the following rules:

• Simple rule (constraint - rank 1)• Working hours (constraint –rank 100)• CPU Utilisation operand and > 70 and < 20 rule• Custom RandomDbParameterElement operand to scale up > 5• Custom ConsoleAction action to print a line to the console when a rule is invoked

• Add subscription name, id, certificate store/location, thumbprint id and storage account credentials to services.xml• Deploy any hosted services as needed

WASABi Further ReadingReferences• Download WASABi: use nuget• Download Enterprise Library configuration editor: use VS.NET extension manager• Download WASABi CmdLets, Tailspin sample application and others:

• http://www.microsoft.com/download/en/details.aspx?id=28189 • Beginners autoscaling guide:

• http://www.windowsazure.com/en-us/develop/net/how-to-guides/autoscaling / • MSDN documentation:

• http://msdn.microsoft.com/en-us/library/hh680949(PandP.50).aspx

WIF and Custom ClaimsTopic 4

What is WIF? What are claims?WIF

• Windows Identity Foundation• Microsoft Framework to manage identity • Processes claims and relates them to an identity• Downloading through the Web Platform Installer• Understands protocols such as SWT and SAML

Claims

• Information given by a third party about you• Can be name, email or a variety of facts • These facts can then be translated into something an application can process

Why do we need WIF or claims?Access Control Services

• ACS is a Special Token Service that has integration with IdPs such as Facebook and Live!• It can be used to build claims aware application• ACS can be configured to be aware of your application• It can be used to allow a user to login to your web portal using credentials from Facebook and others IdPs• ACS is the cement between your application and authentication to an IdP • It returns a set of claims securely to your application which can be inspected

Application development

• WIF allows us to build a claims inspection pipeline so that claims can be used to authorize individuals using a custom roles database

Installing WIF to AzureDownloading WIF

• WIF is not present in the Azure role images• An .msu can be installed which can be downloaded from:• http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=17331

• Download v6.0 for 2008 Server SP 3• Download v6.1 for 2008 Server R2• Run the installation as a startup task

How to

• Add the following to your *.csdef file<Startup>

<Task commandLine="Startup.cmd" executionContext="elevated" taskType="simple" />

</Startup>

• Disable auto updates and install using a script

Configuring WIFAlternative ways of copying WIF

• Use Microsoft.IdentityModel.dll in your code• Don’t set copylocal=true to the dll• References in the web.config file look for a GAC-registered assembly• .NET Framework SDK and WIF are not installed by default• gacutil (.NET Framework v3.5) should be copied to your deployment• Simply change the startup command to:• gacutil –i $mypath\microsoft.identitymodel.dll

Access Control Services

• Configure ACS through the use of the ACS portal: setup default namespace and configure application:• http://www.windowsazure.com/en-us/develop/net/how-to-guides/access-control/

Using Claims with two role instancesProblem• Now you’ve developed a claims-aware application and installed WIF you need to run more than a single instance of this application• Create a second instance of your web role and your site • Since the traffic is now load-balanced between two instances some of the requests and images won’t show

Using Claims with two role instancesSolution• Cookie traffic with the claims token is encrypted using DPAPI with a machine-specific key • The cookie cannot be decrypted if the request goes to an alternative instance• Instead of using DPAPI to encrypt data use a shared X509 v3 certificate• To generate this certificate, self-sign by using makecert – the following will install the certificate to the current user’s personal store – export both .cer and .pfx files• makecert –r –pe –n “CN=lwaugac.accesscontrol.windows.net” –sky exchange –ss my

Using Claims with two role instancesAlternative Solution• Alternatively powershell can be used to consolidate several tasks into one.• In this example the script performs the following actions:• Locate the X509 certificate in the store• Read the BASE64 DER-Encoded data• Export this to the harddrive as a .cer file• Export this as a .pfx file providing a password as a secure string• Two files should now be present on the hardrive

Coding for claims-awarenessWiring in the cookie handling and adding to cloud config• Add the service certificate to a microsoft.identitymodel section in web.config

<serviceCertificate>

<certificateReference x509FindType="FindByThumbprint" findValue="55a65a46311fc9b52dcf1ecb0f11d92d08627c5b" />       

</serviceCertificate>

• In the global.asax Application_Start method wire up the following event handler:• FederatedAuthentication.ServiceConfigurationCreated

Coding for claims-awarenessAuthorising with claims• Reference Microsoft.IdentityModel.Claims in code• Create a new class which inherits from ClaimsAuthorizationManager• Pass in config data which contains information on the roles database• Override the CheckAccess method • Return from the method if the URL requested is not the protected URL• Check the claim (usually email) against a database• Return true to permit access• False gives the user an 401 error• Add to web.config with custom XML

Resources for WIFMicrosoft Identity Resources

• Windows Identity Foundation• http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=4451

• Identity Management Information Pack• http://msdn.microsoft.com/en-us/security/aa570351

• Windows Azure Access Control Services• http://www.windowsazure.com/en-us/develop/net/how-to-guides/access-control /

• Single Sign On and Active Directory Federation Services• http://msdn.microsoft.com/en-us/library/bb897402.aspx

Grid on Windows AzureTopic 5

What’s the

Problem?

Some apps run too slowly on a single computer• Splitting them up into

chunks, then running those chunks in parallel on multiple computers can speed them up

Doing this effectively requires creating a compute cluster, with:• Tools to create and manage

the cluster• A scheduler for running apps

on the cluster

Hardware

Compute

Time

25%10%

200%

Adoption

HPC allows you to do more with less, to execute more computationally intensive tasks in a fraction of the time. Where there is any hardware requirement at all, the use of Big Compute methodologies in Windows Azure allows for a massive reduction in the required hardware when compared to traditional solutions

Typical Scenario

Example Workloads That Need a Cluster

• Digital media rendering and encoding• Financial and insurance risk calculation• Engineering modeling and simulation• Computational life sciences• Earth sciences• Data analytics

HPC Scheduler Windows Azure

Microsoft HPC Scenarios

On-Premises.Your own servers inside your enterprise cloud.

Hybrid.Burst from enterprise to cloud.

Cloud.Head node in cloud or compute in cloudOr both.

HPC Components

On-premise

Desktop User

HPC Head Node

Broker Node(s)

HPC Cluster

Desktop Compute Cloud via Idle Win 7 Workstation Cores

Azure Compute Instances

Azure Compute Proxies

HPC Edition

Optimal Cloud Workloads

PredictabilityPredictable Workloads can be scheduledOnly provisioning resource when necessary.

UnpredictabilityUse Azure to burst on sudden demandElastic resource provisioning Service management API allows for automatedelasticity

Predictable Burst

Unpredictable Burst

Growing Fast“On and Off”

Com

pute

Time

Components in HPC SDK for Windows Azure

Parametric SweepsCluster SOAMPILINQ to HPCExcel

Cluster DeploymentMonitoringDiagnosticsReporting

Job submission API and portalJob queue and prioritiesTask activation and monitoringResource sharing policies

AzureScheduler

Distributed Runtimes

System Administration

Embarrassingly Parallel AppsAn illustration

Cluster of Compute Nodes

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Cluster of Compute Nodes

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Application Logic

Tightly Coupled App (MPI)An illustration

HPC Software Classifications

• Message Passing Interface

• Low level API and executable launch • Low level network access uses NetworkDirect • Creates a cluster by assignment of communicators

• Service Oriented Architecture

• Familiar use of WCF and SOA• SOA client can consume services directly or through HPC Scheduler

SOA Services

A familiar place to WCF Developers

• Use familiar WCF model to create service endpoints• Decorate service with [ServiceContract] attributes• Services hosted in an assembly DLL• Services registered using <System.ServiceModel> section and <microsoft.Hpc.Session.ServiceRegistration>• Add configuration to the <microsoft.Hpc.Broker> node• Can typically develop, host and debug locally using Visual Studio• The model assumes WCF bindings and transports• These will be abstracted in the case of using Brokers but defined in configuration

SOA ClientsA powerful abstraction within HPC

• Use Microsoft.Hpc.Scheduler.Session namespace to encapsulate a SOA call via the scheduler

• Client can use service reference and interact with the service via a proxy synchronously or asynchronously

• Can additionally use class BrokerClient<T> with the service interface to broker the message without a direct reference

SOA Deployment

• SOA must be distributed to the headnode and all files packed in a zip using “hpcpack” tool

• Can be uploaded to storage using “hpcpack upload” where it will be distributed to compute nodes

• Client can either reside on the headnode or external using TransportScheme.WebAPI • BrokerClient<T> can be used with a nettcp binding

Deploying an HPC Cluster to AzureVarious ways

• Use of AppConfigure provided free within the Azure Platform Training Kit• Use of HPC Powershell CmdLets to allow for a scripted deployment • Custom method using combinations of Powershell CmdLets, the Service

Management API and startup tasks

What does a deployment entail?

HPC PowershellCreates a storage account

Creates a SQL Server and runs scripts

Creates/Adds a management certificate

Associates a web portal with the web node

Packages HPC Pack with the deployment to install as a startup task

Creates broker objects

Updates Firewall rules

Updates config with values for all HPC plugins

Deploys application

AppConfigureCreates a storage account

Creates a SQL Server and runs scripts

Creates/Adds a management certificate

Associates a web portal with the web node

Packages HPC Pack with the deployment to install as a startup task

Creates broker objects

Updates Firewall rules

Updates config with values for all HPC plugins

Deploys application

With Visual Studio

Without Visual Studio

HPC Services for Excel 2010

Make better business decisions

Respond to market trends faster

Manage risk and portfolio better

Life Insurance Actuarial workbook examples

1700 records that took 14 hours now take 2.5 minutes 1 million records that took 7.5 days now take 2 hours

Increase business agility and accelerate time to results by easily transitioning desktop calculations to Windows HPC computations

The HPC Scheduler for Windows Azure has brought cluster computing to the masses!

Resources for HPCMicrosoft HPC Resources

• Windows Azure Platform Training Kit• Windows HPC Survival Guide• http://blogs.msdn.com/b/hpctrekker/

• Windows HPC Pack SDK• http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=10505

• Cluster SOA Debugger• http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=21705

• Windows HPC Team Blog• http://blogs.technet.com/b/windowshpc/

REST Clients on Windows AzureTopic 6

Introduction

REST APIs on AzureStorage Services APIService Management APIService Bus Relay EndpointsService Bus Management (some functions)SQL Azure Management REST APIAccess Control Services Management REST API

RESTREpresentational State Transfer: Use of HTTP Verbs, reponse codes and payloads to build APIs.

Microsoft’s support for REST is intrinsic to WCF and Windows Azure.

Service Management underpins everything in Windows Azure

What do we mean by ManagementOperations• CRUD (Create Read Update Delete)• Manage transaction security• Checking full audit history• Managing locations and “affinity groups”• Accessing and managing storage accounts• Track asynchronous long running operations• Get information about the host operating system• Manage services like the Service Bus, ACS and SQL Azure

Core APIs to review• Service Management API• Storage Services API

Managing ServicesCRUD and REST• Listing hosted services• GET http://management.core.windows.net/[subscription id]/services/hostedservices HTTP/1.1

• Creating a hosted service• POST http://management.core.windows.net/[subscription id]/services/hostedservices HTTP/1.1

• Deleting a hosted services• DELETE http://management.core.windows.net/[subscription id]/services/hostedservices HTTP/1.1

• Updating a hosted service• PUT http://management.core.windows.net/[subscription id]/services/hostedservices HTTP/1.1

Things to note• Service Management request are RESTful• This means that they are platform and language agnostic• The same API call coupled with a different HTTP VERB• POX used instead of SOAP in body of request

REST Management is Secure

Certificates and IntegrityCertificates• X509v3 Certificates deployed to portal• Two flavours• Management Certificates (.cer)• Service Certificates (.pfx)

• Use .publishsettings file from SDKv1.6• Use development tools such as makecert • Create chain of trust using trusted third parties

Integrity• Symmetric keys used with all transfers from/to storage, service bus and access control• Integrity checks done through signature schemes within storage HTTP header• Standard use of Authorization header with Shared Access integrity check • HMAC-scheme calculated using primary/secondary symmetric storage key

25%

200%

x-ms-date: Sun, 30 Nov 2011 06:02:27 GMTAuthorization: SharedKeyLite smarxtest:yNeUvY5puNFfdMENJdHxRc1n5E/Qetlyyk9fIX/tVmM= Content-Type: application/xmlx-ms-request-id: 24f43295-e6d5-45f6-abca-6da3b99fe758 (response)x-ms-blob-type: BlockBlob

Example Headers

25%

200%

<?xml version="1.0" encoding="utf-8"?><CreateHostedService xmlns="http://schemas.microsoft.com/windowsazure"> <ServiceName>lwaugtest1.bad</ServiceName> <Label>djE=</Label> <Description>this is the first test service for the user group</Description> <Location>North Europe</Location></CreateHostedService>

Example Request

25%

200%

HTTP/1.1 400 Bad RequestContent-Length: 194Content-Type: application/xml; charset=utf-8Server: Microsoft-HTTPAPI/2.0x-ms-request-id: 3badf9cdddc540cd922993fa01598297Date: Mon, 05 Dec 2011 18:16:45 GMT

<Error xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Code>BadRequest</Code><Message>The hosted service name is invalid.</Message></Error>

Example Response

25%

200%makecert -a sha1 -n CN=ukwaug.cloudapp.net -pe-r-

sky exchange -ss My -sr LocalMachine

Certificate creation

25%

200%

#create a new hosted service called 'ukwaughosted' Step 1$subscriptionid = ‘<subscription id>'$hostedservicename = 'ukwaughosted'$location = 'North Europe'$hostedservicedesc = 'The UK Windows Azure Users Group'$certificate = Get-ChildItem -path cert:\CurrentUser\My\<my thumbprint>New-HostedService -ServiceName $hostedservicename -Label $hostedservicename -SubscriptionId $subscriptionid -Location $location -Description $hostedservicedesc -Certificate $Certificate

Example Powershell New-HostedService

25%

200%

Deploy.Deployment(“<path”>) .WithCertificate(‘<thumbprint>’) .AndInstanceCount(5) .ForRole(‘<role name>’) .AndInstanceCount(5) .ForRole(‘<role name 2>’) .StartAutomatically() .TreatWarningsAsErrors(true) .Go();

Available @ http://blog.elastacloud.com from next week!

Fluent Service Management

Building a REST client Request for Windows AzureCommon format• Build a string Uri• Add an HTTP Verb, PUT, POST, GET or DELETE• Add an application/xml content-type• Add an x-ms-version e.g. 2011-10-01 to denote the API version you want• Add a subscription ID as part of the Url • Add a path to denote a method • Use a StreamWriter to write content to the body and UTF8 encode• Add a certificate to the request OR a shared access auth signature

Parsing a REST client Response for Windows AzureCommon format• Execute the request asynchronously/synchronously and use a delegate to parse the response• Determine the HTTP Status Code to see if an exception has occurred• Check the async operation using a continuation token to see if it is in progress/succeeded or failed• Read the response body using streams and deserialise the XML payload into a .NET object• REST is true write once, run anywhere!

Resources for REST and AzureMicrosoft Azure Resources

• Service Management API• http://msdn.microsoft.com/en-us/library/windowsazure/ee460799.aspx

• Storage Services API• http://msdn.microsoft.com/en-us/library/windowsazure/dd179355.aspx

• Building RESTFul Clients• http://msdn.microsoft.com/en-us/magazine/ee309509.aspx

The Codeplex, MSDN and Git Top 10Topic 7

Neudesic Storage ExplorerDetails

• Fully functional UI for:• Blobs• Tables• Queues

• Full implementation for Create, Read, Update, Delete• Embedded viewer for Blob images and videos• Ability to inspect Queue Storage Message• Table Storage shown in cell based table format showing, Row and Partition Keys• Direct data editing in the tables• Underlying use of the .NET Azure StorageClient library

Azure Accelerator for Web RolesDetails

• Can be found @ https://github.com/WindowsAzure-Accelerators/wa-accelerator-webroles• Developed by Nathan Totten of Microsoft• Can be used to deploy one or more websites across multiple web role instances• Use of WebDeploy which is ordinarily constrained to a single instance and update• WebDeploy much faster deployment strategy compared to normal Azure deployment• A Project template is supplied and allows for the addition of IIS applications, certificates and site bindings• The project acts as a host for the deployment of sites across multiple role instances • WebDeploy allows Administrators and users to deploy ASP.NET and PHP applications for IIS7.• It has a synchronisation mechanism built in to ensure that file copies are the latest version• Synchronises content between all sites under management

Partitioned Cloud QueueDetails• Downloadable @ http://partitioncloudqueue.codeplex.com/ • Limitation of 500 transaction/second per Storage Queue• Partitioned Cloud Queue class extends the scalability limit• Transparent queue sharding/partitioning via simple single-class client library which extends the scalability target to 500*N messages per second. (N- number of partitions) • 1:1 compatibility with CloudQueue (except constructor) which allows PartitionedCloudQueue to replace CloudQueue with just minor code changes. • Round-robin partition access for even load disctribution. • Avoidance of correlated load spikes via randomization of round-robin access order. • Support synchronous and asynchronous operations as defined in CloudQueue. • Auto-discovery of number of partitions.

ASP.NET Table Store Provider for MembershipDetails• Downloadable @ http://code.msdn.microsoft.com/Windows-Azure-ASPNET-03d5dc14 • Provides SessionState and Membership within Table Storage• Framework is written as provider framework to allow easy addition to web.config• Add a reference to the AspProvider project or complied library• Add Login and registration pages• Configure Authentication to use Forms• Configure Authorization to restrict anonymous users • Configure ASP.NET to use the Azure Table Storage Provider• Make updates or changes to the Membership provider config• Provider name is Microsoft.Samples.ServiceHosting.AspProviders.TableStorageMembershipProvider• Add appSettings defining the Table Storage endpoints:

• <appSettings> <!-- account configuration --> <add key = "TableStorageEndpoint" value="http://lwaugbe.table.core.windows.net"/> <add key = "AccountName" value=“lwaugbe"/> <add key = "AccountSharedKey" value="Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="/> </appSettings>

Storage File SynchronisationDetails• Downloadable @ http://code.msdn.microsoft.com/Synchronizing-Files-to-a14ecf57 • Synchronises files from a local store to Storage blobs• Built as a provider using the Sync Framework• Specifically uses the file synchronisation provider as underlying source• Handles things like renames and file moves and updates Blob repo• Need to give the sync framework information about the files • Also need to add an IdentityRule to track custom fields based on a metadata schema• These can be done programmatically through the IdentityRule and CustomField classes• These CustomFields will then sync with Blob Metadata• In order to run the file sync sample the Microsoft Sync Framework v2 SDK must be downloaded from here:• http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=14159

ELMAH for Windows Azure TablesDetails• Elmah (Error Logging and Modules and Handlers) @ http://code.google.com/p/elmah/ • Is a framework for ASP.NET for error reporting• Elmah has and does:

• Logging of nearly all unhandled exceptions. • A web page to remotely view the entire log of recoded exceptions. • A web page to remotely view the full details of any one logged exception, including colored stack traces. • In many cases, you can review the original yellow screen of death that ASP.NET generated for a given exception, even with customErrors mode turned off. • An e-mail notification of each error at the time it occurs. • An RSS feed of the last 15 errors from the log. • Exports CSV files• Can tweet exceptions• Has a whole of back-end stores from SQL Server, Oracle to Microsoft Access• Easy to create your own views with an extensible API• Able to export exception information in JSON as well as XML

• Now has a Table Storage store downloadable via nuget• Install-Package WindowsAzure.ELMAH.Tables

ASP.NET Universal ProvidersDetails• Providers are composed of SessionState, Membership, Roles and Profile data • They are currently in use in many applications and strategies need to be put into place to migrate these seamlessly to Windows Azure• The Web Platform and Tools Team has released the ASP.NET Universal Providers• This contains support for SQL Azure• No changes except the use of the DefaultProviders• No code changes are necessary when you use these• New SQL Scripts can be downloaded from• http://support.microsoft.com/kb/2006191

• These scripts contain modifications to the originals that allow the constraints in the T-SQL, system stored procedures and others that are imposed by SQL Azure to be bypassed• The SessionState table script has been removed because jobs which occur via SQL Agent cannot be run in the cloud• Everything is installable through nuget • Install-Package System.Web.Providers

• These could be removed and timers used with worker roles if essential

Service Bus ExplorerDetails• Downloadable from http://code.msdn.microsoft.com/Service-Bus-Explorer-f2abca5a• Lists out all facets of AppFabric Service Bus

• Queues• Topics• Relay Services

• Allows different queues to be inspected including the dead letter queue• Detects duplicate messages• Inspects all queue properties and the default TTL of the queues and messages• Uses the Service Bus REST API for management• The Explorer tool provides the ability to inject messages and import and export data• All entities can be managed through the UI

Windows Azure Toolkit for Social GamesDetails• Built by Microsoft evangelists• This is a set of samples and tools which are used to build and deploy social games• The toolkit contains several sample games and APIs• All game samples are web-based and are configured to use CDN, Storage Services etc.• There is a default game in the toolkit called Tankster which is a multiplayer game supported by Azure services• The toolkit covers things like:• Leaderboards• Friends, notifications and invites

• Server APIs provided are:• An event service• Game service• User service• And Game Queue Blob

Patterns and Practices websiteDetails

• A guide deployed to http://wag.codeplex.com/• Breaks down common tasks and scenarios and provides samples• Works with the premise that many users will want to migrate applications to the cloud and provides guidance• Provides samples and guidance on:• Authentication and authorization• Scalability and resilience • Data Synchronisation • Creating distributed data applications• Working with the Service Bus relay to provide hybrid applications • Interoperability between ways• General application patterns and designs

Diagnostics in Windows AzureTopic 8

Not just for emergencies

Overview

Diagnostics in Windows AzureApplication LevelService LevelOperating System Level

App

• Things your application chooses to log

Service

• Things logged by what runs your application

OS

• Things logged by OS container

Example Trace Usage

Application:• Error: Cannot save Order

Service:• Error: IO Exception

OS:• Error: Disk is full

Windows Azure Diagnostics

What do you need?Hosted Service

Compute capacity within Windows Azure

Storage AccountStorage capacity within Windows Azure

TableBlobs

Centralised storage area into which each Hosted Service can write

What’s special?Familiar diagnostic challenges

Already solved on premises

Massively scalableNo request affinityUnpredictable application topology

Where do the logs go?!Centralised Logging approach

Example Trace PointsApplication• .net application Tracing• custom• Files written via nLog, log4net etc

• Windows Event LogService• IIS Logs• IIS Failed Request Logs• Azure Diagnostics Infrastructure Logs• Metalog

• Performance CountersOperating System• Windows Event Log• Custom (any known file log)

Transfer

• Two distinct methods of transfer• Same underlying result

On Demand Transfer

Scheduled Transfer

Output formatsType Description Destination

Trace With Trace Listener Table WADLogsTable

IIS Logs W3svc format Blob

Windows Event Logs Table WADWindowsEventLogsTable

Windows Performance Counters Table WADPerformanceCountersTable

Custom Logs File based Blob

Crash Dump Blob

Diagnostic Infrastructure Log Metalog Table WADDiagnosticInfrastructureLogsTable

How you do it

• .net Tracing• Add Trace Listener

• Configure WAD• View Result:

Native code example

Fluent Diagnostics

Attempts to curtail verbosityGet it on Nuget: Install-Package Bareweb.FluentAzureDiagnostics.dll

How to view Diagnostics

• Raw in Tables, as shown before.• Cerebrata Azure Diagnostics Manager

Further information

A summary of available topics and blogs:http://msdn.microsoft.com/en-us/library/windowsazure/gg433048.aspx

Cerebrata Diagnostics Managerhttp://www.cerebrata.com/products/AzureDiagnosticsManager/

Automation with PowershellTopic 9

Total control of Windows Azure

Powershell

For Windows Azure?Implementations of many common Windows Azure tasks are available in Windows Azure. It is not Powershell itself that is performing many of these tasks, but it is often calling the external resource such as the Windows Azure Service Management API in order to achieve these results.

Typical uses are to perform remote setup and configuration tasks initiated from outside the cloud, and complex local tasks such as startup tasks that prepare an instance as it is initializing.

What is it?Windows PowerShell® is a task-based command-line shell and scripting language designed especially for system administration. Built on the .NET Framework, Windows PowerShell helps IT professionals and power users control and automate the administration of the Windows operating system and applications that run on Windows.

http://wappowershell.codeplex.com

• Download through Web Platform Installer• Open source and free!• Over 120 different tasks

Example Tasks

Get-StorageAccount

New-SqlAzureFirewallRule Add-Certificate

Set-RoleInstanceCount

Example Script

Courtesy of http://michaelwasham.com

Cerebrata Powershell Cmdlets

• Consolidate many tasks into single commands• Abstracting away multiple features using a single command

• Strong tooling around Storage

• Paid for at $125USD• 30 Day trial

• Around 100 tools.

Comparison of wapps and Cerebrata cmdlets• Wapps• Open source• Task oriented

• Cerebrata cmdlets• Support larger units of operation

Example Deployment with Cerebrata

Get-Command –PSSnapin AzureManagementCmdletsSnapIn #create a new hosted service called 'ukwaughosted' Step 1$subscriptionid = '67b7c755-8382-4990-b612-0006cd24e1ba'$hostedservicename = 'ukwaughosted'$location = 'North Europe'$hostedservicedesc = 'The UK Windows Azure Users Group'$certificate = Get-ChildItem -path cert:\CurrentUser\My\AA4EF678D0961B6A6C51D4AC657B0ADB71BB3354New-HostedService -ServiceName $hostedservicename -Label $hostedservicename -SubscriptionId $subscriptionid -Location $location -Description $hostedservicedesc -Certificate $Certificate  #deploy the package and config file to the new hosted service Step 2$CscfgFile = 'C:\Projects\Tech Projects\London Windows Azure User Group\HelloCloud\HelloCloud\bin\Release\app.publish\ServiceConfiguration.Cloud.cscfg'$CspkgFile = 'C:\Projects\Tech Projects\London Windows Azure User Group\HelloCloud\HelloCloud\bin\Release\app.publish\HelloCloud.cspkg'New-Deployment -Slot Production -PackageLocation $CspkgFile -ConfigFileLocation $CscfgFile -Label 'UKWAUG Deployment' -ServiceName $hostedservicename -RunDeploymentAfterCreation $TRUE -TreatWarningsAsError $FALSE -SubscriptionId $subscriptionid -Certificate $certificate #remove deployment Step 3Remove-Deployment -Certificate $certificate -SubscriptionId $subscriptionid -ServiceName $hostedservicename -Slot Production #remove hosted service Step 4Remove-HostedService -Certificate $certificate -SubscriptionId $subscriptionid -ServiceName $hostedservicename

Successful case studies enabled by Powershell• Achieve CI with automated deployments• Support team members with automated certificate management• Backup and restore SQL Azure Databases• Download blob storage containers to local file systems• Build auto-scaling engines remotely and abstract Service Management

operations from this• Modify Windows Azure Diagnostics settings at runtime and manage

on-demand transfers• Clear whole tables of data and whole containers of files

Where to find out more

• http://wappowershell.codeplex.com/documentation

• http://michaelwasham.com/

• http://www.cerebrata.com/Products/AzureManagementCmdlets/

SQL Azure MigrationTopic 10

Migrating legacy databases

SQL Azure High AvailabilityScalable SQL database technology

Functionality differencesIn order to support cloud functionalityAnd scalability, some features are unavailable

Sync strategiesEasily share database content between onPremises and cloud based servers

Moving isn’t always easy

Fully or Partially Supported tools

• Constants• Constraints• Cursors• Extension of spatial data types and methods through CLR• Index management and rebuilding indexes• Local temporary tables• Reserved keywords• Spatial data and indexes• Stored procedures• Statistics management• Transactions• Triggers• Tables, joins, and table variables• Transact-SQL language elements such as • Create/drop databases• Create/alter/drop tables• Create/alter/drop users and logins

• User-defined functions• Views

Unsupported Features

• Common Language Runtime (CLR)• Database file placement• Database mirroring• Distributed queries• Distributed transactions• Filegroup management• Global temporary tables• SQL Server configuration options• SQL Server Service Broker• System tables• Trace Flags

Automated migration advice• http://sqlazuremw.codeplex.com/

Bi-DirectionalityThe SQLAzureMW tools greatly simplify schema analysis and migration process. If you don’t have an SQL Azure account and have been thinking about moving your data to the cloud (SQL Azure), but have been afraid to try because of “unknowns” like cost, compatibility, and effort? SQL Azure Migration Wizard SQLAzureMW is a free set of open source applications that have been developed by the database community to help you address these issues. SQLAzureMW will help you analyze your SQL Server database for compatibility issues and will migrate your schema and data to SQL Azure. 

HowSQL Azure Migration Wizard is an open source application that has been used by thousands of people to migrate their SQL database to and from SQL Azure. SQLAzureMW is a user interactive wizard that walks a person through the analysis / migration process.

NotSupportedByAzureFile.config

KeyThis file contains the rules used to validate generated SQL Schema.

Not only does this give clarity on the test to be run, but you can modify it to assert situations such as “If I remove my reliance on replication, will my database be compatible” Full control over migration advice

Migration Wizard

Allows Analysis of Profile TraceRather than scripting schema, capture real usage.

Enable Development Analysis In a greenfield project with eventual SQL Azure databaseyou can use an on premises SQL Server database and regularly test compatibility with target server

Batch processing of AnalysisIt is possible to run the Migration Wizard in a command Prompt by running SqlAzureMWBatch.exe

Data Sync Framework• http://sqlazuremw.codeplex.com/

WhyWindows Azure doesn’t support Replication, so how do you manage an existing on-premises database server syncing to the cloud?

WhatSQL Azure Data Sync enables you to easily create and schedule bi-directional synchronizations from within the Data Sync web site without the need to write a single line of code.

Key Features of Data Sync

• Code free• Security• Sync Groups of two or more databases• Rich reporting in the management portal

DataSync run through

Where to find out more

• SQL Azure Migration Wizard: http://sqlazuremw.codeplex.com/

• Sync Framework: http://msdn.microsoft.com/en-us/sync/bb736753

Q&AThanks for Listening!

Check out http://blog.elastacloud.com – samples will be posted now

For more information on our consultancy, training and product range, contact:

info@elastacloud.com