2014.11.22 Azure for Sql Server Developer - SQLSAT355 Parma

#sqlsatParma

#sqlsat355November 22nd, 2014

Azure for SQL Server Developer

Marco Parenzan

@[email protected]

#sqlsatParma


Sponsors

#sqlsatParma


Organizers

#sqlsatParma


Speaker info/Marco Parenzan

Formazione & Divulgazione con 1nn0va

Microsoft MVP 2014 for Microsoft Azure

Formazione & Progettazione con ITS

Kennedy

Ricerca e Innovazione con Servizi Cgn

www.slideshare.net/marco.parenzan

www.github.com/marcoparenzan

marco [dot] parenzan [at] live [dot] it

www.innovazionefvg.net

@marco_parenzan

Cloud Architect

SQL Server Friend

Loves Functional Programming, Html5 Game Programming and Internet of Things

http://www.innovazionefvg.net/

http://www.tecnicosuperiorekennedy.it/

http://www.cgn.it/

http://www.slideshare.net/marco.parenzan

http://www.github.com/marcoparenzan

http://www.innovazionefvg.net/

#sqlsatParma


Agenda

Why Azure?

OPS Experience

DEV Experience

BI Experience

Conclusions

#sqlsatParma


WHY AZURE?

#sqlsatParma


Trends driving IT pressures & opportunities

#sqlsatParma


Data storage

Cloud identity

VMs on demand

Disaster recovery

Deploying packaged applications

Moving existing applications to the public cloud

Utility ITExample scenarios

Utility ITStrategic

IT

#sqlsatParma


The Microsoft data platform

#sqlsatParma


SQL Server 2014 built-in hybrid technologies

#sqlsatParma


Azure Data Services for Modern App Design

#sqlsatParma


Barriers to Public Cloud AdoptionSecurity

Can a public cloud platform keep my data and applications safe??

ANSWER

You must learn to trust your

public cloud provider

#sqlsatParma


Can I still meet my regulatory requirements in the public cloud??

Barriers to Public Cloud AdoptionCompliance

ANSWER

You must understand the rules

that apply to you

#sqlsatParma


Barriers to Public Cloud AdoptionAvailability

Will public cloud datacenters be up when they’re needed??

ANSWER

They’ll be at least as good as

your own datacenter, and there

are SLAs

#sqlsatParma


Barriers to Public Cloud AdoptionThe unspoken worry

Will public cloud platforms make me lose my job??

ANSWER

Probably not, but you’ll need to

learn some new skills

People who don’t like change shouldn’t work in the technology industry

#sqlsatParma


OPS EXPERIENCE

#sqlsatParma


Handle relational data with IaaS

You can create a VM with Windows Server

You can install any software you want if you have your licence

If you need a licence

In case you don’t have a licence, you have VM images ready with software and pay as you go (hour/minute basis)

You can scale in “minutes” …or in “real” hours…

…but not weeks (or at least days!)

#sqlsatParma


Virtual Machine Storage Architecture

Azure Virtual Machine

C:\

OS DiskE:\, F:\, etc.

Data DisksD:\

Temporary Disk

(Contents can be lost)Disk Cache

#sqlsatParma


The G family

Optimized for data workloads

Up to 32 CPU cores, 448 GB RAM,

6.5 TB local SSD

Latest generation Intel processor

G

#sqlsatParma


Handle relational data with IaaS

Pro

Full power of a Relational DB

Simple Scaling Up

No licencing

Pay Per Use (no CAPEX)

Full power control

Ready images (no installation)

With specific configurations

Just Click, Name, choose,

deploy

Cons

OPEX: highest

Relational (non open) DBMS

are expensive

Bits+Management

You have to be a real

Administrator

Scalability: you can ONLY

scale up

Current sharding needs

application to be updated

Current sharding is in

experimentation stage

#sqlsatParma


DEMO

#sqlsatParma


AZURE STORAGE

One of the first two services (the other were Cloud Services)

#sqlsatParma


Azure Storage Architecture

“Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency”, ACM

Symposium on Operating System Principals (SOSP), Oct. 2011

http://blogs.msdn.com/b/windowsazurestorage/archive/2011/11/20/windows-azure-storage-a-highly-available-cloud-storage-service-with-strong-consistency.aspx

#sqlsatParma


Durability Options

Locally Redundant Storage (LRS) Stores 3 replicas of the data within a single zone (facility) in a single region

Provides data durability for disk, node and rack failures

Geo Redundant Storage (GRS) Stores 6 replicas of the data across two regions (3 in each region)

Provides additional durability to protect data against major regional natural disasters

(e.g., tornado, hurricane, fire, etc, destroying a whole region)

Updates across regions are performed asynchronously

Zone Redundant Storage (ZRS) Stores 3 replicas of the data across multiple zones (facilities) within a single region or

across regions

Provides additional durability to protect data against zone failures (e.g., fire burning

down a facility)

#sqlsatParma


Premium storage

Up to 32 TB of storage per VM

>50,000 IOPS per VM

Less than 1ms read latency

#sqlsatParma


Abstractions – Tables

Tables – Massively scalable NoSQL cloud store Key/Attribute(s) store at scale

Store user, device or any type of metadata for your service

Auto load balances partitions to meet traffic needs

OData protocol (AtomPub or JSON)

#sqlsatParma


Abstractions – Azure Storage Files

Move on-premises applications to cloud

VMs can net use an SMB share using standard file APIs and semantics

SMB 2.1 protocol

VM and storage account within same region

Supports REST and SMB protocol access to same file share

Azure Storage

Blobs

Tables

Queues

Files

Share data stored in Azure Files

among Azure VMs via SMB

Microsoft Azure

SMB

RES

T

API

RES

T

API

#sqlsatParma


Abstractions – Blobs

Blobs – Massively scalable object store in the cloud Simple REST interface (Put, Get, Delete)

Data sharing – share documents, pictures, video, music, etc.

Big Data – store raw data/logs and compute/map reduce over data

Backups – data and device backups

#sqlsatParma


Virtual Machine Storage Architecture

Azure Virtual Machine

C:\

OS DiskE:\, F:\, etc.

Data DisksD:\

Temporary DiskDisk Cache

G:\, H:\, etc.

SMB Share

#sqlsatParma


Make a SQL backup over Azure

Backup works if move data in a far and safe place

You do it on a daily basis

It can be tested

SQL Server 2012/2014 can backup over Azure Storage Set azure credentials in SQL Server to authenticate on-

premise SQL over Azure (share a certificate between Azure and on-premise host)

Indicate the container and the blob you configure for this activity

Do the backup

Restore backup to the new host

#sqlsatParma


Breaking News – Premium Storage

Consistent low latency SSD based with predictable IO throughput

Suitable for high-performance IO-intensive database workloads

Single digit milliseconds latencies

Supports up to 1 TB blob/disk size

Stripe up to 32 disks for a total of

32TB and more than 50,000 IOPS

Premium Storage Disks work in

conjunction with a new VM series

#sqlsatParma


DEMO

#sqlsatParma


Handle relational data with SaaS

You don’t want to handle management issues

You don’t want vertical scalability-related

issues

You don’t need special infrastructural

architecture

#sqlsatParma


Microsoft SQL Platform Continuum

Physical

Virtual

Infrastructure as a Service

Platform as a Service

SQL Server

Physical Machines (raw iron)

SQL Server Private Cloud

Virtualized Machines + Appliances

SQL Server in Azure VM

Virtualized Machines

Azure SQL Database

Virtualized DatabasesSQL

Dedicated

Higher Cost

Shared

Lower

Cost

Higher Administration Lower Administration

Hybrid

Cloud

On Premises

Off Premises

SQL SQL

SQL SQL SQL

SQL SQL SQL

SQL

#sqlsatParma


Azure SQL Database Service Tiers

• •• •••

• •• •••

#sqlsatParma


Scalable Performance

Basic, Standard, and Premium provide increasing performance levels

Relative performance is expressed in database throughput units (DTUs)

DTU is a blended measure of CPU, read IO, write IO, and memory

Scale performance up/down via portal, APIs, PS, or T-SQL

to reflect actual or anticipated demand

Database remains online while scaling

Hourly billing improves cost efficiency

Performance is easily scaled up or down to

meet changing workload and business needs

B S0S1

S2P1

P2

P3

#sqlsatParma


Data Protection

Automatic, continuous backups

Retention varies by service tier: Basic 7 days, Standard 14 days, Premium 35

days

Daily backups also copied to a remote datacenter

“Oops” recovery

Self-service point-in-time restore from local backups, including deleted

databases

Data center disaster

Self-service restore from remote backups to any region at any time

No charge for backups, normal database rates apply for restored databases

Basic, Standard & Premium databases are

protected from accidental data corruption, deletion

& disaster

#sqlsatParma


Handle relational data with SaaS

Pro

OPEX: lower than IaaS

Bits+Management

Full power of a Relational DB

Editions

Clustering

CPU+Memory

No licencing (No CAPEX)

Pay Per Use (OPEX)

No need for images

Cons

OPEX: lower, than IaaS, but

more than other

Partial power of a Relational

DB

…

Configure what is just

available

Scalability: can ONLY scale up

#sqlsatParma


Migrating your on-premise SQL to SQL

Azure

Not all features of On-Premise SQL are availavable in SQL Azure A “backup” is not enough to bring DB to the cloud

Exporting a script

Some tools are available, updated Developer Way: Database Project, targeting SQL

Azure

Operations Way: Database | Tasks | Deploy to Windows Azure SQL Database

https://sqlazuremw.codeplex.com/

https://sqlazuremw.codeplex.com/

#sqlsatParma


SQL Azure performance considerations

Microsoft is making huge investments

Compatibility with full t-sql will improve

DB per customer

Hundreds/thousands of databases

#sqlsatParma


DEMO

#sqlsatParma


Lower costs and improve predictability

Automation

Enable service owners to focus on work that adds business value

Reduce error-prone manual activities while lowering costs

Process automation that simplifies cloud

management

Optimize and extend

existing investments

IntegrationIntegrate into existing systems with

PowerShell integration modules

Build additional PS modules to

enable integrating into other

systems

Deliver flexible and reliable services

Orchestration

Accelerate time to value with flexible process workflows

Improve service reliability across multiple tools, systems, and department silos

#sqlsatParma


Built on PowerShell Workflow

PowerShell Workflow

•Use Windows PowerShell syntax

•Multi-device management

•Running a single task to manage complex, end-to-end processes

•Automated failure recovery

•Connection and activity retries

Centralized store

•Credentials / certificates

•Global variable

•Global connection for runbooks

•Modules

•Runbooks (draft / published versioning)

•Scheduling

Highly Available

•Runbook servers to process jobs

•Odata Web service to submit / retrieve status

•SQL Server clustering / always on

Historical Analysis

•Historical view of runbook jobs

•Reporting through Excel PowerPivot for ROI

•View runbook used for all jobs

PowerShell Workflow

Centralized store

Highly Available

Historical Analysis

Microsoft Azure

Automation

#sqlsatParma


Runbook Authoring

Author Create PowerShell Workflow

runbooks

Call existing runbooks in library

Manage and browse and insert assets in runbooks Modules and Activities

Credentials (PowerShell Credentials and Certificates)

Variables (standard and encrypted)

Connections

Schedules

Test Run runbook and see results within

authoring window

Troubleshoot issues

Publish Edit draft before publishing

#sqlsatParma


Manage Runbooks & Jobs

Dashboard View View runbook jobs over time

Find jobs that may need attention

Runbooks View Filtering of jobs based on status and

date

Identify authoring state (New, In edit, Published)

Filter by tags to group related runbooks

Jobs View History of jobs per runbook

Who last updated and when

Input parameters and output

Drill into each job to view streams generated to help troubleshooting

Stop, Suspend and Resume job

#sqlsatParma


DEMO

#sqlsatParma


Resource Manager

Today Challenge

Deploy or update a group of resources, repeatedly

Manage permissions on a group of resources

Visualize a group of resources in a logical view,

including monitoring/billing

Resource Manager

Application Lifecycle Container

Declarative solution for Deployment and

Configuration

Consistent Management Layer

#sqlsatParma


Resource Groups

Tightly coupled containers of multiple resources of similar or different types

Every resource *must* exist in one and only one resource group

Resource groups can span regions

Resource Group is a unit of management

Lifecycle: deployment, update, delete, status

Grouping: metering, billing, quota: applied & rolled up to group

#sqlsatParma


Instantiation of repeatable config.Configuration Resource Group

Power of Repeatability

Azure Templates can: Ensure Idempotency

Simplify Orchestration

Provide Cross-Resource Configuration and Update Support

Azure Templates are: Source file, checked-in

Specifies resources and dependencies (VMs, WebSites, DBs) and connections (config, LB sets)

Parametizedinput/output

SQL - A WebsiteVirtualMachines

SQL-AWebsite[SQL CONFIG] VM (2x)

DEPENDS ON SQLDEPENDS ON SQL

SQL CONFIG

#sqlsatParma


DEMO

#sqlsatParma


DEV EXPERIENCE

#sqlsatParma


The big issue leading to CQ(R)S

A single model (tipicallyfrom a Database schema) caring about all aspects of the domain is hard

Complex aggregate roots that fits costly intoa relational db mapping cost is higher

than access cost (crud)

Eventual consistency is a must Acid costs

Presentation layer

Application layer

Infrastructure layer

Domain layer

#sqlsatParma


Transaction Script

Data Storage

Application Services

Remote Facade

Client

Request DTO

DTO Returned

Send DTO

Status Returned

#sqlsatParma


Active Record

Data Storage

Application Services

Remote Facade

Client

Request DTO

DTO Returned

Send DTO

Status Returned

Entity Entity

#sqlsatParma


Command/Query Responsibility Segregation

Domain Model in the Command stack Domain Model focused on behavior that alters state

Ad-hoc storage for queries in the Query stack Table-per-view strategy

Query through LET

Command/Query Separation (cit. Bertrand Meyer – 1980s)

Query

> Returns data

> Doesn’t alter state

Command

> Alter state

> Doesn’t return data

#sqlsatParma


CQRS in a sketch

Command Handlers

Client

Send Command

Ack/Nak

Response

Domai

n

Object

Domain

Object

Repository

Query Handlers

Data Storage

Event Bus

Eve

nt

Ha

nd

ler

s

Request DTO

DTO Returned

#sqlsatParma


Write DB with DocumentDB

Produce and consume data at a staggering rate while requiring near instantaneous response times to match user expectations

Developed iteratively with many versions supported concurrently and continuously evolving data models

Increasingly complex, compensating for the lack of query and transactional processing in today’s NoSQL database systems

Experience unpredictable and explosive growth

It is a fully managed, highly scalable, queryable, truly schema-free JSON document database, delivered as a service for modern applications

SQL

query over schema-free JSON

multi-document

transactions

tunable, high performance

designed

cloud first

#sqlsatParma


DEMO

#sqlsatParma


The CAP Theorem

Theorem: You can

have at most two of

these properties for

any shared-data

systemConsistency

Partition

tolerance

Availability

#sqlsatParma


Consistency

Two kinds of consistency:

strong consistency – ACID(Atomicity Consistency

Isolation Durability)

weak consistency – BASE(Basically Available

Soft-state Eventual consistency )

#sqlsatParma


Eventual Consistency

When no updates occur for a long period of time, eventually all updates will propagate through the system and all the nodes will be consistent

For a given accepted update and a given node, eventually either the update reaches the node or the node is removed from service

Known as BASE (Basically Available, Soft state, Eventual consistency), as opposed to ACID

#sqlsatParma


Availability

Traditionally, thought of as the server/process

available five 9’s (99.999 %).

However, for large node system, at almost

any point in time there’s a good chance that a

node is either down or there is a network

disruption among the nodes.

Want a system that is resilient in the face of

network disruption

#sqlsatParma


Asynchronous Geo-replication with SQL

Azure

Standard geo-replication (Standard and Premium) Optional non-readable secondary in paired region

Secondary database is charged at discounted rate

Failover enabled by Microsoft if disaster occurs

Active geo-replication (Premium) Up to 4 readable secondaries

Full control over secondary location and failover

Supports load balancing, application upgrade and relocation scenarios

Can be combined with a non-readable secondary

Geo-replication minimizes business

interruption from possible disasters

DB

1DB

1

DB

1DB

1

DB

1

#sqlsatParma


DEMO

#sqlsatParma


Azure Caching

Opportunity

Lower accesses to SQL Azure and cost less

Cache the page?

Cache the content of a Page

#sqlsatParma


Handle your “repeatable data” with Redis

Declarative approach Configure ASP.NET MVC to use Redis

Session State or Output cache

Look At Andrea Dottor presentazione about it

Redis as an efficient in memory key/value store

Find many .NET clients in nuget Example Stack Exchange implementation

Implements a cache-aside pattern

Query cache first If not found, Query datasource and storie info incache

Cache invalidaton rules have to be handled Explicit

Timeout

#sqlsatParma


DEMO

#sqlsatParma


Azure Search

A search-as-a-service solution allowing

developers to incorporate great search

experiences into applications without

managing infrastructure or needing to

become search experts.

It is based on Elastic Search

#sqlsatParma


DEMO

#sqlsatParma


New development challenges

None of the following is true:

“Network is reliable”

“Latency is near to zero or irrelevant”

“Bandwidth is unlimited”

“Network is secure”

“Topology doesn’t change”

“Transport cost is irrelevant”

“Network is homogeneous”From Mauro Servienti presentation

http://www.slideshare.net/mauroservienti/in-the-brain-designing-distributed-scalable-and-reliable-systems

http://www.slideshare.net/mauroservienti/in-the-brain-designing-distributed-scalable-and-reliable-systems

#sqlsatParma


Latency and Retries

SQL Server

Long latencies and timeouts

Transient errors

Retry strategy

#sqlsatParma


DEMO

#sqlsatParma


Q&A

Questions?

Email: [email protected]

Slideshare:

http://www.slideshare.net/marcoparenzan

GitHub:

https://www.gihtub.cob/marcoparenzan

Twitter: http://twitter.com/marco_parenzan

mailto:[email protected]

http://www.slideshare.net/marcoparenzan

https://www.gihtub.cob/marcoparenzan

http://twitter.com/marco_parenzan

#sqlsatParma


Thanks!

#sqlsatParma

#sqlsat355

Technology

2014.11.22 Azure for Sql Server Developer - SQLSAT355 Parma