ISV Proposition

Keith Burns Data Architect Microsoft UKkeith.burns@microsoft.com

Transparent Data EncryptionExternal Key ManagementData AuditingPluggable CPUTransparent Failover for Database MirroringPolicy ManagementServer Group ManagementStreamlined InstallationEnterprise System ManagementPerformance Data CollectionSystem AnalysisData CompressionQuery Optimization ModesResource GovernorEntity Data ModelLINQVisual Entity DesignerEntity Aware Adapters

SQL Server Change TrackingSynchronized Programming ModelVisual Studio SupportSQL Server Conflict DetectionFILESTREAM data typeIntegrated Full Text SearchSparse ColumnsLarge User Defined TypesDate/Time Data TypeLOCATION data typeSPATIAL data typeVirtual Earth IntegrationPartitioned Table ParallelismQuery OptimizationsPersistent Lookups Change Data Capture Backup CompressionMERGE SQL Statement

Data ProfilingStar JoinEnterprise Reporting EngineInternet Report DeploymentBlock ComputationsScale out AnalysisBI Platform ManagementExport to Word and ExcelAuthor reports in Word and ExcelReport Builder EnhancementsTABLIXRich Formatted DataPersonalized Perspectives… and many more

Top ISV Features in SQL Server 2008

1. Lock Escalation2. Plan Freezing3. Optimise for Ad-hoc Workloads Option4. Resource Governor5. Logging enhancements6. Sparse Data and Indexes7. Hierarchy Datatype8. Spatial Data9. Filestream10. Security (Auditing & Transparent Database

Encryption (TDE)

Lock EscalationPain Points• Lock escalation causes performance problems

(blocking); cannot be controlled on per table basis• Traceflags -T1211 & -T1224 used to disable lock

escalation can help, but often result in other problems

SQL Server 2008• disable lock escalation at a table level ALTER TABLE TabA SET (LOCK_ESCALATION = DISABLE);

• For partitioned tables, results in lock being escalated to the partition, instead of to the table level as in SQL Server 2005

Plan FreezingPain PointsDifficult to tune query plan when T-SQL code not accessibleNo easy way of ensuring plan consistency across upgradesSQL Server 2005 introduced Plan Guides and USE PLAN hint; but these were difficult to use

SQL Server 2008• Plan Freezing builds on plan guides framework and

introduces easier creation process based on plan cache entries (sp_create_plan_guide_from_cache)

• Full DML support INSERT, DELETE, UPDATE supported• New function sys.fn_validate_plan_guide introduced

to validate existing SQL 2005 plan guides

Optimize for Ad hoc Workloads OptionPain Points• Single use ad hoc batches can flood the procedure cache• Using Forced Parameterization option for such scenarios

can result in adverse side-effects

SQL Server 2008• New ‘optimize for ad hoc workloads’ server option• Set using sp_configure, or SQL Server Management

Studio• Stores stub for adhoc compiled plan after first execution,

replaces with actual query plan if the query is re-executed

• Improves efficiency of plan cache

Resource GovernorSQL Server 2008RG is designed for workload balance in database engineConfigurable at connection level; Online change of group/pool allowedDefinition of group/pool transparent to applicationUsers can control server level setting (MAXDOP) at workgroup levelPrevent run-away queries

SQL Server

Min Memory 10%Max Memory 20%

Max CPU 20%

Admin Workload

Backup

Admin Tasks

OLTP Workload

OLTP Activity

Report Workload

Ad-hocReports

ExecutiveReports

High

Max CPU 90%

Application PoolAdmin Pool

Connections

Classifier Function

Logging EnhancementsSQL Server 2008

• INSERT into table supports minimal logging• 3X-5X Performance Boost over fully logged INSERT

Index Insert

Heap Insert

SQL Server 2008SQL Server

Run Time

Sparse Column Storage

ID Column Value

1 Q1 C

1 Q2 1

1 Q10 9

2 Q1 B

2 Q3 4

2 Q5 Low

3 Q1 C

3 Q7 6

3 Q8 5

PK Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10

1 C 1 9

2 B 4 Low

3 C 6 5

4 1 5

5 E High

Blue

6 C 8 9

7 5 7

8 A 2 Red

9 A 3 6

Desired schema Typical Solution

The problem• Need to store sparse data• Possibly 100’s of columns• Typically only few % are populated

Sparse Columns

“Sparse” as a storage attribute on a column• 0 bytes for a NULL, 4 byte overhead for non-NULL• No change in Query/DML behavior• Same limitations as normal tables eg 1024 columnsWide Table / defining a “Sparse Column Set”•Column set columns can still be individually specified in SQL statements•Select * returns all non-sparse-columns + sparse column set as XML•30,000 sparse columns allowed in a table (2Gb), 1000 indexes

// Sparse as a storage attibute in Create/Alter table statementsCreate Table Products(Id int, Type nvarchar(16)…,

Resolution int SPARSE, ZoomLength int SPARSE);// Create a sparse column setCreate Table Products(Id int, Type nvarchar(16)…,

Resolution int SPARSE, ZoomLength int SPARSE, Properties XML COLUMN_SET FOR ALL_SPARSE_COLUMNS);

Filtered IndexesFiltered Indexes and Statistics• Indexing a portion of the data in a table• Filtered/co-related statistics creation and usage• Query/DML Optimization to use Filtered indexes and Statistics• Restricted to non-clustered indexes

Benefits• Lower storage and maintenance costs for large number of indexes• Query/DML Performance Benefits: IO only for qualifying rows

// Create a Filtered IndexesCreate Index ZoomIdx on Products(ZoomLength) where Type = ‘Camera’;

// Optimizer will pick the filtered index when query predicates matchSelect ProductId, Type, Resolution, ZoomLength where Type = ‘Camera’’

HierarchyID• Key properties• Logically encodes the path from

the root of the tree to a node • Rich built-in methods for

manipulating hierarchies• Simplifies storage and querying of

hierarchical data• Comparison a<b is in depth-first

order• Support for arbitrary insertions

and deletions

• Potential Applications• Forum and mailing list threads• Business organization charts• Content management categories• Product categories • Files/folders management

/

/1/

/1/1/

/1/1/1/

/1/1/2/

/1/2/

/2/ /3/

/3/1/ /3/2/

Spatial Data Overview• Storage and retrieval of spatial data using standard

SQL syntax– New Spatial Data Types (geometry, geography)– New Spatial Methods (intersects, buffer, etc.)– New Spatial Indexes

• Offers full set of Open Geospatial Consortium components (OGC/SQL MM, ISO 19125)

• Integration with Virtual Earth

FILESTREAMStorage Attribute on VARBINARY(MAX)

• Works with integrated FTSUnstructured data stored directly in the

file system (requires NTFS)Dual Programming Model

• TSQL (Same as SQL BLOB)• Win32 Streaming APIs with T-SQL

transactional semanticsData ConsistencyIntegrated Manageability

• Back Up / Restore• Administration

Size limit is the file system volume sizeSQL Server Security Stack

Store BLOBs in DB + File System

Application

BLOB

DB

Enterprise security and scalability

Data consistency across heterogeneous systems

High-scale, complex event processing

TRUSTED, SCALABLE PLATFORM

IT & DEVELOPER EFFICIENCY

MANAGED SELF-SERVICE BI

Multi-server management

Virtualization & Live Migration

Accelerated development& deployment

Self-service analytics

Self-service reporting

Streamlined collaboration

& management

Data Warehouse Edition

Master Data Services

Stream Insight

PowerPivot /SharePoint 2010

Report Builder 3.0

Data Tier Applications

Master Data Management

VERSIONING

WORKFLOW APPROVAL

ROLE SECURITY

HIERARCHY MGMT

CRMHR DocERP Asset Mgmt

Purchasing DB

Relational Database Applications

Financial trading Applications

Stream InsightScenarios for Event Processing

Aggregate Data Rate (Events/sec.)

Latency

0 10 100 1000 10000 100000 ~1million

Months

Days

hours

Minutes

Seconds

100 ms

< 1ms

Operational Analytics Applications, e.g., Logistics, etc.

Manufacturing ApplicationsMonitoring Applications

CEP Target Scenarios

Data Warehousing Applications

Web Analytics Applications

Stream Insight

18

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarativequery and pattern specifications with near-zero latency.

Database Applications Event-driven Applications

Query Paradigm

Ad-hoc queries or requests

Continuous standing queries

Latency Seconds, hours, days Milliseconds or less

Data Rate Hundreds of events/sec Tens of thousands of events/sec or more

request

response

Eventoutput streaminput

stream

StreamInsight

Increasing Availability

ClusterNode 1

Network Storage

ClusterNode 2

CONFIGURATION DATA

Virtual Machine

MEMORY CONTENT

VHD

MEMORY SYNC

Virtual Machine

User

Live Migration

Additional SQL Server 2008 R2 report featureso Mapping supporto Report Builder 3.0o Report Partso Export to data feed

Summary

SQL Server 2008 has many features designed specifically for ISV workloads; zero or minimal application change required

There are literally 100s of new and improved features – this session only highlights some of the ones most popular with ISVs

Understanding what problem each feature resolves as well as it’s performance, insights and limitations helps more appropriate usage