Moving a 100TB Application from Oracle to Microsoft SQL Server

Praveen SrivatsaDirector, Asthrasoft ConsultingMicrosoft Regional Director and MVP

DBI311

Agenda

NMS (Large Load) Case Study Comparison between Oracle and SQL implementationScaling Points and Optimizations

Async operations with Service BrokerDistributing data with Table partitionsData CompressionOther optimization options

Case StudyExisting NMS application

The NMS application collects, logs and analyzes the logs of various digital equipment in the networkUsed for mobile towers, wi-max networks, wifi-networks and data centersIt’s agents collects logs 24x7 from all equipment and pumps the same to a central store.The logs are analyzed for security breaches or attacks as well as to monitor the health of the various equipment.

Existing NMS application

LogManager

1. Collect the data

2. Analyze the data

3. Log and Report

Agent

Agent

Existing NMS application

LogManager

Agent

Agent

0.5 KB/record

2500 records/ agent

10 agents

= 25K r/s(12. 5 MB/s)

25K r/s == 12.5 MB/s

1.5M r/min == 750 MB/min

90M r/hr == 45 GB/hr

2.16B r/day == 1.08 TB/day

Reporting on 100 days of data

Case StudyCurrent architecture with Oracle

RAW TABLES

Reporting Dashboard1. Agents collect data

and write to CSV

2. Oracle BULK Loads data

3. Data is indexed and analyzed

4. Materialized Views are created for reporting

Case StudyServer Configurations

App Server2 Quad Core (8 Cores)32 GB RAM1TB HDDRedhat LINUXPython based App

DB Server2 Quad Core (8 Cores)32 GB RAM1 TB HDDSolaris 10Oracle R2 10G

200 TB External storage on RAID 5 (120 TB usable)

Case StudyKey Challenges

Bulk Data Import is very fast, but the data is not ready for analysis or reporting till indexes/materialized views are builtWith the data inserts going on, the indexing and analysis is really slow – so differed to end of the day when a new table is created (EOD : 13 hrs)Data querying for reporting now has to be done across multiple tables (upto 100 tables with a UNION) adding a lot of overheadLAG from data collection to reporting was anywhere from 13 hrs to 37 hrs

Case StudyMoving to SQL Server

SERVICE BROKER QUEUES

Compressed Tables

Partitioned Tables

WCF

Reporting Dashboard

Case StudyMoving to SQL Server

Collection with data co-relation33 records/sec(2.85 M/day)

WCF

Reporting Dashboard

Case StudyMoving to SQL Server

SERVICE BROKER QUEUES

Collection with differed co-relationBatch of 100 recs25 batches/sec/Q (2500/s)X 10 Queues (25000/s)

2.16 B/dayWCF

Reporting Dashboard

Case StudyMoving to SQL Server

SERVICE BROKER QUEUES

Partitioned Tables

Parallelized data co-relation5 batches (100r)/sec = 500r/sX 5 Readers = 2500 r/sec/QX 10 Q = 25000 r/s

2.16 B/day

WCF

Reporting Dashboard

Case StudyMoving to SQL Server

SERVICE BROKER QUEUES

Compressed Tables

Partitioned TablesRaw data saving due

to compression28%

WCF

Reporting Dashboard

Case StudyServer Configurations

App Server2 Quad Core (8 Cores)32 GB RAM500GB HDDWindows Server 200R2WCF Based CollectorPython App for reporting

DB Server2 Quad Core (8 Cores)64 GB RAM500GB HDDWindows Server 2008R2SQL Server 2008R2

200 TB External storage on RAID 5 (120 TB usable) on Windows Storage Server

Case StudyKey Benefits with SQL Server

Data transfer was continuous and could be parallelized using more queuesQuerying and analysis did not slow down the data collectionMore complex analysis and reporting could be added onto the logsLag between collection and reporting came down to 1hr 12 mins

Case StudyAdditional Benefits with SQL Server

Historical data was compressed, reporting data was uncompressed, but partitionedDB Snapshots every hour provided a shorter window of analysis and reportingCLR Procedure provided advanced rules operations on the data packets

demo

NMS Walkthrough

End to End walkthroughof the simulated app

Scale PointsService Broker Advantages

Is a queued operations like MSMQ or MQSeriesScales for rapid insertsCan create multiple parallel queuesCan use activation to process the queueGives a guaranteed onetime only deliveryCan transmit data across DB instances

OptimizingService Broker

Configure multiple packets per conversationBulk load multiple records into a single messageTransform and bulk insert from SSB XML to tableOptimize packet size of SSB Queue

demo

Service Broker Demo

Optimizing SSB performance

Scale PointsTable Partitions and Compressions

PartitionSeparates logical units of data without changing codeAllows fast archiving with the SWITCH Partition operationAllows partition level management rather than at table

CompressionCompressed 20-35% dataLoading data, backups, traversal over network all benefittedPage-Row level configurations possible

OptimizingTable Partitions and Compressions

PartitionCreate a logical partition switch to parallelize insertsAlign partitions to number of disks and procsAlign indexes to the table partitions

CompressionEnable compression at a page levelPlan table row size in alignment with the page size

demo

Table Partition Demo

Inserting into multiple partitions in parallel

Scale PointsOther tweaks

DB SnapshotsGives a point-in-time read-only view of the data

XML operationsFast and easy XML Bulk operations on SSB Queue data

CLR ProceduresCompiled procedures for co-relation and alerts

Summary

Key benefits of moving to SQL Server from the current Oracle architecture included

A shorter lag for critical reporting (from 13-37hrs to about 75 min)

A scale-out model allowing for even greater scalability

An integrated set of technologies (relational database, queuing technologies, compression options, reporting solutions) to work with

Takeaways

SQL Server leverages a scale out modelIt does this by using parallel operations at multiple levels

SQL Service Broker provides a robust queuing mechanismAsync queues provides a horizontal scale operations even across multiple databases

Table Partitions allows tables to scale to billions of recordsEffective segregation allows single tables to handle large volumes of data effectively

Leveraging SQL Server featuresDB Snapshots provides a point in view of the dataCompression allows us optimize our data storage

Resources

Architecting Service Broker Applicationshttp://technet.microsoft.com/en-us/library/aa964144.aspx

Partitioning Tables and Indexeshttp://msdn.microsoft.com/en-us/library/ms345146

My website, blog and EMailweb : http://www.asthrasoft.comblog : http://studidesk.asthrasoft.com/srivatsapraveenemail : srivatsapraveen@asthrasoft.com

Related Content

DBI307 | Automating database migration to Microsoft SQL Server

DBI315 | Microsoft SQL Server in virtualization and private cloud

DBI330 | Can your BI Solutions scale?

Database Platform (DAT) Resources

Try the new SQL Server Mission Critical BareMetal Hand’s on-Labs

Visit the updated website for SQL Server® Code Name “Denali” on www.microsoft.com/sqlserver and sign to be notified when the next CTP is availableFollow the @SQLServer Twitter account to watch for updates

Visit the SQL Server Product Demo Stations in the DBI Track section of the Expo/TLC Hall. Bring your questions, ideas and conversations!

• Microsoft® SQL Server® Security & Management • Microsoft® SQL Server® Optimization and Scalability• Microsoft® SQL Server® Programmability • Microsoft® SQL Server® Data Warehousing• Microsoft® SQL Server® Mission Critical • Microsoft® SQL Server® Data Integration

Resources

www.microsoft.com/teched

Sessions On-Demand & Community Microsoft Certification & Training Resources

Resources for IT Professionals Resources for Developers

www.microsoft.com/learning

http://microsoft.com/technet http://microsoft.com/msdn

Learning

http://northamerica.msteched.com

Connect. Share. Discuss.

Complete an evaluation on CommNet and enter to win!

Scan the Tag to evaluate this session now on myTech•Ed Mobile