Embed Size (px)
Citation preview
SQL CAT (Customer Advisory Team)
Achieving Customer SuccessLG Electronics - is a manufacturing corporation with more than 115 businesses worldwide and more than 80,000 employees, Improved transaction performance of 5TB-sized BI system ‘from minutes to seconds’, KRW 90billion reduction in costs
Case study coming soon
Yahoo!- Tier-1 Mission Critical BI solution, 12TB Analysis Services database, 3.5 billion display advertising impressions, 35 billion segments.
Case study coming soon
Temenos – Tier-1 core banking ISV; 1TB DB, 100k batch requests/sechttp://www.microsoft.com/presspass/press/2010/sep10/09-01temenospr.mspx
Making a Better ProductDrives feedback and product requirements back into SQL Server development teams from deep and strategic customer and ISV engagements
Sharing with the Communityhttp://sqlcat.com//missioncritical – field resources
The SQL Server Customer Advisory Team (SQL CAT) represents the customer-facing resources from the SQL Server Product Group. SQL CAT is comprised of product and solution experts that regularly engage in the largest, most complex, and most unique customer deployments worldwide.
Customer BenefitsDirect engagement from product group (SQL CAT technical/project experience)Reduced riskArchitecture Best Practices and Guidance SQL CAT-led service offerings (i.e. Architecture Design Reviews) optionalLab visit to Redmond or Regional Microsoft Technology Center (MTC) optionalBusiness Investment Funds (BIF) optional/limited availability!Invitations to special events/programs (i.e. TAP) optionalCo-marketing and PR
Example (SQL Server Qualifying Criteria)10+ TB DW, 3k tx/s OLTP, Large 500GB+ Cubes, Competitive migrations, Complex deployments, Server Consolidation (1000+)
Nominating a CustomerComplete a nomination form: http://sqlserver/customers/customeradvisoryteam/default.aspx Submit completed form to the Design Win Program alias “dsignwin”
The Customer Experience (CX) Design Wins Program is how the Microsoft Business Platform Division identifies and invests in strategic and large-scale customer projects with challenging, unique, and complex applications running on the Microsoft platform.
Design Wins Program
Session Objectives and Takeaways
ObjectivesLearn about the largest custom SQL Server projectsSee the techniques they use to scaleSee how they do HA and DR
TakeawaysSQL will scale to handle any size projectSQL is enterprise ready
Top statisticsCategory MetricLargest single database 70 TBLargest table 20 TBBiggest total data 1 application 88 PBHighest database transactions per second 1 db (from Perfmon)
130,000
Fastest I/O subsystem in production (SQLIO 64k buffer)
18 GB/sec
Fastest “real time” cube Millisecond latency
data load for 1TB 30 minutesLargest cube 12 TB
Mission Data High-Volume VLDB Critical HA/DR Warehouse OLTP
> 30,000 DB tx/Second
100% uptime (2008)100% uptime (2009)
180M incremental fact rows/day
7x24x365 Synchronous Mirroring Solution >15B tx/yr
> peta-byte≈ 1 trillion rows
Asynch Mirroring > 400 miles< 60 sec. recovery
>1 peta-byte
>10B rows in 1 table
>80GB daily growth
>50 TB by end of 2010
>90GB daily growthCritical operationsat > 2200 facilities
Critical operationswith > 15,000 users
Mission Critical database > 4 TB
Critical operationswith > 1,000,000,000 tx/day
Large U.S. Financial Organization
Mission Critical table > 1.9 billion rows
COMING SOON!(10+1 Clusters)
Centipede
490TB9 data nodes
70GB daily growth<= 3s query response (80%)
700 million rows/dayIncremental growth
5TB Analysis Services data50TB federated environment
> 3,000 tx/Second> 2B CLR calculations/day
12.4M users (2.7M regular)> 23k batch requests/sec
> 125K tx/second< 50ms latency
Large U.S. Financial Organization
4.4M concurrent users130M monthly users
SAP | geo-cluster > 5 milesNo Data Loss | No Down Time
SQL Server Proof Points
Scaling techniques and features
CompressionTable PartitioningPartitioned ViewsService BrokerSODA – Services Oriented Database ArchitectureDDR – Data Dependent Routing
Scale up or Scale out
SQL Server Case Studies
DATA WAREHOUSE
Telecom
CDR Analytics70TB Relational4TB largest cube100+ concurrent queries64 core with storage system rated over 18GB/sec throughputLoading 1TB in < 30 minutesProcessing 1m rec/sec in AS cubes
Hilton HotelsRoom forecasting systemFull suite of SQL products (SQL, AS, IS, RS)Scale out AS and RS
If you need more capacity, just add another serverLoad Balanced Analysis Services reader machines40 to 50 concurrent users per RS server
Complex queriesLarge data sets returned to many clients
IBM xSeries and IBM Blade Center serversCase study: http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=49192
Hilton Architecture
Stein Mart
First Fast Track case studySaved $50,000 / month after AS/400 migrationFaster results – 3 hours of processing instead of 14 hoursLess people to maintainUsers love the new tools!4 TB data warehousehttp://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000007013
SQL Server Case Studies
OLTP
MySpace
400+ SQL Servers 1,300 databasesTotal data managed > 2 PBData Dependent Routing, Distributed Partitioned Views, Replication, SODACurrently moving to Windows 2008 R2/ SQL 2008 R2Great article on architecture changes as they grew: http://www.baselinemag.com/article2/0,1540,2082921,00.asp
MySpaceSolution
Major Architectural Changes during rapid growth500,000 Users: A Simple Architecture plan (two Web servers talking to a single database server – 3 database servers (1 write, 2 read using replication))1 Million Users: Functional Partitioning Solves Scalability Woes (separate databases for parts of the Web site that served different business and user functions), SODA3 Million Users: Scale-Out Wins Over Scale-Up (cost for scale-up too high), Data Dependent Routing (DDR)9 Million Users: Site Migrates to ASP.NET, Adds Caching Tier26 Million Users: MySpace Embraces 64-Bit Technology, SQL Server 2008, Service Broker
MySpace (cont)
High Availability45 SQL Server Failover Clusters 7+1Unattended patching via custom Powershell scripts and management control screenExtensive testing before patchingContinually improving operations process
Disaster Recovery: 2 data centersDoing SAN snapshots between sitesIn case of loss of one site, can bring up other site to take over within a few hours.
Large Health Care Provider
17 TB OLTP database10,000 user connectionsReplication for reporting databaseSAN hardware replication used for Disaster RecoverySQL Server 2008Scale up
Securities Trading Organization Application: “Real-time” high transaction, low latency stock quoting
Over 280,000 business transactions/secOver 1 million data manipulation calls per second in single databaseLatency per business transaction under 50 microseconds. “Real-time” nature of data flow.
Workload Characteristics:Send large batch containing multiple business transactions, parse and end up inserting all records into a large table which is constantly readUnder 10 tables in the application.
Securities Trading Organization (2 of 2)Hardware/Deployment Configuration:
Load distributed based on alphabetical split. Co-operative Scale-out. Commodity based hardware (4-socket, quad-core) and high performance SAN.
Other Solution Requirements:Mission critical to the business in terms of performance and availability. Require 99.999% uptime overall and 100% during business day.Treat system like their mainframe operationsUtilize SQL Server HA features to help support 5 9’s uptime requirement and geographical redundancy
SQL Server Failover Clustering for local (within datacenter) availabilityDatabase Mirroring (High Availability/Async) for geo-availability
Distance 250 miles30MB/sec log generation with no-redo queue
• World’s biggest publicly listed online gaming platform
• World’s biggest publicly listed online gaming platform
• World’s leading provider of online Sports Betting
• One of the largest Poker networks
• Comprehensive range of Payment Service Providing
• Integrated gaming portal - 22 languages, 25 core markets
• More than 20 million registered customers
• 1,500 employees
• bwin builds on the strengths of the web in order to tie responsibility and gaming
• 15 million page views and up to 980,000 users a day
• Environment• 5 DBA’s & 1 Database Engineer• 100+ SQL Server Instances• 120+ TB of data, • 1,400+ Databases• 1,600+ TB storage • 5,000+ GB RAM• 450,000+ SQL Statements per second on a single server• 500+ Billion database transactions per day
• Scale UP and Scale OUT
• Scale UP main financial transactions
• Scale OUT other application functions
• High Availability
• 2 data centers
• Synchronous Database Mirroring adds 1 ms per transaction
• Replication for reporting
• Log Shipping for DR• Backup 2 TB per hour over the network
http://sqlcat.com/whitepapers/archive/2009/08/13/a-technical-case-study-fast-and-reliable-backup-and-restore-of-a-vldb-over-the-network.aspx
• Case studyhttp://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000001470
Retail Application
1,200 stores SQL Standard10 GB average
15,000 cash registers SQL Express10 MB database average
1 Corporate Server SQL EnterpriseWindows Cluster + DB Mirroring + Log ShipSQL Server, AS, IS, RS12+ TB database, 1.5 TB cube.
Retail Application (cont)Merge replication for products & pricingService Broker for all transactions
One of the largest Service Broker and replication projects in the world
2 Million SKUs (products)25 Million Accounts10 million transactions / day
Temenos - Banking ISVDB size of the benchmark is about 800GBNEC Express 5800 64 core to host SQL Server. App servers are scale-out could be consolidated on few high end machines like DL980 or spread out on lower cost servers. Transactions/sec 5203 tps –business transactions. SQL Server transactions are much lower as they bundle multiple biz trans in single database tran. Clustering used for high availability.Log Shipping for Disaster Recovery. Repl is used in some deployments as way of reporting
GeoSpatial HA/DR and Performance
Agoda.com , an international web travel company, has deployed a number of SQL Servers in Asia and USAThe main problem was the high availability of data critical to the main applicationThey were unsure which SQL Server high availability/disaster recovery technology would address their problem plus provide a strategic platform for the futureThe core database had approximately 500 tables (with a combination of cascading update/delete constraints and after update/insert table triggers) and over 3,000 stored procedures
SQL HA/DR Technologies Review
Clustering While failover clustering could have been utilized to ensure high-availability within the data centre, it was not used due to the additional hardware cost and the company’s requirement for multi master nodes
Database Mirroring and Log ShippingAlthough both technologies could have been used , it was the customer requirement that all servers be utilized as master databases. Currently, mirrored and secondary database cannot be written into
SQL HA/DR Technologies Review
PeerToPeer (P2P) Replication benefitsUtilize all the existing serversMinimize downtime and business impactEnable all servers to be utilized in a multi-master fashionRedundancy of key tablesA single site can bring down one server for maintenance without affecting user experience
Final topology
Asia Core 1
Asia Core 2
U.S. Core 1
U.S Core 2
P2P Reference
P2PFinancial
P2PReference
P2P Financial
P2P Reference
P2PFinancial
Asia Web
DW
Read Only
U.S. Web U.S. WebU.S. Web
Web Publication
P2PReference
P2PFinancial
P2P Financial
P2PFinancial
P2P Reference
P2P Reference
P2P Financial
P2PReference
Asia Web Asia Web
P2P Financial
P2PReference
Asia Web Asia Web
Web Publication
Web Publication
U.S. Web U.S. Web
Web Publication
P2P Best Practices
We were able to achieve the best P2P performance by using:
Stored procedures with pull as delivery methodUse WAN Accelerator (if you have slow WAN)Windows 2008 R2 and SQL 2008 R2Have 2 nodes in the same data center to offload/balance transaction distribution
For more details, please look here in www.sqlcat.com.
How do they achieve this?
HA/DR
SQL Server Failover ClusterDatabase MirroringLog ShippingPeer to Peer Replication
Scale
Scale UPScale OUTPeer to Peer Replication
HA Technology Review
No Data Loss(RPO=0)
Failover Unit AutoFailover(RTO)
Inst DB Tab
+ **
Read Mult-iple
Write
*
*
*
Solutions
Log Shipping
DBM Sync
Async
Cluster
Transactional Replication
Peer-PeerReplication
RPO FailoverRedundancy and
Utilization
Hard-ware
App PerfImpact
Manag-eability
Low Low Low
Low High Low
Low Low Low
High*** Low *** Low***
Low Low High
Low Low High
Cost
* Database Mirroring and Log Shipping can provide point in time read capability using STANDBY or database snapshots respectively** Database Mirroring provides fastest failover to hot secondary*** Depends on SAN technology
How do these projects scale?
Scale UPTable PartitioningNUMA Worker Threads
Scale OutTable PartitioningPartitioned ViewsSODAData Dependent Routing
Core System Components
Disk Subsystem
Server
NIC
Memory
Network1
53
4
2
SQL File Layout
HBA
The key is to build a Balanced System without bottlenecks
SQL Server is only part of the equation. Eco system needs to scale.
Memory
LP 0 LP 1 LP 2 LP 3 LP 4 LP 5 LP 6 LP 7
Concepts - NUMA
Front side bus contention increases w/ higher #CPUs
Symmetric Multiprocessor Architecture
Memory
LP 0
LP 1
LP 2
LP 3
Memory
LP 4
LP 5
LP 6
LP 7
Non-Uniform Memory Access
Local Memory AccessForeign MemoryAccess
Foreign memory access > local memory access
> 64 thread supportexploits NUMA
Scaling UP
Windows Server 2008 R2 the limit is 1024 Cores
New concept: Kernel GroupsA bit like NUMA, but an extra layer in the hierarchy
SQL Server 2008 R2 the limit is 256 coresCurrently, largest x64 machine is 96 CoresAnd largest IA-64 is 128 cores (256 Hyperthread)
2 TB RAM
And it Looks Like This...
Scaling Out SQL Server
How do you:Manage an 80 TB databaseBack up a petabyteBuild an index on a trillion row table
Scaling Out SQL Server
How do you:
Manage an 80 TB databaseBack up a petabyteBuild an index on a trillion row table
Answer
Break it into manageable size pieces
Scalability Feature: Table Partitioning
Scalability Feature: Local Partitioned View
Scalability Feature: Cross Database Partitioned View
PDW Appliance
Database Servers
Control Nodes
Active / Passive
Landing Zone
Backup Node
Storage Nodes
Spare Database Server
Du
al
Fib
er
Ch
an
nel
SQL
SQL
SQL
SQL
SQL
SQL
SQL
SQL
SQL
Management Servers
Compute Rack Control Rack
Compute Nodes
SQL
SQL
SQL
Du
al
Infi
nib
an
d
Example Based on HP Architecture
Compute Nodes:HP DL380 G6HP MSA
Scalability Concept: SODA – Services Oriented Database Architecture
Separate your data by business functionExample: HR, Payroll, Accounting, etc
Or by user functionExample: Login, chat, email, pictures, etc
Each function goes in a different databaseA common database for shared tables
Scalability Concept: Data Dependent Routing
Database Configuration Research Project
announcing
Contact DTJones & IsaacK for details
Summary
SQL Server will scale to handle any size you need
Database sizeTransactions per secondConcurrent users
SQL Server is enterprise ready
Please Complete An Evaluation FormYour input is important!
Multiple ways to access Online Evaluation Forms:
CommNet stations located throughout conference venuesVia a Windows mobile deviceVia the CommNet Windows Phone 7 Evaluation and Session Scheduling applicationFrom any wired or wireless connection to https://www.MyTechReady.com
For more information please refer to your Pocket Guide
Speaker – Click Hereto Launch Video
1.2.3.
4.
© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to
be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
