Designing Role-Based Database Systems to Achieve Unlimited Database Scalability

2

Designing Role-Based Database Systems to Achieve Unlimited Database Scalability

Fernando G. GuerreroCEOSolid Quality Mentors

Session Code: DAT451

5

What forced me to prepare this session

The trigger

6

The "unpredictable workload" case

Millions of copies of a win app around the worldAll of them waiting for file safety, before using them

At undefined timesUnlimited number of potential files to checkUnpredictable level of workload at any date and timeUnpredictable geographical distribution of workload

Potentially, millions of requests per hourCan we solve it?

7

The "unreasonable scalability" case

The system already reached its limitsThey need to support workload 50 times higher

Scaling up is not an optionScaling out is not a simple task

They need to support over 15K transactions per second, with textual based data

Can we solve it?

8

Fernando G. GuerreroCEO of Solid Quality Global LLC, USAPresident of the SolidQ subsidiaries around the Worldfguerrero@solidq.com

Microsoft Regional Director for Spain since 2004SQL Server MVP from year 2000 till 2007Usual suspect at many international conferences

Introductions

9

The SolidQ recipe

4 SQL Server MVPs old friends speaking at a conference in Palm SpringsA pleasant dinner at a nice restaurant2 bottles of my favorite Spanish wine1 bottle of a great single Malt WhiskyLots of honest and trustful work for 6 years

10

What are you going to see and hear today

This session

11

This session is about…

Lesson learned from “unreasonable” and amazing real systems from real enterprise customersBased on actual work performed by our Solid Quality mentorsLearning to think out of the box when designing DB systemsThinking first, acting later

12

This session is NOT about…

CodeDemosObscure trace flags and registry hacksAny particular commercial product or service offeringDescription about standard scalability techniques

13

First things First

14

First… Listen

15

Then, try to understand what their needs really are…

And what they really want

16

A database system is just one more tool to help achieving

business goals

17

Your technical common sense is your most powerful tool

18

If you don’t need to buy anything to implemnent a

solution, it does not mean that this is the cheapest solution

19

Role based DB system design

A single box might be sufficientDifferent tasks might imply different rolesRoles might be incompatibleYou’ll never know until you measure it

20

The unpredictable workload case

The Problem

21

Role based DB system design

A single box might be sufficientDifferent tasks might imply different rolesRoles might be incompatibleYou’ll never know until you measure it

22

Massive attack

23

The unpredictable workload case

The Solution

24

A common sense powered queue

Multi-layered server roles:The ReaderThe WriterThe Master Root

“Magic array” of virtual server namesSelf updated definitionsProprietary advanced hash-based checkingLocal vs remote storage

25

The Magic Array

26

The Reader role

Organized as a Readers tree with a Reader Root at the topVerifies signature of requested fileReplies to client with simple response:

SafeUnsafeUnknown

If unknown, checks parent Reader for further informationReceives updates from parent Reader

27

The Writer role

Works as a Reader as wellOrganized as a flexible array of WritersSends signature to the Master Root

and sends file asynchronously if necessaryReplies immediately to client with simple response:

SafeUnsafeUnknown

Receives updates from Distributor

28

The Master Root

Stores new files and signaturesCalls the qualification engine to qualify new filesDistributes updates to root ReaderDistributes updates to array of Writers

29

Massive dynamic deployment

30

The unreasonable scalability case

The problem

31

Scalability needs Jan 2005

10

10090 90

300 300

Workload vs CapacityWorkload Initial C Improved C Series4

32

Scalability needs Aug 2006

10

200

300

90 90 90

300 300

600

Workload vs CapacityWorkload Initial C Improved C

33

Side note: is there such thing as a write-

only transaction?or…

how much writing activity do you really expect?

34

Scalability needs Dec 2008

10 200 300 500

1500

3000

90 90 90 90 90 90600 600 600 600

2000

4000

Workload vs CapacityWorkload Initial CImproved C Role Based C

35

Scalability needs beyond 2009

10 200 3005001500

3000

6000

12000

90 90 90 90 90 90 90 90600600 600 600 600 6002000

4000

8000

16000

Workload vs CapacityWorkload Initial CImproved C Role Based C

36

The unreasonable scalability case

The solution

37

Role-based architecture

38

Detailed role assignment

39

Technology alone cannot beat what an experienced engineer can design on the back of an

envelope

40

Q & A

www.microsoft.com/teched Tech·Talks Tech·Ed BloggersLive Simulcasts Virtual Labs

http://microsoft.com/technet

Evaluation licenses, pre-released products, and MORE!

Resources for IT Professionals

Related Content

Breakout Sessions (session codes and titles)

Interactive Theater Sessions (session codes and titles)

Hands-on Labs (session codes and titles)

Hands-on Labs (session codes and titles)

Track Resources

Resource 1

Resource 2

Resource 3

Resource 4

1 Year Subscription!

Complete anevaluation onCommNet andenter to win!

46

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED

OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.