Database Storage Considerations
Database Storage Considerations
Adam BackmanAdam Backman
White Star SoftwareWhite Star Software
[email protected]@wss.com
DB-05:DB-05: DB-05:DB-05:
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
OutlineOutline
Why is DB storage so important?Why is DB storage so important? Hardware optionsHardware options
RAID vs. JBODRAID vs. JBOD Network Storage (SAN, NAS, …)Network Storage (SAN, NAS, …)
Database setup optionsDatabase setup options Block sizeBlock size Cluster size (DB and BI)Cluster size (DB and BI) Splitting Data into areasSplitting Data into areas
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Why is DB Storage So Important?Why is DB Storage So Important?
Everything starts from the diskEverything starts from the disk Disks need to be reliable and fastDisks need to be reliable and fast All physical disks are only capable of doing All physical disks are only capable of doing
approximately 100 operations per secondapproximately 100 operations per second Salespeople are weaselsSalespeople are weasels
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Who cares about 100 I/O operations per second?Who cares about 100 I/O operations per second?
Example:Example:
10000 read operations10000 read operations
90% buffer hit rate90% buffer hit rate
1 disk would take 10 seconds to complete 1 disk would take 10 seconds to complete
10 disks would take 1 second to complete10 disks would take 1 second to complete
Which of these would your users choose?Which of these would your users choose?
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Stop I/O before it gets to the disksStop I/O before it gets to the disks
Application code Application code Use indexesUse indexes Watch out for can-findWatch out for can-find Bulk updates vs. many small updatesBulk updates vs. many small updates
Use memory not diskUse memory not disk Add system memory (no swapping or paging)Add system memory (no swapping or paging) Increase buffers (-B, -Bt, -Bp)Increase buffers (-B, -Bt, -Bp) Add cache to disk controllersAdd cache to disk controllers
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
A little more about buffer hit rateA little more about buffer hit rate
Using same example: 10000 Logical readsUsing same example: 10000 Logical reads
90% buffer hit rate = 1000 Physical reads90% buffer hit rate = 1000 Physical reads
95% buffer hit rate = 500 Physical reads95% buffer hit rate = 500 Physical reads
96% buffer hit rate = 400 Physical reads96% buffer hit rate = 400 Physical reads
See the effect buffer efficiency has on disks! See the effect buffer efficiency has on disks!
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
What causes disk I/O?What causes disk I/O?
Operating system (swapping and paging)Operating system (swapping and paging) ProgressProgress
Database (DB and BI)Database (DB and BI) Application (code and temp files)Application (code and temp files)
Other applicationsOther applications
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Hardware OptionsHardware Options
RAID vs. JBODRAID vs. JBOD Buying disksBuying disks What is varianceWhat is variance Simplicity and cost vs. control and budget Simplicity and cost vs. control and budget
Network vs. Locally Attached StorageNetwork vs. Locally Attached Storage DefinitionsDefinitions ConfigurationConfiguration Pros and ConsPros and Cons
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Buying DisksBuying Disks
Buy small disksBuy small disksEach disk regardless of it’s size is capable of Each disk regardless of it’s size is capable of
doing the same number of I/Os per seconddoing the same number of I/Os per second Buy fast disksBuy fast disks
Slow disk = slow performanceSlow disk = slow performance Buy reliable disksBuy reliable disks Buy many disksBuy many disks
The inner portion of the disk is 20% faster than The inner portion of the disk is 20% faster than the outer portion of the diskthe outer portion of the disk
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Balancing Disk I/OBalancing Disk I/O
Balancing disk I/O is the process of making Balancing disk I/O is the process of making sure you are using all of the available disk sure you are using all of the available disk resources (file systems, disks and resources (file systems, disks and controllers) are working equally as hard at controllers) are working equally as hard at load. This is also called eliminating load. This is also called eliminating variance. variance. A well tuned system will have less A well tuned system will have less than a 15% variance across all disks.than a 15% variance across all disks.
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
How can I eliminate variance?How can I eliminate variance?
Warning: This is a religious argument. Warning: This is a religious argument. Everyone has an opinion and everyone is Everyone has an opinion and everyone is right.right.
There are two schools of thought.There are two schools of thought. The system should take care of this.The system should take care of this. I will take care of this.I will take care of this.
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
The system will take care of this.The system will take care of this.
Disk I/O can be effectively distributed by the Disk I/O can be effectively distributed by the operating system with the use of smart I/O operating system with the use of smart I/O systems. Most notably RAID. systems. Most notably RAID.
Basic Definition of RAID: Basic Definition of RAID: RRedundant edundant AArray rray of of IIndependent ndependent DDisksisks
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
What RAID really meansWhat RAID really means
RAID has many levels. I will only cover a RAID has many levels. I will only cover a few.few.
RAID 0: This level is also called RAID 0: This level is also called striping.striping. RAID 1: This is referred to as RAID 1: This is referred to as mirroringmirroring.. RAID 5: Most popular RAID levelRAID 5: Most popular RAID level RAID 10: This is mirroring and striping. RAID 10: This is mirroring and striping.
Also known as RAID 0 + 1Also known as RAID 0 + 1
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Network vs. Local StorageNetwork vs. Local Storage
Network storage optionsNetwork storage options Benefits of network storageBenefits of network storage Drawbacks of network storageDrawbacks of network storage
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Network Storage OptionsNetwork Storage Options
SAN (Storage Area Network)SAN (Storage Area Network)A dedicated storage network supporting multiple A dedicated storage network supporting multiple
computerscomputers NAS (Network Attached Storage)NAS (Network Attached Storage)
A dedicated storage device on an existing network A dedicated storage device on an existing network including a SANincluding a SAN
iSCSIiSCSIA storage networking protocol allows storage traffic to A storage networking protocol allows storage traffic to
travel through general-purpose networks instead of the travel through general-purpose networks instead of the specialized Fibre Channel SANspecialized Fibre Channel SAN
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Benefits of Network StorageBenefits of Network Storage
Centralized data store - buy a SAN, hook up Centralized data store - buy a SAN, hook up many serversmany servers
Easier to maintain (backup one place) Easier to maintain (backup one place)
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Drawbacks to Network StorageDrawbacks to Network Storage
More complex performance monitoringMore complex performance monitoringMany servers all making requests at the same time make it Many servers all making requests at the same time make it
difficult to resolve conflictsdifficult to resolve conflicts Overall more complexOverall more complex
More things to breakMore things to break Single point of failure Single point of failure
Many vendors support array to array mirroring to resolve Many vendors support array to array mirroring to resolve this issuethis issue
Generally more expensiveGenerally more expensiveUnless you have a large number of servers using the Unless you have a large number of servers using the
network storagenetwork storage
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Creative Hardware Solution: Solid State DisksCreative Hardware Solution: Solid State Disks
PROsPROs ReliableReliable FastFast
CONsCONs ExpensiveExpensive Really expensiveReally expensive
Conclusion: Good for logs (AI and BI) and Conclusion: Good for logs (AI and BI) and limited portions (hot spots) of the databaselimited portions (hot spots) of the database
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
I will take care of thisI will take care of this
Manual spread of data across disksManual spread of data across disks Better control as you can see where the I/O Better control as you can see where the I/O
is goingis going More attention by system administrator is More attention by system administrator is
neededneeded Administrator must think about reliability Administrator must think about reliability
as wellas well
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Database Setup OptionsDatabase Setup Options
Database StuffDatabase Stuff Storage areasStorage areas Blocksize and Records per blockBlocksize and Records per block Type II areasType II areas
BI cluster sizeBI cluster size Effect of BI cluster sizeEffect of BI cluster size Why are APWs affected by BI Cluster sizeWhy are APWs affected by BI Cluster size
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Storage AreasStorage Areas
BenefitsBenefits Greater control of location of dataGreater control of location of data Minimize downtime for utilitiesMinimize downtime for utilities Stripe some, leave some on straight disksStripe some, leave some on straight disks
DrawbacksDrawbacks More things to breakMore things to break More complex to monitorMore complex to monitor
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Storage Areas - ControlStorage Areas - Control
A Storage Area can hold 1 or more data A Storage Area can hold 1 or more data objects (index, table, schema, …)objects (index, table, schema, …)
Separate schema from data if possibleSeparate schema from data if possible Isolate the logs (AI and BI)Isolate the logs (AI and BI) Try to keep the number of areas Try to keep the number of areas
manageable, only add more areas for valid manageable, only add more areas for valid business reasonsbusiness reasons
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Database Block SizeDatabase Block Size
8k for most (4k for NT)8k for most (4k for NT) More in synch with the operating systemMore in synch with the operating system More stuff per readMore stuff per read Especially good for index readsEspecially good for index reads Watch the number of records per block so Watch the number of records per block so
you don’t waste block spaceyou don’t waste block space
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Records per BlockRecords per Block
Can be set to a binary number between 1 Can be set to a binary number between 1 and 256and 256
If it is set too low you will waste space at If it is set too low you will waste space at the end of each blockthe end of each block
If you set it too high you run the risk of If you set it too high you run the risk of fragmenting records (most true for records fragmenting records (most true for records that grow over time)that grow over time)
More important to get right for type II areasMore important to get right for type II areas
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Setting Records per BlockSetting Records per Block
First do a database analysis.First do a database analysis.proutil <dbname> -C dbanalys > filenameproutil <dbname> -C dbanalys > filename
Then take the mean record size of the table(s) Then take the mean record size of the table(s) you are working with and add 20 bytes for you are working with and add 20 bytes for record and block overhead.record and block overhead.
Divide the result above into your block size.Divide the result above into your block size.Last take a binary number just above the Last take a binary number just above the
result above.result above.
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Example: Setting Records/BlockExample: Setting Records/Block
Table 1 has a mean record size of 120 bytesTable 1 has a mean record size of 120 bytes Add 20 bytes of overhead (140)Add 20 bytes of overhead (140) Divide block size (8192) by result (140)Divide block size (8192) by result (140)
This (58.51) is how many mean-sized records This (58.51) is how many mean-sized records will fit in a single database block will fit in a single database block
Take the next higher binary number (64)Take the next higher binary number (64)
Are there exceptions, you bet!Are there exceptions, you bet!
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Type II AreasType II Areas
The table and index blocks from individual The table and index blocks from individual objects are clustered togetherobjects are clustered together
The cluster size is “tunable” to 8, 64 or 512 The cluster size is “tunable” to 8, 64 or 512 block clustersblock clusters
Benefits: Potential performance benefitsBenefits: Potential performance benefits Drawbacks: Yet another thing to try to get Drawbacks: Yet another thing to try to get
right, potential wasted space right, potential wasted space
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
How Type II Areas WorkHow Type II Areas Work
The data blocks are clustered together and in turn The data blocks are clustered together and in turn these clusters are chained togetherthese clusters are chained together
Scan type operations need only find the first block Scan type operations need only find the first block and the rest of the blocks will followand the rest of the blocks will follow
Example: When you do an index rebuild, the first Example: When you do an index rebuild, the first step is to delete all of the index blocks. In a type 1 step is to delete all of the index blocks. In a type 1 area all of the blocks in the area are scanned and area all of the blocks in the area are scanned and the index blocks are deleted. In a type II area the the index blocks are deleted. In a type II area the first cluster is put on the free chain and no other first cluster is put on the free chain and no other updates are needed.updates are needed.
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Why are APWs Affected by BI Cluster SizeWhy are APWs Affected by BI Cluster Size
The BI cluster size determines the length of The BI cluster size determines the length of the checkpointthe checkpoint
The APW efficiency determines the The APW efficiency determines the efficiency of the checkpointefficiency of the checkpoint
Generally, I start with 1 apw and the Generally, I start with 1 apw and the “correct” BI cluster size and then add 1 apw “correct” BI cluster size and then add 1 apw if I see buffers flushed at checkpoint during if I see buffers flushed at checkpoint during my “important” processing times.my “important” processing times.
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
BI Cluster SizeBI Cluster Size
The default value of 512 KB is wrong for The default value of 512 KB is wrong for most peoplemost people
Monitor the DB during your high update Monitor the DB during your high update portion of the dayportion of the day
Set the cluster size high enough to keep Set the cluster size high enough to keep checkpoint lengths greater than 2 minutescheckpoint lengths greater than 2 minutes
Generally, 1 MB to 8 MB for most sitesGenerally, 1 MB to 8 MB for most sites
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
ConclusionConclusion
Look at your storageLook at your storage No RAID 5No RAID 5 RAID or JBODRAID or JBOD To network or not to networkTo network or not to network
Get your DB setup rightGet your DB setup right Block size (remember records per block)Block size (remember records per block) Rational split of your data into storage areasRational split of your data into storage areas Type II areas (where needed)Type II areas (where needed) Get your BI cluster size rightGet your BI cluster size right
© 2005 White Star Software© 2005 White Star SoftwareDB-05 Database Storage Considerations
Use Both MethodsUse Both Methods
Use mirroring to protect your dataUse mirroring to protect your data Use stripping to get your data onto more Use stripping to get your data onto more
physical drivesphysical drives Use storage areas and extents eliminate Use storage areas and extents eliminate
variancevariance Use storage areas and extents to further Use storage areas and extents to further
spread your dataspread your data