Storage Lecture Notes

1

Storage Devices

Why is storage important?

• Web 2.0 applications are an extension of your Desktop

• SaaS (Software as a service) is here and growing

• Broadband is a reality

• Storage costs are dropping

• Everyone expects near-unlimited storage online – Youtube, Flickr, Facebook et al are storing your life online*

• (.. And yea … let’s not forget your personal bit-torrent collection)

* it would take 1400 TB to store your entire life in video. 5700 TB if you want to know what was happening around you. Another 73 TB for the audio files of everything you heard (MP3 quality). That’s about 6000 TB for a copy of your life

Agenda:

• Hard disks

SATA ,SAS,FC (Fiber Channel), Solid state

• RAID

• DAS

• SAN

• NAS

SATA (Serial advanced Technology Attachment):

• It is a computer bus interface for connecting host bus adapter (connect the host to other n/w & storage devices) to mass storage devices such as hard drives and optical drives.

• Advantage

reduce the cable-bulk and cost

Faster and more efficient data transfer

2

SAS (Serial Attached SCSI):

• It is a computer bus used to move data to and from computer storage devices such as

hard drives & tape drives

SCSI (Small Computer System Interface):

• It is a set of standards for physically connecting and transforming data between computers and peripheral devices.

FC (Fiber Channel):

• It is a gigabit Speed n/w technology primarily used for storage networking.

primarily – supercomputer

Now - SAN (Standard connection type)

Choosing your Hard Disk

(SATA, FC, SAS, SCSI, Solidstate)

Introduction to Hard Drives:

• Basic physical storage unit (aka Physical block device)

• Variables to consider when selecting a drive

Type (SAS, SATA, FC)

RPM (revolutions per minute)

Capacity

MTBF (Mean Time between Failures)

Life Expectancy

3

SATA

(Serial ATA)

SAS

(Serial Attached SCSI)

FC

(Fibre Channel)

Typical Use low-cost, high-volume, low-speed, large-storage environments

CDP / Backups

Replacement for SCSI

High performance transaction oriented applications with high IOPs requirement

High performance transaction oriented applications with high IOPs requirement

Performance Average

Typically 7200 RPM

Good (Similar to FC)

10k / 15k RPM

Good (Similar to SAS)

10k / 15k RPM

Hard drive capacities

Typically - 250 GB, 500 GB, 750 GB, 1TB

Typically – 73 GB, 146 GB, 300 GB, 400 GB

Typically – 73 GB, 146 GB, 300 GB, 400 GB

Hard Disk types:

SATA

(Serial ATA)

SAS

(Serial Attached SCSI)

FC

(Fibre Channel)

Price per Gig

(based on max drive capacity retail web price)

$ 0.33 $2 $3

Miscellaneous - • Backward compatible with SATA

• Allows mixing SATA drives on same backplane

-

Hard Disk Conclusions:

4

• For high IOPs, database applications, low-storage requirements – you have a choice between FC and SAS

• SAS currently seems like the better option

• Future SAS standards promise to be faster than FC (though it is likely they may remain neck to neck)

• For high-storage requirements (video server, file servers, photo storage, archivals, mail servers, backup servers) SATA is the way to go

• One may combine SAS and SATA to reduce average cost and achieve your goals – especially since the backplanes are cross-compatible

• Readup the spec sheet of the hard drives you plan on using for determining specifics

Solid State Drives:

• Uses solid state memory to store persistent data

• Eliminates mechanical parts

• Useful for creating efficient in-between caches or storing small to mid-sized high performance databases

5

Solid State Drives:

Advantages Disadvantages

• Faster startup – no spinning

• Significantly faster on Random IO (From 250x to 1000x+)

• Extremely low latency (25x to 200x better)

• No noise

• Lower power consumption

• Lesser heat production

• Significantly more expensive ($10-30/GB for Flash based, $100-200/GB for DDR RAM based)

• Slightly slower on large sequential reads

• Slower random write speeds incase of Flash based storage

RAID Primer

redundant array of inexpensive disks or redundant array of independent disks

(0, 1, 2, 3, 4, 5, 6, TP, 0+1, 10, 50, 60)

Introduction to RAID:

• allows multiple disks to appear as a single contiguous physical block device

• This uses a pool of disks to save data. Rather than spend billions building special high-capacity disks, greater capacity is achieved by simply putting PC disks into RAIDs.

• Typically we need a RAID controller on the host

• provides redundancy / high availability

• A raid group appears as a single physical block device

• There are several types/levels of RAID

RAID 0:

6

• RAID 0 writes blocks to multiple disks without redundancy

• Because the data is being written to multiple disks the controller can work in parallel on both read and write, improving performance

• If any error occurs data can be lost

• Don’t use on mission critical data; only for performance

• Ideally you have one drive per controller

RAID 1:

• This is mirroring. The same data is written to two disks. If either disk fails a complete copy of the data is available at the other disk

• Uses 2X the storage space, can get better performance because the OS can pick the disk with the least seek or rotational latency

RAID-5:

7

• RAID5 uses “parity” or redundant information. If a block fails, enough parity information is available to recover the data

• The parity information is spread across all the disks

• High read rate, medium write rate

• A disk failure requires a rebuild as the parity information is used to re-create the data lost

RAID-10:

• RAID-10 is striping plus mirroring, so you get good performance plus a fully mirrored data, at the expense of 2X disk

Storage:

• RAID-5 is a reasonable choice most of the time.

• There are many commodity vendors of RAID arrays

• SCSI RAID arrays are expensive, the disks are expensive, and the disks have low capacity, but the RAID arrays have good performance

• ATA raid arrays have excellent price (1/3-1/2 that of SCSI drives) and capacity, somewhat lower performance

• Apple ATA RAID: 7 TB, $11.5K

• Promise Vtrak 15110: $4K plus 15 400GB SATA disks at $300 = 6 TB for $8,500

Comparison of Single RAID Levels:

8

RAID 0 RAID 1 RAID 5 RAID 6

Diagram

Description Striping Mirroring Striping with Parity Striping with Dual Parity

Minimum Disks

2 2 3 4

Maximum Disks

Controller Dependant

2 Controller Dependant

Controller Dependant

Array Capacity

No. of Drives x Drive Capacity

Drive Capacity

(No. of Drives - 1) x Drive Capacity

(No. of Drives - 2) x Drive Capacity

RAID 0

RAID 1 RAID 5 RAID 6

Storage Efficiency 100% 50% (Num of drives – 1) / Num of drives

(Num of drives – 2) / Num of drives

Fault Tolerance None 1 Drive failure 1 Drive failure 2 Drive failures

High Availability None Good Good Very Good

Degradation during rebuild

NA Slight degradation

Rebuilds very fast

High degradation

Slow Rebuild

(due to write penalty of parity)

Very High degradation

Very Slow Rebuild

(due to write penalty of dual parity)


Random Read Very Good Very Good Very Good

9

Performance Good

Random Write Performance

Very Good

Good (slightly worse than single drive)

Fair (Parity overhead)

Poor (Dual Parity Overhead)

Sequential Read Performance

Very Good

Fair Good Good

Sequential Write Performance

Very Good

Good Fair Fair

Cost Lowest High Moderate Moderate+


Use Case

Non critical data

High speed requirements

Data backed up elsewhere

Typically used as RAID 10 in OLTP / OLAP applications

Non-write intensive OLTP applications / file servers etc

Non-write intensive OLTP applications / file servers etc

Misc - - Parity can considerably slow down system

Not supported on all RAID cards

Comparison of Nested RAID Levels:

RAID 10 RAID 50

10

Diagram

Description Mirroring then Striping Striping with Parity then Striping without parity

Minimum Disks

Even number > 4 > 6

Maximum Disks

Controller Dependant Controller Dependant

Array Capacity

(Size of Drive) * (Number of Drives ) / 2

(Size of Drive) * (No. of Drives In Each RAID 5 Set - 1) * (No of RAID 5 Sets)

RAID 10 RAID 50

Storage Efficiency 50% ((No. of Drives In Each RAID 5 Set - 1) / No. of Drives In Each RAID 5 Set)

Fault Tolerance Multiple drive failure as long as 2 drives from same RAID 1 set do not fail

Multiple drive failure as long as 2 drives from same RAID 5 set do not fail

High Availability Excellent Excellent

Degradation during rebuild

Minor Moderate degradation

Slow Rebuild

(due to write penalty of parity)

RAID 10 RAID 50

Read Performance Very Good Very Good

11

Write Performance

Very Good Good

Use Case OLTP / OLAP applications Medium-write intensive OLTP / OLAP applications

Nested RAID Misc Notes:

• RAID 10 is faster and better than RAID 0+1 for the same cost

• RAID 60 is similar to RAID 50 except that the striped sets with parity contain dual parity

• Ideally RAID 10 and RAID 50 will be the only nested RAID levels you will use

RAID Considerations:

• Select your Stripe Size by empirical testing

smaller stripe size increases transfer performance, decreases positioning performance, and vice versa

ideal stripe sizes depend on your application, typical data read in a read, sequential vs random reads etc

• Try and select hard drives from separate production batches

• Maintain sufficient Spares in a large array (typically 1 per 10-15 disks is sufficient)

• Use Global spares across RAID groups if your controller supports it

RAID Considerations:

• Use hardware RAID unless performance is not a consideration

Especially nested RAID levels or parity based RAID – consume more CPU cycles and increase rebuild time if implemented in software

• Ensure the controller has battery backup to retain its cache in case of power failure

• For internal RAID Controller cards use faster PCI buses (PCI-x)

Storage Technologies :

12

• A secondary or tertiary storage may connect to a computer utilizing computer networks. This concept does not pertain to the primary storage, which is shared between multiple processors in a much lesser degree.

• Direct Attached Storage (DAS)-is a traditional mass storage, that does not use any network. This is still a most popular approach. This term was created lately, together with NAS and SAN.

• Network Attached Storage (NAS)-is mass storage attached to a computer which another computer can access at file level over a local area network, a private wide area network, or in the case of online file storage, over the Internet.

• Storage Area Network (SAN)-is a specialized network, that provides other computers with storage capacity. The crucial difference between NAS and SAN is the former presents and manages file systems to client computers, whilst the latter provides access at block-addressing (raw) level, leaving it to attaching systems to manage data or file systems within the provided capacity. SAN is commonly associated with Fibre Channel networks

Passive Disk Enclosure based Direct Attached Storage (PDE based DAS):

Passive Disk Enclosure based DAS:

• DAS – Direct Attached storage

• RAID controller inside host machine

• External chassis is simply a JBOD (Just a Bunch Of Disks)

(or what I’d like to call Passive Disk Enclosure or PDE) Eg Dell Powervault MD1000

13

• Passive Disk Enclosure can consist of SAS, SATA or FC drives

• Passive Disk Enclosure to RAID Controller connectivity can be SAS, FC, SCSI (possibly different from the backplane)

• Multiple PDEs can be daisy chained if they support it

• Array of disks can be divided into multiple RAID groups

• Array of disks can be divided into multiple heterogeneous RAID groups

• Size and type of a RAID group depends on RAID card

• PDE may have multiple paths to system with possibility of multiplexing for increased speed

• Global spares can be defined on the RAID card

• Maximum storage size = maximum number of PDEs that can be daisy chained x size of drives

14

• Performance Considerations

Drives

RAID configuration

PDE Interconnect

PDE to RAID Card connect

RAID card config (cache etc)

PCI bus

Active Disk Enclosure based Direct Attached Storage (ADE based DAS)

Active Disk Enclosure based DAS:

• ADE Difference -> RAID Card is not in the host machine but in the enclosure

• Host machine has a SAS/FC Host Bus Adaptor (HBA) depending on ADE to Host connectivity support

Some ADEs may support multiple connection protocols

• ADE may support SAS/FC/SATA drives

• ADE can support daisy-chaining PDEs

• Eg of ADE – Dell MD 3000, Infortrend eonstor devices, Nexsan Satabeast and Sataboy etc

15

Active Disk Enclosure based DAS:

• ADE may support dual RAID Controllers

• RAID Controllers can be used as Active-Active (incase of multiple RAID Groups) – otherwise as Active Passive

• RAID Controller to HBA connectivity can be multiplexed - if supported - for higher throughput

• ADEs are wrongly but commonly referred as SAN (SAN device would still be alright)

Partitioning and Mounting:

Logical Volumes:

• A RAID Group is a physical unit of storage

• At the Operating System a Logical Group can be created out of multiple RAID Groups

• Each Logical Group can be further divided into Logical Volumes

• Each Logical Volume represents a mountable block device

• In Linux this is done using LVM (logical volume manager for the Linux kernel; it manages disk drives and similar mass-storage devices, in particular large ones )

• In LVM Logical Volumes are resizable

16

SAN (Storage Area Network):

• Multiple host machines connected to an ADE through a SAN switch

• SAN refers to the interconnect + Switch + ADE + PDE

• Switch and HBA can be SAS / FC depending on interconnect type supported by ADE

• ADE would support creation of Volumes

• These can be mounted onto Client and further subdivided

• Care must be taken to mount each Logical Volume onto a single client (unless you are running a Clustered File System)

• This can be achieved by host masking supported by ADE and/or the Switch

• Without careful host masking and mounting data corruption can take place

17

• Complex SAN configs include multiple hosts and multiple ADEs connected to active-active switches with multiplexed connections

• Client hosts can be of heterogeneous operating systems

• (Funnily ADE to PDE paths sometimes are not be multiplexed)

• While this looks complex – just think of it as removing hard disks from the machine and hosting them outside in separate enclosures

• Each machine mounts an independent partition from the SAN

• Performance Considerations

• All variables we covered before

• Switch config

• Ensure that switch / HBA / interconnect does not become the bottleneck and full hdd throughput can be utilized

Throughput Calculations:

• Hard disk performance – Type, RPM etc

• Data distribution and Type of Data access

• RAID performance, number of drives, RAID type

• RAID card performance – cache, active-active config etc

• ADE to switch connection speed

• Switch to HBA connection speed

• HBA to PCI bus speed

18

Storage Technologies:

Technology Advantages Limitations Applications

Compact disc, recordable( CD-R ) or rewritable ( CD-RW ) and DVD

Low cost per megabyte

Unlimited capacity with multiple discs

Portable

Widely-supported I/O interfaces

Can be formatted for different data formats

Long life

High data density

Immune to corruption once data is written (CD-R and DVD only)

Limited capacity on one disc(though much greater than diskette)

Slow to moderate read/write speed

Data archiving

Data distribution

Data migration

Localized file sharing

Offsite storage


19

Diskettes, 1.44 MB

Simple to use

Portable

Can be formatted for different data formats

Limited capacity

Limited read/write speed

Not supported by many newer computers

Local data transfer of small files

Storage of small files or programs


Hard drive, external

High read/write speed

Can be moved among computers

Limited capacity

Awkward for data transfer among multiple computers

Local backup

Local archiving


http://searchstorage.techtarget.com/gDefinition/0,294236,sid5_gci212542,00.html

20

Hard drive, internal

Convenient; usually comes with the computer


Convenient for use with single computer (but can be shared among multiple computers with proper support

Most common form of data storage

Limited capacity

Without special support, confined to a single computer or server

Storage in a single computer

Swap files


Removable storage(ZIP disks, JAZ disks, etc.)

Simplicity

Portability

Unlimited capacity with multiple disks

Convenient for use with single computer

Proprietary media

Limited read/write speed

High cost per megabyte

Personal computing

Local data transfer of small files

Local backup

Local archiving

it is the part of an operating system which is responsible for interacting directly with hardware and does this by using your device drivers. think of the kernel as a manager managing and using the processes between the other parts of the operating system and the hardware. it executes tasks to be done and handles errors and access to your computer.

some other functions includes managing directly with your computer memory, allocating resources, "communicating" directly with the cpu and other devices such as your printer and flash drives etc and many more.

it is also responsible for "booting" up your computer, that is after your bios is processed and passes control of computer to your bootloader and from there your kernel given the control and initiates the rest.

the kernel is like the motherboard in your pc which holds and manage everything together.




21

Solid-state storage(USB devices, flash memory, smart cards, etc.)

No mechanical parts


Small form factor

Limited storage capacity

High cost per I/O operation

Swap files

Local data transfer

Internet service providers

Video processing

Relational databases

High-speed data acquisition


Direct-attached storage(DAS)

Simplicity

Low initial cost

Ease of management

Storage for each server must be administered separately

Inconvenient for data transfer in network environments

Server bears load of processing applications

Data and application sharing

Data backup

Data archiving







22

Disk library

High speed

High storage capacity

High data availability

Not as quickly accessible as DAS; intended for "write once, read rarely" data

Disk-to-disk (D2D) backup

Data archiving

Near line storage


Disk-to-disk-to-tape( D2D2T )

Redundancy


Unlimited capacity with multiple tapes

Complexity

Incremental backups

Storage virtualization

Offsite storage

Data archiving


Fibre Channel(See Storage area network below)

Used to transmit data between devices at gigabit speeds

Frequently used in storage area networks (SANs)

Flexible in terms of distance

High cost

Management complexity

Large databases

Bandwidth-intensive applications

Storage area networks (SANs)

Offsite storage

Mission-critical applications


http://searchstorage.techtarget.com/sDefinition/0,,sid5_gci212937,00.html

23

iSCSI(See Storage area network below)

Used to transmit data between devices using the Internet Protocol (IP)

Frequently used in storage area networks (SANs)

More flexible in terms of distance than Fibre Channel (but not as fast)

May not compare favorably with Fibre Channel for large database transfers


Applications involving remotely distributed databases

Storage area networks (SANs)

Offsite storage



Magnetic tape

Low cost per megabyte

Portability

Unlimited capacity with multiple tapes

Inconvenient for quick recovery of individual files or groups of files

Data archiving

Limited-budget businesses

Offsite storage


Network-attached storage(NAS)

Fast file access for multiple clients

Ease of data sharing


Redundancy

Ease of drive mirroring

Consolidation of resources

Less convenient than storage area network (SAN) for moving large blocks of data

Data backup

Data archiving

Redundant storage



24

Redundant array of independent disks(RAID)

High speed


High data availability

High reliability

Security

Fault tolerance

Users may develop false sense of security

Recovery from failure is difficult in some systems

High cost for optimum systems

Swap files

Internet service providers

Redundant storage


Storage area network(SAN)

Excellent for moving large blocks of data

Exceptional reliability

Wide availability

Fault tolerance

Scalability

High cost

Lack of standardization


Large databases

Bandwidth-intensive applications




Documents

Storage Lecture Notes