Session Title: Serial Attached SCSI (SAS) Technology ...aixportal.ru/images/stories/kirill/100709/Power_Systems_SAS... · © 2008 IBM Corporation Session Title: Serial Attached SCSI

© 2008 IBM Corporation

Session Title: Serial Attached SCSI (SAS) Technology Overview and Configuration

Alexey

[email protected] R. [email protected]

IBM Training – Power Systems Technical University – Chicago 2008

2 IBM Power Systems© 2008 IBM Corporation

Serial Attached SCSI Defined by SCSI Trade Association...

Serial Attached SCSI (SAS) is the logical evolution of SCSI, retaining SCSI's long-established software advantage....

In addition, the Serial ATA (SATA) electrical and physical connection interface, which has much smaller connectors and cables, benefits SAS with greater system cooling and lower system cost...

With enterprise storage requirements escalating and becoming more complex, factors such as larger capacity, greater density, security, scalability and accessibility are more critical than ever...

Enterprise data centers must be online all the time, fulfill requests from numerous users simultaneously, allow for constant growth and expansion and be maintained during operation. SAS meets all of these demands while providing the highest performance.



Parallel SCSI vs. SAS Key ComparisonsParallel SCSI SAS

Architecture Parallel, all devices connected to shared bus Serial, point-to-point, discrete signal pathsPort expander used for fan out

Performance320 Mbytes/sec (Ultra320 SCSI); performance degrades as devices added to shared bus. Maxed out.

3.0 Gbits/sec, roadmap to 12.0 Gbits/sec; performance maintained as more drives added. Performance headroom.

Scalability “15”

drives Over 16,000 drivesOnly limited configurations supported

Compatibility Incompatible with all other drive interfaces Compatible with Serial ATA (SATA)

Max. Cable Length

12 meters totalCan use SCSI repeaters to exceed this limit but they are expensive

8 meters per discrete connection; total domain cabling thousands of feetIBM Supports 15m X & Y cables

Cable Form Factor Multitude of conductors adds bulk, cost Compact connectors and cabling save

space, cost

Hot Plug ability

Not optimized Some care required Yes

Device Identification

Manually set, user must ensure no ID number conflicts on bus

Worldwide unique ID set at time of manufacture; no user action required

Termination Manually set, user must ensure proper installation and functionality of terminators

Discrete signal paths enable devices to include termination by default; no user action required



SAS Comparison to Fibre ChannelFC SAS

Media Copper(!) & fiberUp to 100 km with optical

Copper only15 m discrete connection. Domain cabling 100s of meters

Speeds 1.1, 2.1, 4.2, 10.5, 12.8

Gb/s, full-duplex 1.5, 3.0, 6.0, 12

Gb/s, full-duplex

Disk Drive Connectors

SCA-2 (40 pins)drives use FC switches similar to SAS expanders

Dual-ported version of the SATA connector (7+7+15 pins)drives use expander switches

Topologies point-to-point serialarbitrated loops, 127 devices, obsoleteswitched fabric, 16M devices

point-to-point serialno loopsexpanders,16k max addresses in fabric

Login/Logout three layers of logins:fabric loginport login/logoutprocess login/logout

no logins

Addressing each node has node nameeach port has a port nameset by factorynot used for addressing (FC 24-bit address is assigned)

each port has SAS addressset by factoryused for addressing

Virtualization N_Port

ID (NPIV), AIX ‘08, Linux & i ‘09 Single Root I/O, future

By permission SCSI Trade Assn & HP Corp



The Generations of Parallel SCSI

5 10 2040

80

160

320

0

50

100

150

200

250

300

350

Meg

abyt

es p

er s

econ

d

SCSI-1 SCSI-2 UltraSCSI

Ultra2SCSI

WideUltra2

Ultra 160 Ultra 320

1986

2001

1997



SAS Roadmap

↕Shipping in volume



SAS Devices–End Device

•Initiators (SAS Controllers)•Targets (SAS hard drives, SATA hard drives, SAS tape drives

–Expander

Connects initiators to targets, thereby creating a SAS domain. Not necessary, but helps a single initiator to communicate with more targets.

Service Delivery SubsystemThe part of an I/O subsystem that transmits information between an initiator and a target. Typically cables connecting an initiator and target with or without expanders and backplanes constitute a service delivery subsystem.SAS DomainAn I/O system consisting of a set of SAS devices that communicate with one another by means of a service delivery subsystem. A SAS domain may contain up to 16,256 devices.SAS FabricThe summation of all paths between all SAS controller ports and all IO device ports in the SAS subsystem including cables, enclosures, and expanders.SAS AddressUnique worldwide names for all SAS devices assigned at manufacturing to simplify identifying SAS devicesSES DeviceSCSI Enclosure Service device contained within the expanderSAS physical links (phys)A set of four wires used as two differential signal pairs that transmit in opposite directions

SAS Terminology



SAS Physical Links (phys)

Twisted Pairs with shields

Transmit @ 3Gbs

Receive @ 3Gbs

SAS chipSAS chip

AC coupled

AC coupled

A set of four wires used as two differential signal pairs:One signal transmits in one direction, while the other signal transmits in the opposite direction. Data may be transmitted in both directions simultaneously.

High speed serial interface using the same basic technology as IB and PCI-E•Point to point connections•Uses Port expanders for fan out (Switch chips)•Multiple lanes can be used for higher bandwidth

•SAS -

Single, Dual, or Quad lanes•PCI-E -

1X, 4X, 8X, 16X•IB -

1X, 4X, 12XHigher speeds over time:

SAS 1.5GHz (SATA), 3.0GHz, 6.0GHzPCI-E 2.5GHz Gen 1, 5.0GHz Gen 2 Gen 3 expected to be 10GHzIB 2.5GHz SDR 1, 5.0GHz DDR 2 10GHz QDR

≈300MBs



IBM SAS Chip

8 SAS ports in two groupsof 4 ports each

1.5 / 3.0GbsSAS or SATA

PCI-X DDR266 MHz, 64-bit

(PCI-E version later)

1.5 / 3.0GbsSAS or SATA

This chip is integrated into the system unitsSAS adapters

Battery Backup on RAID Adapter

Host Side

Drive SideObsidian

SAS Adapter chip

DDR2 RAM for adapter data buffers

CMOS SRAM

recovery logging reads & writes



IBM SAS Port expander

SAS Port expander

chip1.5 / 3.0GbsSAS or SATA 1.5 / 3.0Gbs

SAS or SATA

Port expanders are essentially switchesKey to SAS scale-out to provide large enterprise configurationsThey contain powerful processorsThey provide SES (SCSI Enclosure Services) for drive hot swapThey can make one big expander look like several smaller expandersCurrently you can buy switch chips with 10, 12, 18, 24 or 36 ports.Each port can run at a different speed if needed and can support SAS or SATA devices



SAS & SATA Drives

1.5GbsSingle SATA porthalf duplex

SATA drive7200 and 10K RPM

+5V & +12V power

1.5 / 3.0GbsSAS port 1

SAS drive10K and 15K RPM

1.5 / 3.0GbsSAS port 0

+5V & +12V power

No Master / slave jumpers

No SCSI IDs!

Dual

SAS ports for availabilitySAS WWIDs uniquely identify drives

SAS drive connecter bottom

Connectors quite small...will fit 3.5 or 2.5 in. drives

S1-S7 = Primary InterfaceP1-P15 = Power

SAS drive connecter top



Internal SAS

Port expander

Port expander

More drives or second adapter...

Fault tollerantWorks with bad expander or broken wires

SASDrive

IDEDVD

SAS driveshave two

SAS ports

No SCSI IDs!All drives, adapters, and port expanders have SAS WWIDs

2 lanes to each expander

SASChip

SATADVD

SATAto IDE

Converter

IDE

SASTape

Example only...Actual implementation

varies by system

Direct connections



PCI-X SAS Adapters

Feature 5900 (cadet) Entry SAS adapter (replaced by FC 5912)–

RAID 0,1,10 No write cache

–

Single adapter support only (no dual adapter support)Feature 5902 (squib) PCI-X DDR Dual - x4 3Gb SAS RAID Adapter

–

RAID 0, 5, 6, 10 and write cache

–

Pairs of adapters are always required

for dual write cache and fault tolerance

–

Managed by IBM SAS Disk Array Manager

–

SAS media devices not supportedFeature 5912 (cadet2) Entry SAS adapter

–

RAID 0,1,10 No write cache

–

Single and dual adapter support

ObsidianSASChip

PCI-X DDR

SAS 4X connecter(SFF-8088)

SAS 4X connecter

59xx

SAS Controller Support in AIX packagedevices.common.IBM.sissas

IBM i



SAS 15k RPM 3.5 in. Disk Drives

Feature Capacity (GB) OS Notes

3647 146

AIXLinux

Default 512B Sector3648 300

3649 450

3677 139.5

IBM i Default 520B Sector3678 283.7

3658 428



SAS Configuration ExamplesFor Power 520 and 550



Power 550 and 520 entry SAS backplane Not compatible with RAID option FC 5679

Power 520 or 550

FC8341

KaleriObsidian

Elana

VSES

SATAto IDE

DVD

I2C

Tape

No port expanders single connection to each drive



Feature 5679required for internal RAID.

RAID 0, 5, 6, and 10 supported.

Power 550 and 520 High function SAS Backplane

Internal RAID option

Power 550 or 520

FC 8345SAS

ExpanderSAS

Expander

KaleriObsidian

Mini SAS 4i

Elana

VSES

SATAto IDE

DVD

I2C

Tape

+V

A B

+V

1 1

Dual paths to drive via

SAS port expanders



DASD/Media Backplanes

TWO options (must pick one)–

#8345 is only option offered on 9407-M15, 9408-M25, 9409-M50 Required if IBM i on the system –

ensures support if i-formatted disk drives used or IBM i accesses the DVD/tape drives in CEC

Recommended for capability for future growth & performance –

#8341 not available on 9407-M15, 9408-M25, 9409-M50

DASD/media backplane#8341 #8345

IBM i support No YesCan use 175MB protected write cache (disk performance & RAID-5/6) No Yes

Can attach one EXP 12S Disk Drawer to CEC SAS port (with 175MB cache) No Yes

Lowest priced backplane Yes No8203 split backplane option n/a Yes, but*

Only adds $280

All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary

* Split backplane not supported by IBM i



Power 520 Optional 175MB Write Cache #5679

Augments disk performance and provides RAID-5 and RAID-6 capability for imbedded SAS disk controller

Does not use a PCI slot

Protected cache. Really two cards. 2nd card provides auxiliary write cache and battery (CCIN 57B7=aux cache, 57B8=write cache)

Protected write cache–

Auxiliary cache holds duplicate copy of write cache data. If write cache fails, content can be restored from auxiliary cache when write cache replaced.

–

Hot-plug battery helps protect cache contents on both cards

Supports up to 18 disk drives with SAS disk expansion drawer–

Supported on 520 2-, 4-core

–

520 1-core has a maximum of 6 disk drives



Power 520 Disk Configuration Rules

All i disk drives must be protected–

Must mirror or RAID-5/6

–

Applies to SAS or SCSI drives

–

Assumes SAN drives have their own protection established

All disk controllers with write cache must be protected–

Either use auxiliary write cache or mirror disk controllers

–

System unit 175MB SAS write cache #5679 includes auxiliary write

cache.

–

90MB write cache SCSI disk controller* (#5776/#5737/#0648) and 40MB write cache SCSI disk controller* (#5703/0628) must be mirrored since they have no auxiliary write cache option

–

757MB and 1.5GB write cache controllers* can use either auxiliary write cache or can be mirrored.

Reason: To further reduce possible single points of failure, especially ones with potential longer outages, more strict rules apply to POWER6 processor-based servers. (Same rules apply to POWER6 570.)

Controller

Controller or aux cache

* IBM i statement. AIX/Linux partitions do not support SCSI disk controllers with write cache on POWER6 servers.



520, 550

Logical scheme

8345/8346/8310 + 5679 + (3668 or 3674) + Cable = 1 EXP12S to base controllerIBM i, AIX, Linux

#8345

Basecontroller

175 MB#5679

Bulkhead

#3669 / 3670

#3668 / 3674

EXP12S(#5886)

PCI-X (-e)Adapter

disk drive

disk drive

disk drive

disk drive

disk drive

disk drive

8345/8346/8310 + 5679 + (3669 or 3670) + Cable = 3 internal drives to external controllerAIX, Linux

IBM i,AIX,Linux

AIX, Linux

one per CEC

#8345 – IDE DVD#8310 – SATA DVD #8346 – SATA DVD and 8 SFF disks



Power 520 and 550 High Function SAS Backplane with External Drive Drawer


1.5m or 3m YI cable The internal cable is FC 3674 on 520The internal cable is FC 3668 on 550

5886ESM

ESM

4x

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

2x

2x

Power 520/550

FC 8345

4x

SASExpander

SASExpander

KaleriObsidian

Mini SAS 4i

Mini SAS4x (S)

Elana

VSES

SATAto IDE

DVD

I2C

Tape

+V

A B

+V

1 1

5886Only one 5886 drive drawer allowed.

Additional drawers can be supported with external SAS adapter

System and drive drawer must be in the same rack

up to 18 drives on internal adapter...



1.5m or 3m YI-cable To 5886 Drive drawer

Power 550 and 520 internal SAS to external drive drawer configuration

C1-T1 C1-T2 C2-T1 C2-T22X

4X

The internal cable is FC 3674 on p520The internal cable is FC 3668 on P550



Power 520 and 550 with split backplanes

4x

4x

4x

4x

Kozeez

SASExpander

SASExpander

Kaleri Obsidian

Mini SAS 4iMini SAS

4x (S)

Popov

VSES

SATAto IDE

DVD

I2C

Tape

+V

A B

1 1

+V

5900 or5912

AI cable

5900/5912

AI-cable

For split backplane operationThe internal cable is FC 3670 on 520The internal cable is FC 3669 on 550



Power 550 and 520 SFF Drive SAS Backplane FC 8346


8 SFF (2.5 inch) drives

Dual paths to drive via SAS port expanders

Power 520 or 550

SASExpander

SASExpander

Obsidian

Mini SAS 4i

Popov

VSES

SATAto IDE

DVD

I2C

Tape

+V

A B

+V

1 1

FC 8346

Announced but GA has moved to 4-2009



Power 520 and 550 with Split Backplanes 8 SFF 2.5-inch Drives

For split backplane operationThe internal cable is FC 3670 on p520The internal cable is FC 3669 on P550

5900 or5912

4x

4x

4x

4x

SAS

ExpanderSAS

Expander

Obsidian

Mini SAS 4i

Mini SAS

4x (S)

Popov

VSES

SATA

to IDEDVD

I2C

Tape

+V

A B

1 1

AI cableFC 8346




Power 520 and 550 SFF Drive SAS Backplane With External Drive Drawer


Only one 5886 drive drawer allowed.

System and drive drawer must be in the same rack

1.5m or 3m YI cable

Power 520 or 550

4x

SASExpander

SASExpander

KaleriObsidian

Mini SAS 4i

Mini SAS4x (S)

Popov

VSES

SATAto IDE

DVD

I2C

Tape

5886ESM

ESM

4x

+V

A B

1 1

+V

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

2x

2x

FC 8346

The internal cable is FC 3674 on p520The internal cable is FC 3668 on P550




SAS Configuration ExamplesFor Power 575



Power 575 SASIH

Komachi

Azusa

ObsidianMini

SAS

4x

Note:

P575 SAS disk drives are all 2.5".

VSES I2C

SAS

Redriver



Power 575 with External SAS Drives

CharlotteESM

ESM

IH

Komachi

Note:-Komachi SAS disk drives are all 2.5".

4x

Azusa

ObsidianMiniSAS4x

VSES I2C

SASRedriver

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

2x

2x

IH

Komachi

Note:-Komachi SAS disk drives are all 2.5".

CharlotteESM

ESM

4x

CharlotteESM

ESM

Azusa

ObsidianMiniSAS4x

VSES I2C

SASRedriver

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

4x4x

2x

2x

YO cable

EE cables

YO cable

FC 5886 SAS drive drawers

Power 575

Power 575

up to 26 drives on internal adapter...



SAS Configuration ExamplesFor Power 570



560, 570

Logical scheme

Basecontroller

#5909/5911**0 cache

PCI-e adapter

PCI-X (-e)Adapter

disk drive

disk drive

disk drive

disk drive

disk drive

disk drive

IBM i*, AIX, Linux

one way per CEC

#3650PCI-e slot

#3651PCI-e slot

PCI-X (-e)Adapter

* IBM i supported only with 3651 and 5908 (PCI-x 1.5 GB SAS RAID adapter)

**5911 allows to use 3rd

SAS port



IBM Power 570 SAS

36 5 4 12

012

2

3

3

4

4

5

5

6

6

7

7 Phy

Phy

Slot

8

8

SAS Expander

01

B

1

2

9

9

IBM Power 570

Uhura

SAS Expander

A

12

Ob

sid

ian

VSESDVD

I2CSATAto IDE

McCoy+V

+V

+V

Chapel

Connecter for3650 & 3651 option cards

Note: hardware RAID not supported by intenal

SAS adapter



IBM Power 570 split backplane requires a FC 3650 card in slot 3 and a 5900 adapter in slot 4

Ca detLuc id

AI- cab le

36 5 4 12

012

2

3

3

4

4

5

5

6

6

7

7 Phy

Phy

Slot

8

8

SAS Expander

01

B

1

29

9

P6 570

Uhura

SAS ExpanderA

12

Obsidi an

VSESDVD SATAto IDE

DVD

I2CSATAto IDE

McCoy+V

+V

+V

Chapel

1

2

Mini SAS 4x (S)

5900 or5912

4x

4x

4x

4x

AI Cable

3650

car

d With jumpers set on the 3650, the port expander allows the external adapter to see three drives...

...and the internal adapter to see the other three drives

2nd

port availablefor use. Another backplane

or 5886

IBM Power 570



IBM Power 570 split backplane requires a FC 5909 card in slot 3

36 5 4 12

012

2

3

3

4

4

5

5

6

6

7

7 Phy

Phy

Slot

8

8

SAS Expander

01

B

1

29

9

P6 570

Uhura

SAS ExpanderA

12

Obsidi an

VSESDVD SATAto IDE

DVD

I2CSATAto IDE

+V+V

+V

Chapel

1

2

Mini SAS 4x (S)

5909 4x

4x

4x

4x

AI Cable

3650

car

d

With jumpers set on the 3650, the port expander allows the external adapter to see three drives and the internal adapter to see the other three drivesIBM Power 570

AI-cable

No Connection allowed

Feature code 5909Combined cable and adapter in slot 3



2nd

port available for use.Another backplane or

5886

IBM Power 570 internal drives to external adapter requires a FC 3651 card in slot 3 and a 5900/5912 adapter in slot 4

CadetLucid

AI-cable

36 5 4 12

012

2

3

3

4

4

5

5

6

6

7

7 Phy

Phy

Slot

8

8

SAS Expander

01

B

1

29

9

P6 570

Uhura

SAS ExpanderA

12

Obsidia n

VSESDVD SATAto IDE

DVD

I2CSATAto IDE

McCoy+V

+V

+V

Chapel

Mini SAS 4x (S)

4x

4x

1

2

5900 or 5912

4x

4x

3651

car

d

AI Cable

The port expanders do not allow the internal adapter see any of the drives.The external adapter sees all the drives

IBM Power 570



IBM Power 570 internal drives to external adapter requires a FC 3651 card in slot 3 and two SAS RAID adapter in slots 4 and 5

SquibLucid

YR-cable

The port expanders do not allow the internal adapter see any of the drives.The external adapter sees all the drives

36 5 4 12

012

2

3

3

4

4

5

5

6

6

7

7 phy

Phy

Slot

8

8

SAS Expander

01

B

1

29

9

IBM Power 570

Uhura

SAS Expander

A12

Obsidi an

VSESDVD SATAto IDE

DVD

I2C36

51 c

ard

SATAto IDE

McCoy+V

+V

+V

Chapel

Mini SAS 4x (S)

4x

1

2

4x

2x

2x

59024x

4x

4x

4x5902



IBM Power 570 8-way internal drives to external RAID adapters requires a FC 3651 card in slot 3 and 5902 adapter in slots 4 of both systems

SquibLucid

YR-cable +V

+V

Obsidian

VSESDVD SATAto IDE

DVD

I2C

+V

36 5 4 12

012

2

3

3

4

4

5

5

6

6

7

7 Phy

Phy

Slot

8

8

SAS Expander

01

SATAto IDE

B

1

2

9

9

IBM Power 570

Mini SAS4x (S)

Uhura McCoySAS Expander

A12

4x

1

2

+V

+V

Obsidia n

VSESDVD SATAto IDE

DVD

I2C

+V

36 5 4 12

012

2

3

3

4

4

5

5

6

6

7

7 Phy

Phy

Slot

8

8

SAS Expander

01

SATAto IDE

B

1

2

9

9

IBM Power 570

Mini SAS4x (S)

Uhura McCoySAS Expander

A12

4x

1

2

4x2x

2x

4x

4x

5902 4x

4x

4x

2x

2x

5902



SAS Configuration ExamplesMedia Drawers



Little Horn 7214-1U2 1U 19 inch rack 2 Tape bays

Or 1 Tape and 2 DVD bays

Rear

19 inch SAS Media drawer

Tape DVD

DVD IBM

Tape Tape IBM

ACSAS

Rear of box

SAS 4X connecter



24 inch SAS Media drawer

Big Horn FC 5720 1U 24 inch rack 1 Tape and 2 DVD bays

SAS 4X connecter

Tape DVD

DVD

SAS

Rear of box

Power cable to first IO drawer



Media drawer cabling Single initiator Single path

The media drawer can only be attached to the SAS JBOD adaptersIt cannot be attached to the integrated adapter on Power 550 and

520 or to SAS RAID Adapter

590x

A

CSA

S

Rear of box

3m or 6m AE SAS cable

Optional to a FC5886 drawer or another Little Horn

3m or 6m AE SAS cable

SAS expander

SAS Tape

SAS Tape

SATAto IDE

ConverterDVD

SATAto IDE

ConverterDVD

7214-1U2

590x



SAS Configuration ExamplesCabling EXP12S (5886)



1 2 3 4

9 10 11 12

5 6 7 82U tall

Front 12x 3.5 inch SAS drives

Rear ESM cards and Power suppliesESM cards(SAS port expanders)

Power Supplies and Fans YO or X cable

C1-T1 C1-T2 C2-T1 C2-T2

SAS drive drawer FC 5886AIX, IBM i, Linux



FC 5886 Drawer Cables Two drawers in series

YO or X cableEE-cable

C1-T1 C1-T2 C2-T1 C2-T2

C1-T1 C1-T2 C2-T1 C2-T2



5900 or 5912 Adapter to One 5886 Drawer Single initiator dual path

• JBOD (non RAID) uses 512 byte sector drives•

Hardware

RAID 0,1, or 10 require 528-byte sector drives

Note: AIX LVM mirroring is software

RAID

JBOD = Just a Bunch Of Drives

4x

4x

5886ESM ESM

4x

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

2x

2x

YO Cable

5900 or5912



5900 or 5912 to two 5886 drawers Single initiator dual path

4x

4x

5886ESM ESM

4x

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

2x

2x4x

5886ESM ESM

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

2x

2x

YO Cable5900 or5912

Performance*Optimized

*Must take into account all other performance influencing factors



5900 or 5912 Adapter to max of 4 FC 5886 drawers Single initiator dual path

4x

4x

4x

4x

5886ESM ESM

5886ESM ESM

5886ESM ESM

5886ESM ESM

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

2x

2x

2x

2x

2x 2x2x 2x

YO Cable

EE Cable

5900 or5912

CapacityOptimized

can use 3 or 4 drawers...



SAS Adapters and cables



SAS Controller Functionality

Feature 5912(replace to 5900) 5902

CCIN 572A 572B

Write cache size NA 175MB

Description PCI-X External Dual-x4 3Gb SAS Adapter

PCI-X External Dual-x4 3Gb SAS RAID

AdapterHigh availability (HA) multi-initiator RAID Yes Yes

HA two-system JBODNote: HA single-system JBOD

configuration not supported

Yes No

OS IBM i, AIX, Linux AIX, Linux



4x

4x

4x

4x

CharlotteESM ESM

CharlotteESM ESM

CharlotteESM ESM

CharlotteESM ESM

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

2x

2x

2x

2x

2x 2x2x 2x

Cadet

Each cable contains four SAS phys typically organized into single 4x port or two 2x portsOnly specific cabling configurations are supported, although many could be constructed that are not supported and will either not function or generate errors.Each mini SAS 4x connector is keyed to help prevent cabling an unsupported configEach cable end has a label that graphically describes the correct component port to which it is designed to be connected such as:

–SAS adapter

–Expansion drawer

–System external SAS port

–Internal SAS disk slots connectionCable routing is important

–YO, YI, and X cables must be routed along right side of the rack

frame (as viewed from the rear) when connecting to a disk expansion drawer (so short/long end of cable hit correct ESM card...the adapter is particular about which ESM connects to which lanes)

–X cables must be attached to the same numbered port on both SAS adaptersAlways select the shortest cable that will provide the needed connectivity

SAS Cable Planning Considerations



SAS cables and connecters

Mini SAS 4x receptacle

Keys of 2,4,6 are possible and you can use combinationsA cable with all 3 keys will plug into any thingA connecter with no keys will accept any cable

Mini SAS 4x plug

(SFF-8088)

Plug Use Icon Key Slot Positions

Attaches to end device, enclosure out port, or universal port ♦ 2, 4

Attaches to end device, enclosure in port, or universal port • 4, 6



dual-adapteroptimized

SAS Cable Schematics all using industry standard 4X connecters

SAS YO or YI cable2 sets of transmit and receive pairsat storage drawer connecter

4 sets of transmit and receive pairs (“lanes”)

4 sets of transmit and receive pairs at adapter connecter

SAS 4X cable (AE, AI )

SAS X cable

Adapter Key

Adapter Key

Drawer Key

Drawer Key

Drawer Key

Drawer Key

Adapter or drawer key

Adapter Key

Adapter or drawer key



SAS Cable TypesType Connects L (m) FC Label

AI adapter to internal disk slots through #36xx cable card 1 3679

AE adapter to media expansion drawer or two adapters to disk expansion drawer in JBOD configuration

3 3684

6 3685

EE disk expansion drawer to another in a cascaded configuration

1 3652

3 3653

6 3654

YO adapter to disk expansion drawer

1.5 3691

3 3692

6 3693

15 3694

YI system external SAS port to disk expansion drawer1.5 3686

3 3687

X two SAS adapters to disk expansion drawer in RAID configuration

3 3661

6 3662

15 3663

YR connects two SAS RAID Adapters to a bulkhead port for accessing internal SAS disks

(Power 570) 1 3667



Single Adapter (initiator) Configurations

SASChip

Port expander

Port expander

Drive drawer

SAS expander

SAS TapeSAS Tape

SATAto IDE

ConverterDVD

SATAto IDE

ConverterDVD

Media Drawer

Single initiator dual path

Single initiator single path

A cable

Y cableAdapter = Initiator



5902 or 5912 to 5886 Drawer RAID -

Dual initiator dual path

5886ESM ESM

2x

2x2x

4X

2x

4X

4x

4x

4x

4x

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

2x

2x

2x

4X

2x

4X

X cable

Dual 5902orDual 5912

For RAID512 bytes per

sector disks are not to be used functionally, but will be

available to be formatted to 528 bytes per sector.

Provides redundant controllers in a single system configuration.Typically used with Multi-Path I/O (MPIO). Or they can be in separate partitions/systems for HACMP-like support

Boot support only in

single systemRAID config



5902 or 5912 Adapter to two FC 5886 drawers RAID

-


2x

2x2x

4X

2x

4X

4x

4x

4x

4x

2x

2x2x

4X

2x

4X

5886ESM ESM

5886ESM ESM

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

2x

2x

2x

4X

2x

4X

2x

2x

2x

4X

2x

4X

X cable

X cable

5902 or5912



Can define

multiplemixed RAID arrays

5902 or 5912 Adapter to Max of 4 FC 5886 Drawers RAID -


2x

2x2x

4X

2x

4X

2x

2x

4X

2x2x4X

4x

4x

4x

4x

5886ESM ESM

5886ESM ESM

5886ESM ESM

5886ESM ESM

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

2x

2x

2x

4X

2x

4X

2x 2x

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

C2-T2(4x)

C2-T1(4x)

C1-T2(4x)

C1-T1(4x)

2x

2x

4X

2x2x

4X

2x 2x

X cable

EE cables

X cable

EE cables

5902 or5912



Consider HA RAID over HA JBOD for increased redundancy, performance,

& reliability

Dual 5912 adapters to Single 5886 Drive Drawer in an HA JBOD configuration

JBOD only (no RAID 0, 1, or 10)512 byte sector drives only

Both adapters can see the drives but cannot see each other.

Controller settings must be changed from Dual Initiator

to JBOD HA Single Path via SAS Disk Array Manager

Dual initiator single path

4x

4x

5886ESM ESM

C2-T2(4x)

C2-T1(4x)

4x

5912

4x

4x

4x

C1-T2(4x)

C1-T1(4x)

AE cable

AE cable5912

No boot supportin two system

JBOD configuration

Note: Two-

system only. HA single-

system JBOD configuration not supported



Multi-initiator and High Availability For all HA RAID configurations

–

One controller functions as the primary controller and performs direct management of the physical devices.

–

The second controller serves as a client of the primary controller and does not perform reads or writes to devices directly. Commands directed to disk arrays through the secondary controller are forwarded through the SAS fabric to the primary controller for execution.

–

The secondary controller detects if the primary controller goes offline and becomes the primary controller. When the original primary controller comes back online, it now serves as the secondary controller, unless it was previously designated as the preferred-primary controller.

–

The primary controller logs errors related to problems with a disk array. These array errors are not forwarded to the secondary controller.

–

Typical reasons for the primary and secondary controllers to switch roles:•

for asymmetric reasons. For example, one controller detects more

disk drives than the other. If the secondary controller is able to find devices that are not found by the primary controller, an automatic transition (failover) occurs. The controllers will communicate with each other, compare device information, and switch roles.

•

Powering off the primary controller or the system that contains the primary controller •

Failure of primary controller or the system that contains the primary controller •

If the preferred primary controller is delayed in becoming active, the other controller assumes the role of primary controller. After the preferred primary controller becomes active, an automatic transition (failover) occurs.

•

If the primary controller loses contact with the disks that are also accessible by the secondary controller

•

Downloading controller microcode might cause an automatic transition (failover) to occur.HA single system configuration

–

provides for redundant controllers from a single system to the same set of disks and disk arrays. –

This feature is typically used with the Multi-Path I/O (MPIO), which is part of AIX and can be used to provide a redundant IBM SAS RAID Controller configuration with RAID protected disks.



Performance



SAS Arrays

RAID Level Description Considerations

0Stripes data across the disks for optimal performance. High potential I/O rate but no data redundancy

Backup regularly to protect against data loss100% usable disk capacity

5Stripes data across all disks in the array. Parity data spread across all disks. Supports failure of one

disk.

3 –

18 drives per array67-94% usable disk capacity

6Stripes data across all disks in the array. P and Q parity data is spread across all disks. Supports failure of two

disks.

4 –

18 drives per array50-89% usable disk capacity

10

Uses mirrored pairs to redundantly store data. The data is striped across the mirrored pairs. When created, the controller will automatically attempt to select disks for each mirror from a different controller connector.

2 –

18 drives per arrayArray must have even

number of With 2 drives it is really RAID 1 (mirrored)With 4 or more drives it is RAID 10 50% usable disk capacity

All drives in arrays using hardware RAID must be formatted to 528 byte sectorsDownside of larger arrays:

–higher probability of failed drive

–larger number of drives with degraded performance in case of failure



General Performance NotesPCI-X SAS adapter capacity approximately 800 MBs (reads) RAID, JBOD 1.2 GBs(reads)SAS drive approximately 100 MBs, sequential large block→ about 5 drives to saturate adapter→ striping performance a bit less, about 6 drives to saturate adapter→ random, need a lot more drives to saturate adapterRAID 5/6, writes slower, but reads should be about the sameOn RAID 5 and especially on RAID 6 the write cache can hide a lot of write latencyIn general, the choice between RAID 5 and RAID 10 is highly dependent on the application IO workload and the storage configuration.Given a random IO workload and the same number of disks, with a portion of the IOsbeing writes, RAID 10 will provide more IOPS bandwidth than RAID 5Given a 100% random read workload and the same number of disks, RAID 5 performance is equivalent to RAID 10Given a sequential write workload and the same number of disks, RAID 5 will outperform RAID 10 in thruputRAID 5 random writes will take longer than RAID 10 random writes when no write cache is usedIn many cases for random workloads where we don't stress the storage IOPS bandwidth, RAID 5 and RAID 10 will provide equivalent performance



Disk Drive Performance Review

Disk Performance (IO/s)= 1/service time = 1/

(seek time

(3-4 ms)

+ rotational latency

(avg half rev, 2 ms@15k)+ data transfer time

(assume 4k block)

+ host bus overhead+ command overhead){≈1 ms

Operations may be queued at drive for additional seek and latency improvements



Notes on Lab Measurements

For measurements on following slides:all use 2 SCSI busses vs. 1 SAS adapter (8 phys @3gbs)

SAS and SCSI drives essentially the same, performance-wise, in this test. Both 15k, etc

For transactional workloads with small blocks, disk data transfer time is not very important

NV FW CacheFC 5737 = 90MBFC 5902 = 175 MB



Performance Notes -1 of 3

Observations:1.

approx 2X performance improvement

2.

Factors:

cache size

adapter processor speed



Performance Notes –

2 of 3

Observations:

1.

96% throughput improvement on right

2.

Factors:

on left, adapter processor speed, esp

4k & 8k blocks

on right, DRAM throttles performance

SAS bandwidth



Performance Notes –

3 of 3

Observations:

1.

245% throughput improvement on right

2.

Factors:

on left,

adapter processor speed esp

4k & 8k blocks

on right, cache size

SAS bandwidth

full-stripe writes



References

SCSI Trade Associationwww.scsita.org

T10 SAS Standardwww.t10.org/ftp/t10/drafts/sas/sas-r05.pdf

SAS Cable Planningpublib.boulder.ibm.com/infocenter/systems/index.jsp?topic=/iphad/sascabling.htm

Power Systems Expansion Units and Disk-Drive Enclosures, SA76-0151publib.boulder.ibm.com/infocenter/systems/topic/iphdx/sa76-0151.pdf

Power Systems SAS RAID Controller for AIX Reference (formerly SA76-0112)publib.boulder.ibm.com/infocenter/systems/scope/hw/topic/arebj/arebj.pdf

Power Systems SAS RAID Controller for Linux Referencepublib.boulder.ibm.com/infocenter/systems/scope/hw/topic/arebk/arebk.pdf

The latest version of this presentation is avaialable on Techdocsw3.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS3351partners.boulder.ibm.com/src/atsmastr.nsf/WebIndex/PRS3351



Acknowledgements

Pat BucklandIBMPower Systems I/O Hardware Development

Clark AndersonIBMI/O Subsystem Performance


72 IBM Power Systems© 2008 IBM Corporation72

TrademarksThe following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.

The following are trademarks or registered trademarks of other companies.

* All other products may be trademarks or registered trademarks of their respective companies.

Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance

ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.This publication was produced in the United States. IBM may not

offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information

on the product or services available in your area.All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the

United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.

For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:

*, AS/400®, e business(logo)®, DBE, ESCO, eServer, FICON, IBM®, IBM (logo)®,

iSeries®, MVS, OS/390®, pSeries®, RS/6000®, S/30, VM/ESA®, VSE/ESA, WebSphere®, xSeries®, z/OS®, zSeries®, z/VM®, System i, System i5, System p, System p5, System x, System z, System z9®, BladeCenter®

Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market.

Those trademarks followed by ®

are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.



Backup



SAS Adapter firmware

BulletinsBulletins for System p

PCI-X266 Ext Dual-x4 3Gb SAS/SAS RAID Adapter,SAS RAID Enablement Card

Microcode Update for PCI-X266 Ext Dual-x4 3Gb SAS/SAS RAID Adapter, SAS RAID Enablement Card

Version: 02200065 The following is provided with this version: -

Fix for out of resources error. The SAS/SCSI controller would previously allow for up to 16 resources per drive. This has been increased to 20 to coincide with the theoretical maximum. Visit

Microcode downloads

for all the latest updates.

Doc number: 4418

Published date: 2008-10-24

http://www14.software.ibm.com/webapp/set2/firmware/gjsn



SES Microcode Update

Bulletins for System p

Microcode update for SAS Enclosure Services for 7214-1U2, FC 5720/CCIN 509D

Version: 0020 The following is provided with this version: -

Moved some interrupt handling into RAM so it can run quicker on

sending COMINIT

-

Change priority of PHY interrupt from low to medium. -

Disable all PHYs

before completing firmware update.

Visit

Microcode downloads

for all the latest updates.

Doc number: 4455

Published date: 20081119

http://www14.software.ibm.com/webapp/set2/firmware/gjsn



SAS Adapter firmware –

Determining Level for AIX

Before you install the microcode, it is important to determine the microcode level of the Adapter installed in the target system. Use the following instructions to read the

ROM level stored in the Adapter's VPD.1)

List the SAS RAID Adapters installed in the system by typing:lsdev -C | grep sissas

Note: All of the SAS RAID initiators addresses that are installed will be displayed. The names will be sissasX, where X is 0, 1, etc.2) To check the current microcode level for the adapter or controller enter the following command

lscfg -vl sissasXWhere X is the number/s of the previous command. The Command will produce output similar toFor 572ADEVICE

LOCATION

DESCRIPTIONsissas0

U78A0.001.DNWFZZT-P1-C4-T1

PCI-X266 Ext Dual-x4 3Gb SAS AdapterPCI-X266 Ext Dual-x4 3Gb SAS Adapter

:Part Number.................42R7989FRU Number..................42R7989Serial Number...............YL3027138706Manufacture ID..............0002EC Level....................0ROM Level.(alterable).......02200026Customer Card ID Number.....572AProduct Specific.(Z1).......1Hardware Location Code......U78A0.001.DNWFZZT-P1-C4-T1

If the ROM Level is less than 02200065 then you should update the microcode.



‘smit disk’



→ List All Defined Disks



SAS Disk Characteristics



SCSI Disk Characteristics



‘smit disk_array’



IBM SAS Disk Array Management



IBM SAS Disk Array Manager

≈

½

min / GB



AIX sissasraidmgr

Command-line InterfaceTask Command

General Help sissasraidmgr -h

Viewing the Disk Array Configuration sissasraidmgr -Ll controller name -j1

Preparing Disks for Use in SAS Disk Arrays sissasraidmgr -P -z disk list

Changing pdisks to hdisks sissasraidmgr -U -z pdisk list

Creating a SAS Disk Array sissasraidmgr -C -r raid level -s stripe size -z pdisk list

Deleting a SAS Disk Array sissasraidmgr -D -l controller name -d array name

Adding Disks to an Existing Disk Array sissasraidmgr -A -l array name -z pdisk list

Creating Hot Spare Disks sissasraidmgr -H -z pdisk list

Deleting Hot Spare Disks sissasraidmgr -I -z pdisk list

Recovering from Disk Failures sissasraidmgr -R -z pdisk list

Viewing the SAS device resource locations sissasraidmgr -Z –o0 –j3 -l adapter name

Viewing the SAS device resource information sissasraidmgr -Z –o1 –j3 -l adapter name

Viewing the SAS path information for the attached device sissasraidmgr -T –o1 –j3 -l device name

Viewing SAS path information graphically for attached device sissasraidmgr -T –o0 –j3 -l device name

Alternative to Disk Array Manager

Documents

Session Title: Serial Attached SCSI (SAS) Technology ...aixportal.ru/images/stories/kirill/100709/Power_Systems_SAS... · © 2008 IBM Corporation Session Title: Serial Attached SCSI