w

teaming

Processor Node Processor Node

CPU CPU CPU CPU

MEM MEMHCA HCA HCA HCA

Switch Switch

SwitchSwitchTCA

Drives

TCA TCA TCA TCA TCA TCA

TCA

Controller

Router

Subnet Manager

To other Subnets,LAN, WAN, Internet

InfiniBandStorageSubsystem

I/OChassis I/O

Chassis

I/OM

odul

e

I/OM

odul

e

I/OM

odul

e

I/OM

odul

e

I/OM

odul

e

I/OM

odul

e

Graphics Video Ethernet SCSI Fibre Channel

Infiniband HCAHardware

Hardware Specific DriverProvider

Infiniband Verb/APIMid-Layer

IPoIBUpper-Layer

SDP LibraryUser APIs

IP BasedApplication

AccessApplication

SRP

Socket BasedAccess (DB2)

Block StorageAccess

uDAPL

GPFSManagement

GPFSBlock Device

GPFSNSD Server/ClientGPFS

AIX/Linux/WindowsOS

r

r

driverfailovertimeminimizeoption

r

LUN3

LUN2

LUN1

LUN0

/dev/sdc/dev/sdd/dev/sde/dev/sdf

/dev/sdg/dev/sdh/dev/sdi/dev/sdj

RDAC/dev/sdc/dev/sdd/dev/sde/dev/sdfDM-Multipath/dev/dm-3/dev/dm-4/dev/dm-5/dev/dm-6

Controller A

Controller B

Owner

Owner

ActiveStandbyActiveStandby

StandbyActiveStandbyActive

LUN LUN LUN LUN

Mdisk Mdisk Mdisk Mdisk

Mdisk group

Vdisk Vdisk Vdisk Vdisk Vdisk

IBM SVCCluster

AIX Solaris Windows

LUNs fromHeterogeneousStorages

Server ...

............

IBM SVC Logical ViewIBM SVC Physical View

IBM storage EMC storage

VirtualDisk

ManagedDiskGroup

ManagedDisk

IBM SVC Cluster

SAN Fabric

Heterogeneous Storages

Block 1 Block 2 Block 3 Block 4 Block 5 Block 6Block 7 Block 8 Block 9 Block 10 Block 11 Block 12

NSD1 NSD2

/gpfs/somefile

GPFS block size 1 MBRAID5 (Using 4+P) per LUNsStorage LUN Segment Size 256 KB

Block 1

Block 4

Block 7

Block 10

Block 2

Block 5

Block 8

Block 11

Block 9

Block 12

Block 3

Block 6

NSD0

r

r

D ata path failover

true

O nlinetransaction processing (O LTP)

O nlineanalyticalprocessing (O LA P)

r

rr

C ontentcreation

C ontentediting

renderfarm

Hardwarearchitecture/operatingsystem/applications

Content creationEditing workstation(Animators' workstation)

Render Farm

Scalable network backbone

Centralizedstorage

Linkage to otherproduction facilities

Animators'network

Hardwarearchitecture/operating system/application

Infrastructure, gateways,backups, Internetservices, assetmanagement, productionworkflow automation,production dashboard

petabyteag e

GPFS

LinuxKVM GPFS

VirtualDesktop

VirtualDesktop

VirtualDesktop

VirtualDesktop

LinuxKVM GPFS

VirtualDesktop

VirtualDesktop

VirtualDesktop

VirtualDesktop

LinuxKVM GPFS

VirtualDesktop

VirtualDesktop

VirtualD

3sktop

VirtualDesktop

GPFS

LinuxKVM GPFS

VirtualDesktop

VirtualDesktop

VirtualDesktop

VirtualDesktop

LinuxKVM GPFS

VirtualDesktop

VirtualDesktop

VirtualDesktop

VirtualDesktop

LinuxKVM GPFS

VirtualDesktop

VirtualDesktop

VirtualD

3sktop

VirtualDesktop

Application NodeWith

GPFS Daemon

Application NodeWith

GPFS Daemon

Application NodeWith

GPFS Daemon

Application NodeWith

GPFS Daemon

Application NodeWith

GPFS Daemon

VSD/NSDSERVER

VSD/NSDSERVER

HIGH SPEED IP INTERCONNECTGbE, 10GbE, Myrinet, IB, HPS

SAN-A SAN-B

DISK SUBSYSTEM POOL

Application NodeWith

GPFS Daemon

Application NodeWith

GPFS Daemon

Application NodeWith

GPFS Daemon

Application NodeWith

GPFS Daemon

Application NodeWith

GPFS Daemon

DISK SUBSYSTEM POOL

SAN FABRICHIGH SPEED IP INTERCONNECTGbE, 10GbE, Myrinet, IB, HPS

Application NodeWith

GPFS Daemon

Application NodeWith

GPFS Daemon

Application NodeWith

GPFS Daemon

Application NodeWith

GPFS Daemon

Application NodeWith

GPFS Daemon

NSDSERVER

NSDSERVER

HIGH SPEED IP INTERCONNECTGbE, 10GbE, Myrinet, IB, HPS

SAN-A SAN-B

DISK SUBSYSTEM POOL

Application NodeWith

GPFS Daemon

ExampleVisualization Workstation

r

metanode

init

quorum

Server1Quorum

Server2Quorum

Server3Quorum Server4

/gpfs

Server1Quorum

Server2Quorum

Server3Quorum Server4

/gpfs

Tiebreaker NSD Disks

Server1Quorum

Server2Quorum

Server3Quorum Server4

/gpfs1

Desc

Failure Group1

Desc

Failure Group2

Tiebreaker

Desc

Failure Group3

Desc

Failure Group4

Tiebreaker

Desc

Failure Group5

DescOnlyNSD

Failure Group6

/gpfs2Failure GroupA

Tiebreaker

Failure GroupB

DescOnlyNSD

Failure GroupC

/gpfs3

Desc

Failure Group7

Desc

Failure Group8 Failure Group9

System pool - HighSpeed

Archive Pool Low speed

DescOnly NSD does not have data and metadata, and uses small volume size

RHEL5GPFS3.4

RHEL5GPFS3.4

RHEL5GPFS3.4

RHEL5GPFS3.4

Windows2008 R2GPFS3.4

AIX 5.3GPFS3.4

/gpfs

/gpfsvol

/gpfs

/gpfsvolFail

SharedStorage

NSD OwnershipRedistribution

r r

r

r

peerG PF S clusters

Client File Stream

NSD1

NSD2

NSD3

/gpfs

Server1 Server2 Server3

NSD1:Server1,Server2NSD2:Server2,Server3NSD3:Server3,Server1

GPFS NSD Server

GPFS NSD ClientB1 B2 B3 B4 B5 B6 B7 --- ---B1 B2 B3 B4 B5 B6 B7 --- ---

NSD1 NSD2 NSD3/gpfs

Server1 Server2 Server3

NSD1: No assignNSD2: No assignNSD3: No assign

GPFS NSD Server

Server 1B1 B2 B3 B4 B5 B6 B7 --- ---B1 B2 B3 B4 B5 B6 B7 --- ---

NSD1

NSD2

NSD3

Server1 Server2 Server3

NSD1

NSD1

NSD1

Server1 Server2 Server3

NSD1

NSD2

AIX AIX

NSD1

NSD2

Window Window

NSD1

NSD2

Linux Linux

Multi Platform

Shared Storage

Local Storage

Three GPFS Cluster Configurations

NSD1

NSD2

NSD3

Server1 Server2 Server3

NSD1

NSD2

NSD3

Server1 Server2 Server3

NSD1

NSD1

NSD1

Server1 Server2 Server3

NSD1

NSD1

NSD1

Server1 Server2 Server3

NSD1

NSD2

AIX AIX

NSD1

NSD2

Window Window

NSD1

NSD2

Linux Linux

NSD1

NSD2

AIX AIX

NSD1

NSD2

Window Window

NSD1

NSD2

Linux Linux

Multi Platform

Shared Storage

Local Storage

Three GPFS Cluster Configurations

File System Replication

Failure Group1NSD1~3

Failure Group2NSD4~6

Failure Group3NSD7~9

storag epool

Metadata Pool

Metadata Pool

DataPoolOne

Mirror

DataMirror

Mirror

Solid State Disks

Fibre Channel Disks

Fibre Channel Disks

Fibre Channel Disks

Fibre Channel Disks

Solid State Disks

Fibre Channel Disks

Fibre Channel Disks

Fibre Channel Disks

Fibre Channel Disks

Solid State Disks

Solid State Disks

Fibre Channel Disks

Fibre Channel Disks

Fibre Channel Disks

Fibre Channel Disks

Fibre Channel Disks

Fibre Channel Disks

Fibre Channel Disks

Fibre Channel Disks

DataPoolOne

DataPoolTwo

DataPool

Three

DataPoolTwo

SATA Drives

SATA Drives

SATA Drives

SATA Drives

SATA Drives

SATA Drives

SATA Drives

SATA Drives

SATA Drives

SATA Drives

policies

snapsh ot

stub file

RH

EL5

RH

EL5

RH

EL5

RH

EL5

RH

EL5

RH

EL5

RH

EL5

RH

EL5

RH

EL5

RH

EL5

P S

1

4

P S

2

3

7

8

9

10

M M 13

M M 24

1

2

5

6

1

2

OK

T X /R X

L in k

IP

O K

T X /RX

L in k

IP

TXRX

LOG0

TXRX

LOG15

TXRX

LOG16

TXRX

LOG17

TXRX

LOG18

TXRX

LOG19

TXRX

LOG0

TXRX

LOG15

TXRX

LOG16

TXRX

LOG17

TXRX

LOG18

TXRX

LOG19

O K

T X /R X

L in k

1

T X /R X

L in k

2

T X /R X

L in k

3

T X /R X

L in k

4

T X /R X

L in k

5

T X /R X

L in k

6

R S 2 3 2

O K

T X /RX

Lin k

1

T X /RX

Lin k

2

T X /RX

Lin k

3

T X /RX

Lin k

4

T X /RX

Lin k

5

T X /RX

Lin k

6

RS 2 32

DS5100 3.2 GBpsFC Link 4 Gbps 8 ChannelRead 3.2 GBps , Write 2.5 GBpsPerformance depends on how many disk units use

2 Port HBA (4 Gbps x 2) 8 GbpsMulti Path DriverWrite Performance 700 - 750 MBpsFailover DriverWrite Performance 350 - 370 MBps

4 Gbps Trunk

GPFS NSD clientonboard 1 Gbps NIC

95 - 110 MBps

4 node Blade delivers2.8 GBps performance10 Gigabit Switch deliversfull performance

10 Gibabit EthernetSwitch