Deep Dive: Cluster File System 6.0 new Features & Capabilitiesvox.veritas.com/legacyfs/online/veritasdata/SA B13.pdf · SA B13: Deep Dive: Cluster File System 6.x New Features and

SA B13 1

Deep Dive: Cluster File System 6.0 new Features & Capabilities

Carlos Carrero Technical Product Manager

Agenda

SA B13: Deep Dive: Cluster File System 6.x New Features and Capabilities 2

Storage Foundation Cluster File System Architecture 1

Producer-Consumer Workload Use Case 2

Fast Failover for Critical Applications Use Case 3

Clustered NFS use Case 4

Scale-Out Applications Use Case 5

What is Cluster File System


SW stack

App C

VCS

CFS Instance

Virtualization

App D

SW stack

App A

VCS

CFS Instance

Virtualization

App B

SW stack

App E

VCS

CFS Instance

Virtualization

App F

Private Network

Ethernet with VCS Protocols

Node 1 Node 2 Node 64

/MNT1

SAN (Fiber Channel, iSCSI or SAS)

Disk Arrays

Cluster File System Architectural Overview


Cluster-wide Low Latency Transport

Group Atomic Broadcast

Global Lock Manager

File system instance Cluster Volume Manager

Cluster File System

Node-instance CFS/CVM agents RAC, NFS, Apps

Veritas Cluster Server, agents for mounts/disk groups/volumes

Agenda







Producer and consumer workload


Financial data

EOF

Consumers constantly trying to read file

Producer application periodically

appending to file

Challenges

Concurrent access for many clients

Granular file IO

Cache coherency

File

http://www.iconfinder.com/icondetails/3784/128/blank_document_file_page_paper_icon

Global lock management


GLM Instance

Proxy

Master

Locks mastered by

this node

Locks held by this node

http://www.iconfinder.com/icondetails/50832/128/lock_private_safe_secure_icon






Global lock management (continued)


Name Node ID

GAB membership Lock type

….

Algorithm

Hash value

Name Node ID

GAB membership Lock type

….

Table

Node ID

Node ID

Node ID

Master

Master

Hash value





Node grants in Global Lock Manager


GLM instance

Master

GLM instance

CFS thread

CFS thread

CFS thread

Proxy 1

2

3 4

5

6 Minimize communications

overhead






Lock operation and cache coherency


GLM proxy Node grant NULL

Node A

GLM Proxy Node grant NULL

Node B

EX

Node A has EX grant

APP

PAGE-CACHE

GLM master for that lock

Load cache and cache coherency 2


GLM proxy A Node grant EX

GLM proxy B Node grant NULL SH

Node A has EX grant

SH

Node A has SH grant

Node B has SH grant

WRITE APP

PAGE-CACHE

READ APP

PAGE-CACHE

Node B

GLM master for that lock

Range locks


FILE

W1 W3

W2 Serial Access

File level read/write LOCK

Serial Access

Inode locking does not allow parallel access



Range locks (continued)


W1 W3

W2 Parallel Access

Serial Access

FILE

Range lock Range lock

R1 R2

Range locks allow parallel access to the same file




Quiz

SA B13: Deep Dive: Cluster File System 6.x New Features and Capabilities 14 14

TRUE or FALSE: To maintain cache coherency, CFS sends the data blocks over the network to the requesting node

What mechanism does CFS use to minimize node to node communications overhead?

FALSE: Just request a lock

Node Grants

Agenda







Fast-failover for critical applications


Failover as fast as application restart

Fast failovers

• Achieve sub-minute failovers - 90% faster

failover times

Failo

ver

INST1 INST1

http://www.iconfinder.com/icondetails/34218/128/add_cross_delete_edit_exit_remove_icon

Service recovery time delays


Failo

ver

INST1

Database

Start database processes

Check file system

Mount file system

Import disks

Detect failure

Unmount file system

Deport disks

http://www.iconfinder.com/icondetails/40763/128/

http://www.iconfinder.com/icondetails/8938/128/cog_gear_preferences_system_icon


Service recovery time delays


Cluster File System Failo

ver

INST1

Database

Start database processes

Check file system

Mount file system

Import disks

Detect failure

Unmount file system

Deport disks

CFS mount points

CVM shared disk groups

GLM recovery




Membership arbitration and data fencing

SA B13: Deep Dive: Cluster File System 6.x New Features and Capabilities

19

19

INS1 INS2 Interconnect

Node 2: Win Node 1: Lose Immediate link

failure detection

SCSI3 coordination

disk

Data Protection Mechanism

Race upon server failure to determine node survival

Winner ejects the other node via SCSI3 key on data disk

Node that lost the race triggers a kernel panic

Kernel-based fencing algorithm

Data protection







Cluster File System


APP

Mitigate Infrastructure Failure Protect services from storage failure Utilise alternate I/O paths Eliminate Data Loss During Failover

I/O Shipping

I/O Shipping





Cluster File System


Increase uptime A node can join CVM if disk is accessible

from other nodes

Node is able to join cluster

DiskGroup A DiskGroup B

Node lost visibility for DiskGroup B storage

Storage is present through other node

Cluster Volume Manager Asymmetry










Improved Performance Sequential logging based DRL Minimize DRL updates DRL used as staging for DCO update

Better Scalability Independent of volume size &

workload Per node DRL map with async-

update to DCO

Rw1 Rw2 Rwn

Circular Region Log

Faster Mirror Resync

ESX


RDM-P SCSI3 PR for Data Protection vMotion and DRS are prohibited

VMDK Take advantage of ESX multi-writer capability Provide cache consistency Allow vMotion and DRS Use Coordination Point Servers

VMFS + multi-writer

VMDK

CFS

CPS ES

X

ESX

ESX

ESX

ESX

CFS

SCSI3 Fencing

CFS in VMware Environments








Quiz


Name one feature that allows CFS to make data immediately available

Which technology provides membership arbitration & data protection

CFS Mount Points, CVM Shared Disk Groups, GLM Recovery

Fencing

Agenda







Clustered NFS


CFS lock management

26

Availability and Scale for NAS

• Scale NFS across multiple low-cost

nodes

• Scale performance separately from storage capacity

• Complements NFS v3 advisory locks with

CFS/GLM locks

Better price/ performance vs. NAS arrays

Cache Cache Cache Cache

NFS NFS





http://www.iconfinder.com/icondetails/18147/128/computer_monitor_pc_screen_icon







Directory sharing with partitions


Directory

Directory level read/write LOCK

W1

W2 W3

Serial Access

W1

W2

W3

t

Locking at directory level limits performance




http://www.iconfinder.com/icondetails/9068/128/directory_inode_icon

Directory sharing with partitions (continued)


Parallel Access

W1

W2 W3 W1 W2 W3 T/3

Directory

Directory Lock Partition

Directory Lock Partitions support concurrent I/O




http://www.iconfinder.com/icondetails/9068/128/directory_inode_icon




Delegation


File System

Node gets a share of resources to work with

Extent Allocation Units

Inode Allocation Units


Inode Allocation Units


Inode Allocation Units Inode Allocation Units


http://www.iconfinder.com/icondetails/48976/128/database_storage_icon








Block locking


inode inode inode inode

Block name

GLM proxy

GLM master

Single request block lock

Block allocation

Delayed allocation during writes


Me

mo

ry

Write

Write

Write Single request

File System




Quiz


How does CFS share resources among nodes?

Delegation

Agenda







Scale-out applications


Performance for parallel apps

•Minimal application overhead

• File access scales linearly as additional nodes are added

• Single namespace for database copies

Cluster File System Cluster File System

http://www-306.ibm.com/software/websphere











Scale-out applications (continued)


Near Local Performance

Decentralized Decision Making

Dynamic Ownership

Minimized Messaging

Data & Metadata

Partitioning

Range Locking

Node Grant Caching

Per-Node Transaction Independence

Delegation

Partitioning

Quiz


What is the primary performance design goal for CFS?

Near Local Performance

Cluster File System Major Use Cases and Benefits


Scale Out Applications

Clustered NFS Fast Failovers

Producer-Consumer

Reduce Storage Costs & Improve Performance

Improve Service Recovery Times

Cost Effective and Scalable NAS solutions

Scalable Architecture with Increasing Business Demands

Value Proposition Differentiator

38 SA B13: Deep Dive: Cluster File System 6.x New Features and Capabilities

Integrated Solution Linear Scalability

ASL/HW Support Avoid Data Migration

Storage Optimization Subminute Failover

Storage Foundation Cluster File System HA


Thank You!

[email protected] +34 659 352 408

Documents

Deep Dive: Cluster File System 6.0 new Features & Capabilitiesvox.veritas.com/legacyfs/online/veritasdata/SA B13.pdf · SA B13: Deep Dive: Cluster File System 6.x New Features and