Upload
glennboynton
View
138
Download
0
Embed Size (px)
DESCRIPTION
IBM® Edge2013 - A Brief History of SVC and Storwize Family
Citation preview
© 2013 IBM Corporation
A Brief History of SVC & Storwize FamilyWhat, How and Why?
Barry Whyte – IBM Master Inventor – Virtual Storage Performance Architect
June 2013
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
2
Session goals and welcome
This session presents a brief history of IBM SAN Volume Controller (SVC) and IBM Storwize family.
Starting back in 1999, it will explain why IBM decided to develop SVC, and the research projects that lead to the clustering architecture and the I/O processing architectures.
The session goes on to describe the benefits of storage virtualization and how the software and hardware have evolved over time to where we are today, with the same software running across twelve different hardware platforms in as many years.
From a field-hardened code base to a modern storage appliance/controller architecture that is the fastest thing in anyone's SAN -- even with today's SSD devices – you'll learn why these IBM offerings lead the way in storage virtualization.
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
Professor Steven Hawking's seminal work...
By the end of this session, I'll show how we :
– We are 14+ years ahead of the recent software defined storage trend
– Set several trends in the storage industry• First to use and prove commodity hardware as a storage platform• First to implement integrated Flash technology
A Brief History ...
Recently added a time-machine to the Storwize family
– Without the need for a Flux-capacitor...
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
4
From Direct Attached to expanding SANs
A Brief History of SVC and Stowize Family
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
Storage Industry – IBM circa 1995 - Direct Attached Storage
Before the mid 1990's most storage systems were direct attached
IBM Serial Storage Architecture– Developed by the IBM UK Hursley Storage team (who develop the Storwize Family)– Dual serial disk loops per adapter – 96 drives per loop - 160MB/s
• Spatial re-use – only seen now in todays SAS topologies• 8 way server attachment• 2 way RAID architecture
– Over 95% attachment rate to IBM RS6000 server platform (Power Systems)– Original storage disk adapters in IBM Enterprise Storage System (ESS)
• Evolved into IBM DS8000 with FC-AL interfaces using same adapter firmware– Returned over $2 billion revenue between 1996 and 2003
SANs started to make direct attach obsolete, when many hosts could attach to one or more disk system
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
Moore’s Law
60-100% CGR
60-100% CGR
25% CGR
25% CGR
Gbi
t / s
q uar
e in
c h
'Ye canny change the laws of physics, Captain!
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
Back in 1999 we could see the second slow down was coming
SAN complexity had grown “out the box”Storage itself was becoming a commodityStoage islands – wrong place – wasted capacity?
At the same time :– Custom storage function silicon development was too slow and costly – 3yrs etc– Commodity (x86) silicon was proving to be enterprise ready
Questions we asked :– How can we tame heterogenous vendor SANs?– How can we consolidate the storage functions?– How can we make better use of what customers already have?– How can we ride the x86 technology wave for years to come?
State of the Storage Industry – circa 1999
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
More questions... can something help answer them all?
In addition to the technology questions, several business and operational questions were also being asked :
– The continual growth in data center costs– Inability of IT organization to respond quickly to business demands– Poor availability or service levels– Lack of skilled staff for storage admin tasks– Poor asset utilisation
So while SANs had opened the potential for more flexibility– In reality, they opened up another set of problems
Could we radically change the way we think about storage?
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
Storage Virtualization – an often over-abused term
Server virtualization, wasn't really around in 1999, other than on mainframe systems
But what if we could virtualize all those disk systems– Consolidate the disparate capacity
• Make better use of whats already there• Help slow down the continual need to “buy more disk”
– Provide mechanism to migrate around downtime– Implement a common set of storage functions
But also make sure we answered the technology questions– Use a commodity hardware base– Ride Intels technology wave– Write the intelligence into the software
We'd need a flexible – future proof software architecture
+ Software Function+ Software Function
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
10
Host Zone
Storage Zone
Host Zone
Storage Zone
In-band Appliance• Caching and scale-out nodes enhances performance• Consolidates Copy Services• Grows with business needs
Array Based• No additional HW required• Not scalable - limited by box performance• Consolidates Copy Services
Out-of-band – Switch Based• Potentially scales to large networks• Array-based or Switch-based Copy Services
Storage Zone
Host Zone
Storage Virtualization – The Approach War
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
IBM Research – 1999 Storage Software Architecture
1999 IBM Systems Journal– The software architecture of a SAN storage
control system
Cluster of processing nodes
Abstraction layers– Interface layer
• Abstract the I/O from the protocol
• Common platform I/O (plio) definition
– Peer to peer comms• Abstract inter-node comms
– Common Configuration API• Object manipulation
– Common cluster API model• Cluster state interfaces
Stack of I/O components– Each implements a specific,
encapsulated storage function– Upper/Lower common plio interface
I/O Component
I/O Component
I/O Component
I/O Component
Clusterin g
Configur ation
Peer C
o mm
unic ations
Interface Layer
User Interfaces
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
12
Cache
FlashCopy
Replication
SCSI Target
Mirroring
Thin Provisioning
Virtualization
Forwarding
Forwarding
RAID
SCSI Initiator
Forwarding
Easy Tier
Compression
A Brief History of SVC and Storwize Family
The Birth of SVC
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
All virtualized storage common across entire cluster (visible to all nodes)– Controller volumes (LUNs) grouped into common attribute storage poolsC
Control Enclosure
Storage pools of similar performing / RAIDed LUNs
All storage pools visible to all SVC nodes
SVC nodes deployed as active/active caching pairs
Virtual volumes provisioned from striped capacity from a given storage pool
Associated with a caching node pair
IBM SVC – Inband Storage Virtualization Concept ~2000
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
IBM SVC v1.1.0 – 2003
2000 – Hursley development begins work on turning the Almaden research work into an in-band virtualization appliance.
By July 2003 the first release was ready for GA
– 4-way node clustering– FlashCopy and MetroMirror
functionality – Online storage migration
Initial controller support– IBM DS and FAStT product ranges– EMC DMX and Clariion
Latest Hardware (4F2)– X335 Server Base– 2x 1core 2.8Ghz Xeon (Prestonia)– 533MHz FSB– 4 GB Cache– 4x 2Gbit Fibre Channel
Legend
Virtual
Mdisk
Cache
FlashCopy
Replication (MM)
SCSI Target
Virtualization
SCSI Initiator
Supported Hardware
SVC ● 4F2 Node
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
IBM SVC – v2.1.0 and v3.1.0 - 2004/2005
Support for up to 8-way node clusteringOS and controller support ++
2004 Latest Hardware (8F2)– X336 Server Base– 2x 1core 3.0Ghz Xeon (Irwindale)– 800MHz FSB– 8 GB Cache– 4x 2Gbit Fibre Channel– ~ 25% IOPs boost (FSB)
2005 Same as above except: (8F4)– 4x 4Gbit Fibre Channel– 2x GB/s boost
Internal code changes – DV (DA)-V– Political debates
Legend
Virtual
Mdisk
Cache
FlashCopy
Replication (MM)
SCSI Target
Virtualization
SCSI Initiator
Supported Hardware
SVC ● 4F2 Node● 8F2 Node● 8F4 Node
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
IBM SVC – v4.1.0 - 2006
Continued OS and controller support ++
First major new set of functional enhancements
Multiple-target FlashCopy– Up to 256 targets from single volume– No overhead (only really 1 copy unless split
again)
GlobalMirror– Added asynchronous replication functions– Using same replication layer– Sync or Async at create time Legend
Virtual
Mdisk
Cache
FlashCopy
Replication
SCSI Target
Virtualization
SCSI Initiator
Supported Hardware
SVC ● 4F2 Node● 8F2 Node● 8F4 Node
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
IBM SVC – 2007 – Multi-core CPUs
Late 2006 saw the evolution away from the GHz race– More cores per chip– Early samples of Intel Woodcrest dual core CPU packages– Fundamental change to they way we had previously
allocated resources
Cores, Threads and Spinlocks– SVC up till now had run two threads, one pre CPU– Spinlocks relatively low impact– While mutli-threaded, base platform code
– With four threads, spinlocks, or rather spinlock contention wasting ~15% CPU cycles – idle cores...
Without giving too much IP away– Binding of threads, ports and cores– Increased threading, batching while locked...
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
SVC and Storwize Family Architecture
Since 2007 each node has at least 4 cores– Piggy-back on Intel development– Latest and greatest PCIe bus/memory
Generally run a single “fibre thread” per core– Queues of lightweight fibres– No interrupts or kernel code– Thread polls hardware from user mode
One or more “driver polling functions” per thread– Each port has its own polling function– Some management functions – like PCIe driver– Some kernel functions – like iSCSI
Object ownership maybe distributed across cores– Attempt to avoid context switch between threads if possible
Fibre Thread (bound to a single core)
FC Port polling fn
SAS Port polling fn
iSCSI Port polling fn
Fibre Queue
Fibre
Fibre
Fibre
Fibre
Fibre
Fibre
...
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
IBM SVC – v4.2.0 - 2007
Continued OS and controller support ++
First release of new multi-thread changes– 2x IOPs boost– 33% boost to existing nodes
Enhanced FlashCopy– Incremental– Cascaded multi-target
Latest Hardware (8G4)– X3350 Server Base– 2x 2core 2.33Ghz Xeon (Woodcrest)– 1333MHz FSB– 8 GB Cache– 4x 4Gbit Fibre Channel
Legend
Virtual
Mdisk
Cache
FlashCopy
Replication
SCSI Target
Virtualization
SCSI Initiator
Supported Hardware
SVC ● 4F2 Node● 8F2 Node● 8F4 Node● 8G4 Node
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
IBM SVC – v4.3.0 - 2008
Continued OS and controller support ++
Forwarding for asymmetric access
Volume Mirroring– Two copies of a single volume stored on
potentially disparate storage controllers– Like LVM mirroring in the SAN– Migration using split-mirror
Thin Provisioning– Fine grained thin provisioning (32KB - 256KB)– Use volume mirroring for thick to thin migrate– Mix within pools of thick and thin– Volume and pool usage warnings
New hardware – 8A4 – Entry Edition
Legend
Virtual
Mdisk
Cache
FlashCopy
Replication
SCSI Target
Virtualization
SCSI Initiator
Supported Hardware
SVC ● 4F2 Node● 8F2 Node● 8F4 Node● 8G4 Node● 8A4 Node
Mirroring
Thin Provisioning
Forwarding
Forwarding
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
21
Cache
FlashCopy
Replication
SCSI Target
Mirroring
Thin Provisioning
Virtualization
Forwarding
Forwarding
RAID
SCSI Initiator
Forwarding
Easy Tier
Compression
A Brief History of SVC and Storwize Family
The 1,000,000 IOPs Era (but we were first!)
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
22
2008 - Quicksilver - Scale-out Flash with SVC
Quicksilver was an SVC and Flash technology demonstration in 2008– Proved the capability of a scale-out clustered storage system– Proved SVC stack was Flash ready... already
Achieved 1.2 million IOPS (70/30 4KB), at under 1 ms response time
Technology demo used– 14 current generation SVC nodes (8G4)– 32 System X servers with Flash PCIe cards
– Each of these running a cutdown SVC virtualizing local block Linux /sd devices
– Presenting the /sd devices as “mdisks” to the 14 node SVC cluster
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
IBM SVC – v5.1.0 - 2009
Continued OS and controller support ++First integrated Flash support in virtualizer
– Integrated SAS HBA / driver• What could we use that for in the future...
– First single system capable of more than 1M IOPS– Low latency key factor – 50us overhead
Up to four Cluster Partnerships for Replication iSCSI – 1Gbit
– SVC software upgrade strategy– All users, all hardware now iSCSI capable
Stretched Cluster (Split Cluster) HAZero detect on new CF8 HardwareLatest Hardware (CF8)
– X3350M2 Server Base– 1x 4core 2.4Ghz Xeon (Nehelan)– QPI replaces FSB– 24 GB Cache– 4x 8Gbit Fibre Channel
Legend
Virtual
Mdisk
Cache
FlashCopy
Replication
SCSI Target
Virtualization
SCSI Initiator
Supported Hardware
SVC ● 4F2 Node● 8F2 Node● 8F4 Node● 8G4 Node● 8A4 Node● CF8 Node
Mirroring
Thin Provisioning
Forwarding
Forwarding
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
24
Cache
FlashCopy
Replication
SCSI Target
Mirroring
Thin Provisioning
Virtualization
Forwarding
Forwarding
RAID
SCSI Initiator
Forwarding
Easy Tier
Compression
A Brief History of SVC and Storwize Family
Introducing the Family
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
What we need is an Enterprise stack for all...
++
=
If you built it.. they will come...
Software function – everyone wants it– What if we could take what we've done for the last
ten years, and make it available to all..– SVC is just software running on a server– But what about drives, and RAID
• Remember that SAS driver• Remember our SSA history
Build a dual-controller integrated disk system
Time to market...– Just a year away...
Lessons learnt...– One code base, one build, one binary...
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
IBM SVC, Storwize – v6.1.0 - 2010
Continued OS and controller support ++
RAID– Taken from SSA RAID firmware– SAS already added for CF8 Flash Drives
• New SAS network topologiesEnclosure Services
– SES integration, RASEasyTier
– Optimal use of 5% Flash capacityPCIe Driver
– Inter-node communications inside enclosure
Latest Hardware – V7000
• Dual node SVC inside 2U
Enclosure with disks
Legend
Virtual
Mdisk
Cache
FlashCopy
Replication
SCSI Target
Virtualization
Supported Hardware
SVC ● 4F2 Node● 8F2 Node● 8F4 Node● 8G4 Node● 8A4 Node● CF8 Node
Mirroring
Thin Provisioning
Forwarding
Forwarding
RAID
SCSI Initiator
Forwarding
Easy Tier
Supported Hardware
Storwize● V7000
Drive
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
IBM SVC, Storwize – v6.2.0, v6.3.0 - 2011
Continued OS and controller support ++
10Gbit Ethernet– iSCSI 10Gbit support on SVC and V7000
Storwize V7000 – 480 drive clusters
Storwize V7000 Unified– NFS, CIFS, SMB, HTTP, SCP file modules
Latest Hardware (SVC CG8)– X3350M3 Server Base– 1x 6core 2.4Ghz Xeon (Westmere)– QPI 6.4GT– 24 GB Cache– 4x 8Gbit Fibre Channel
Legend
Virtual
Mdisk
Cache
FlashCopy
Replication
SCSI Target
Virtualization
Supported Hardware
SVC ● 8F2 Node● 8F4 Node● 8G4 Node● CF8 Node● CG8 Node
Mirroring
Thin Provisioning
Forwarding
Forwarding
RAID
SCSI Initiator
Forwarding
Easy Tier
Supported Hardware
Storwize● V7000● V7000 Unified
Drive
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
Compression
IBM SVC, Storwize – v6.4.0, v6.4.1 - 2012
Continued OS and controller support ++
FCoE Support– Using existing 10Gbit HBAs
Volume Migration– NDVM – Volume move between IO/Groups
without disruption
960 Drive V7000 Clusters
Real-time Compression– Temporal Locality– (I promised you a time machine)
Storwize V3500, V3700– Entry level controller
Legend
Virtual
Mdisk
Cache
FlashCopy
Replication
SCSI Target
Virtualization
Supported Hardware
SVC ● 8F2 Node● 8F4 Node● 8G4 Node● CF8 Node● CG8 Node
Mirroring
Thin Provisioning
Forwarding
Forwarding
RAID
SCSI Initiator
Forwarding
Easy Tier
Supported Hardware
Storwize● V3500● V3700● V7000● V7000 Unified
Drive
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
29
Cache
FlashCopy
Replication
SCSI Target
Mirroring
Thin Provisioning
Virtualization
Forwarding
Forwarding
RAID
SCSI Initiator
Forwarding
Easy Tier
Compression
A Brief Future of SVC and Storwize Family
The Future...
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
2003 We were Here →
Cache
FlashCopy
Virtualization
1st Hardware model3 Features :
Metro Mirror, FlashCopy and Migration (Virtualization)
1 Block protocol :Fibre Channel
Potential to be:– Extensible– Flexible– Augmentable
Was the architecture right?
Configur ation
Peer C
o mm
unic ations
Interface Layer
Clusterin g
SCSI Initiator
SCSI Target
Replication (MM)
Fibre Channel
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
2013 We are Here →
Compression
Cache
FlashCopy
Virtualization
Mirroring
Thin Provisioning
Forwarding
Forwarding
RAID
Easy Tier
12 New hardware models running the same software*
8 new major features :Global Mirror, Volume Mirrroing, Thin Provisioning, Stretched Cluster, EasyTier, Compression, RAID, Global Mirror with Change Volumes
6 Major and 17 Minor Software releases :
(1.1, 1.2, 2.1, 2.1.1, 3.1, 4.1, 4.1.1, 4.2, 4.3, 4.3.1, 5.1, 6.1, 6.2, 6.3, 6.4, 6.4.1 … 7.1)
4 new block protocols:iSCSI, FCoE, PCIe, SAS
Extensible, flexible, augmentable - YES!
Get the platform and software architecture and philosophy right on day one, and there are no limits or boundaries
* Only 4F2 hardware limited to running no later than 5.1 Software due to 32bit CPU
Configur ation
Peer C
o mm
unic ations
Interface Layer
Clusterin g
SCSI Initiator
Forwarding
SCSI Target
Replication
Fibre Channel
iSCSI
FCoE
SAS
PCIe
© 2013 IBM Corporation
A Brief History of SVC and Storwize Family
SVC and Storwize Family – Into the second decade
CG8 Hardware Enhancements - 1H13– 2x FC ports (16x 8Gbit per node pair)– 12 core CPU per node
• 3x Compression throughput
FlashSystem + SVC– Continues the evolution of SVC as the ultimate performance and function platform– Greater than 1M Production IOPs in half a rack
More information this week :– SV-1038A Optimizing Virtual Storage Performance– SV-1218A IBM Storage Virtualization Product Directions– PU-1670A IBM SAN Volume Controller and Storwize Family: What's New