Scalar Decisions: Emerging Trends and Technologies in Storage

© 2013 Scalar Decisions Inc. Not for distribution outside of intended audience

Emerging Trends & Technologies Friday, February 28th, 2014

Neil Bunn, P.Eng. [email protected]


Ø  About Scalar Ø  NAND/Flash Memory Technologies §  Flash Arrays §  Hybrid Storage §  Flash-In-The-System

Ø Object Storage §  What is Object Storage §  Leaders in Object Storage §  Open Object Storage (Ceph/Openstack)

Ø  Software Defined Storage §  What is Software Defined Storage §  What Types of Implementations Are There? §  Implications of Software Defined Storage


}  Traditional Disk Arrays –  Largely commoditized –  Limited future performance/scale

improvements without change

}  Tape –  Great Medium, but limited innovation

}  Disks themselves –  4TB/5TB currently, with clear roadmap to

6/8TB, but limited performance improvements (5yrs plus!) (HAMR / BPR)

}  Parallel Filesystems –  Lustre & GPFS are cool. But far from

practical in the enterprise for broad application


}  Broad innovations causing product churn

}  Many new suppliers, many of whom may not exist in their present form in 5yrs (acquisitions, death, etc.)

}  Trends that are shaping the design of next-generation application platforms (ie: ‘Cloud’)

}  The concepts are the key, the benefits vary from client to client


INTRODUCTION


}  Agnostic Solutions Vendor –  ~120 Employees –  Cross Country Coverage –  Offices in Toronto, London & Ottawa –  ~100M+ In Annual Revenue –  Technically focused organization ﹘  ~40+ Technical Personnel

–  Own & Operate clusters of our own (over 1,200 servers and growing)


}  Some Of Our Partners


Source: Seagate Technology


1980

2013

2018+

Source: Multiple, IBM, HP, EMC, US Gov.


}  The sky is falling!

}  So, we know we have multiple problems: –  Performance issue (storage is always a bottleneck) –  Capacity issue (requirements continue to outstrip

traditional options) –  Management issue (divergent needs / classes) –  Locality issue (electrons take time…) –  Security issue (data protection costs a lot) –  Networking issue (back to electrons taking time…)


}  There are coherent roadmaps to address almost all of the recognized issues.

}  Triggers change in the enterprise.

}  Change is bad for some, but a terrific opportunity for others.

}  Storage focus: –  Disruptive Innovation based on: ﹘  Simple Management (Web / REST / Pervasive) ﹘  Higher Abstraction (Object / Software Defined) ﹘  Flash and future memory technologies (Flash / PCM / SCM) ﹘  Intensive services (Compression, Deduplication)


NAND / Flash


}  Originally slow, consumable, persistent storage

}  Rapid advancements in fabrication technologies leading to tremendous density improvements

}  Similar improvements in process leading to significant longevity improvements

}  Available in a variety of forms: –  Traditional ‘drive’ formats (SSDs) –  PCIe adapters (leveraging higher speed data

bus) –  Integrated DIMM packaging (Diablo controller) –  Chip Embedded Storage (next-gen)

}  Broadly these trends apply to traditional “NAND” an also to other non-volatile high speed storage for the future such as PCM / DWM, etc.


}  Commoditization of x86 technology, and robustness of new platforms with Linux and/or BSD derivatives results in low cost of entry for new storage vendors and the resulting proliferation of them.

}  Three main approaches: –  Accelerators –  Primary Storage –  Hybrid / Tiered Storage Systems


}  Simple, limited function

}  Typically Block devices

}  Connectivity options often: –  Fibre Channel –  Infiniband (SRP)* –  10/40G* Ethernet (iSCSI)

}  Often limited total scaling, for both cost and complexity reasons

}  Key Players: –  Fusion IO (ION Accelerator) –  IBM (FlashSystem) –  Violin Memory

}  Note: Non-Flash Accelerators as well! ie: Kove


}  Differentiation predominantly based on: –  Density –  Latency (large battle ground), much debate

about consistency –  Scalability

}  Largely a space relegated to niche problems at the moment with high costs ($500k+ for even moderate storage solutions)

}  Early movers differentiated on controller architecture, SSD vs. dedicated cards, etc.

}  Expect rapidly falling prices due to rapid commoditization, and loss of many players

Fusion-io Confidential

17

Fusion-io Confidential

18


}  Designed based on commodity platforms to combine flash (typically SSD) with low-cost bulk storage (ie: SATA drives)

}  Predicated on use-cases following an 80/20 or 90/10 rule (80% of work on 20% of data) or on low-rate-of-change data

}  Really the value is the caching infrastructure

}  Ideally suited to replicated uses such as VDI (linked clones) or other scale-out topologies where force-multipliers can be put in place against the SSD usage


}  Lower function devices

}  Typical file (NAS) devices or iSCSI type for supporting virtualization platforms

}  Heavily focused on virtualization environments

}  Connectivity options often: –  1/10/40G* Ethernet

}  Scale-out design, sometimes with aggregated administrative interface, but often based on VAAI for almost complete management from Virtualization interface.

}  Key Players: –  Nimble –  Tintri –  All traditional players in some “form”


Performance fuel gauge

Datastore performance stats


VM Name Latency

Real /me Graphs


}  High function to compete (compression, deduplication, etc.)

}  Typically Block devices

}  Connectivity options often: –  Fibre Channel –  Infiniband (SRP)* –  10/40G* Ethernet (iSCSI)

}  High scaling via clustering approaches, but typically require very-high performance internode interconnect (Infiniband)

}  Destroy the “tiering” approach of traditional vendors. There is data you need to access, and data you don’t. No in-between.

}  Key Players: –  Pure Storage –  Solidfire –  Whiptail (Cisco) –  Xtreme IO (EMC)


}  Significant focus on: –  Distributed data parity (similar to XIV in the disk

world) –  Potential for n+x protection in the future as

distributed data parity systems often using the basis for erasure coding under the covers

–  Block level deduplication ﹘  Differentiation on inline dedupe vs. post-process dedupe

–  Compression ﹘  This adds challenges in doing array to array replication,

depending on the compression method used

–  Thin provisioning ﹘  SSD/Flash math currently requires all of thin provisioning +

dedupe + compression against traditional disk array $/GB without those features





}  At the system level Flash / NAND or other non-volatile storage is appearing in four areas: –  Basic Disk Subsystem (SSD’s) –  PCIe / Card based accelerators (FusionIO) –  Memory-Coupled Systems (ULLtraDIMM) –  On-CPU NVRAM (futures, not necessarily

Flash based)

}  SSD’s are well understood, leveraging standard SAS/SATA interfaces and characterized by a current “Moore’s Law” associated with capacity/density improvements


}  Controller tech largely developed by Ottawa based “Diablo Technologies”

}  Commercialized by Smart Storage (SanDisk)

}  Shortly to be broadly available from IBM (Lenovo?) as eXFlash DIMMs –  With Intel E7-8xxx v2 processors enables 12+

TB memory systems

}  Likely the first of many implementations of another “mid-layer” of storage, closer to RAM than even PCIe storage


}  Opens up tremendous possibilities for all databases being mostly in-memory databases, and at least all indexes in memory

}  Realistic state-snapshotting of in-execution environments (ie: virtual machines) beyond just disk-state snapshotting

}  ULLtraDIMM represents the first phase (no pun intended) in transitioning to “Storage Class Memory”


Object Storage


}  The fringe of scale is driving a new look at storage for the future.

}  Facebook / Google / Flickr, etc.

}  Argonne, Lawrence Livermore, etc.

}  Different problem domains, similar storage issue – most POSIX filesystems don’t scale well.

}  Exceptions: –  ZFS, BTRFS

}  Others fake it… (Lustre, GPFS, etc.)


Object storage simplifies the back-end requirements on how data is housed, however also implies that the user or application-space technologies capture, handle and manage metadata associated with the objects.


}  Fully separate metadata and BLOB functions from each other.

}  Both in terms of performance, medium, and reliability characteristics.


SITE 3 SITE 1 SITE 2

Gateway

Simple Object Metadata stored and protected

by customer/application

Named Object (S3 Compatible)

Metadata stored and protected

OpenStack Swift Swift compatible API

Applications use an API for access.

Embedded Accesser Java Libraries

Custom Application


Dramatic increases in R&D funding and investment over the most recent decade have largely taken object stores from the corner cases of extreme-scale providers only to the more common use-case.

Growth has also been driven by the rapid uptake of “Big Data” type repositories.


•  Leaders mostly driven by a focus on heavy use of commoditized hardware with open API access

•  Initially supportable only by development organizations capable of writing internally to leverage object stores

•  Standardization driven by Amazon S3 adoption rates and similar API (OpenStack Swift) has enabled a rapidly maturing base of supported applications

Cap

abili

ties

Strategies


}  Simple –  Object storage solutions handle the vast

majority of failure scenarios in software –  Operationally, common repairs should

be “KISS” based

}  Inexpensive –  Success with Ceph/Swift tends to rest

on metrics of $/TB with reasonable access patterns

}  Standard –  Basic, industry standard components,

adaptable and interchangeable with commonly available hardware


}  Replicate industry success –  Organizations running large object

storage solutions focus on highly commoditized hardware

–  Custom / White-box is pervasive in Facebook, Google, eBay, Apple, etc. because of unique form factors & low price

–  Density rules –  Performance & Reliability required

but also predominantly provided by scale


}  Not focused on single object performance, designed for high-scale, large touchpoints. (ex. Flickr)

}  Typically use the concept of “eventually consistent” or “optionally consistent” – it takes data a while to replicate.

}  Limited update capability, just emerging, for byte-range updating of containers/objects, but still “eventually consistent”.

}  Makes for an excellent backup / archive target.

}  Now supported by many applications (Avere, Commvault, etc.)



}  Erasure Coding –  Mostly via Cleversafe / Scality –  Limited inclusion in open-source

products thus far, but on roadmaps

}  Filesystem / POSIX interface –  Being built into Ceph

}  API capabilities –  Vast differentiator, some (Swift /

S3) provide limited specific API, others broadly allow different approaches (Cleversafe / Ceph)


Erasure Coding… (Cleversafe / Scality)

SITE 3

1 2 3 4 5 8

SITE 1 SITE 2

DATA Accessers®

6 7 9

1

[Subset to read = THRESHOLD]

Slicestors®

1 [Total slices = WIDTH]

Data is virtualized, encrypted, and sliced using Information Dispersal Algorithms.

1

2 Slices are dispersed to separate disks, storage nodes and geographic locations.

3 Even with individual servers or entire sites down, real time bit perfect data is retrieved from a subset of slices.

2 3 4 5 86 7

Accessers®

DATA

SECURE RELIABLE SCALABLE

10 11 12


Software Defined Storage


What’s in common?

•  Uniformity •  x86 hardware •  Local Disks •  Software defined networks •  Flat Layer 2 Designs •  (Minimal overhead lighting… PUE

driven!)

What’s missing?

•  Fibre Channel Networks •  Disk shelves •  Physical SAN/NAS controllers •  Large UPS Systems •  Multitudes of Cabling

Prineville, Oregon Lenoir, North Carolina Chicago, Illinois

Next Generation Datacentres


In general characteristics of software-defined storage could include any or all of the following features

}  Pooling and abstraction of the logical storage services and capabilities from the underlying physical storage systems. This theme was also reflected in the technical term storage virtualization.

}  Automation with policy-driven storage provisioning with service-level agreements replacing technology details. This requires management interfaces that span traditional storage array products, as a particular definition of separating "control plane" from "data plane"

}  Virtual volumes or additional abstraction layers (objects, buckets, containers, volumes, etc.) supporting different characteristics

}  Commodity hardware with storage logic abstracted into a software layer. This is also described as a clustered file system for converged storage.

}  Scale-out storage architecture (typical).


}  Largely “Software Defined Storage” really just means storage virtualization.

}  But there is more to it than that, as truly defining storage only via software allows architects to break away from historical physical concepts.

}  Will be a long time before things change. Witness historical terms in server virtualization (virtual machine, virtual disk, virtual network) rather than “instance”, “datastore / container”, “data transport”, etc.


}  EMC technology meeting the definition of both an Object Store, and Software Defined Storage

}  Emerging capabilities to support the realities of heterogeneous storage demands in the enterprise

}  Primary focus is on automation & intelligent provisioning

VNX Isilon 3rd Party

VMAX

ViPR Data Services

ViPR Controller

Commodity

EMC ViPR Platform

Provisioning Self-Service Reporting Automation


What is Virtual SAN? Key Features

§  Converged compute + storage solution §  Utilizes DAS and server attached disks §  Auto-tiering between SSD and HDD §  Intelligent policy driven data placement

across the cluster §  vCenter-integrated, instant storage

provisioning

§  High performance storage at up to 50% lower cost

§  Radically Simple Storage – configure and manage without complex workflows

§  Designed for dynamic scalability and high resiliency

Clusters internal server disks to provide scalable shared storage with cloud agility and efficiency.

Customer Benefits

vSphere

Virtual SAN

Hard disks

SSD

…………….

Hard disks

SSD Hard disks

SSD

Distributed Storage Aggregated Datastore

vCenter Server


}  As data moves between VM2 and the LUN (shown with gold arrows), the FVP layer acts as a read and write-back cache while also mirroring the write-back data to another host node (shown with a white arrow).

}  This is done over the vMotion network by default, but you can change this to use a different network, or make a dedicated network, if desired.

}  If the virtualization administrator or DRS migrates VM2 from host ESXi A to B with a vMotion, the flash will already contain the hot write-back data and performance should remain relatively high.

}  In the case of write-through data, FVP will allow the host to read from a peer host’s flash tier to avoid going back to the SAN until the local flash has warmed up.


Virtual Machine/Virtual Disk

Virtual Storage Control Virtual Storage Control

Flash HDD

Fastest Performance

Localization, tiering, deduplication

Hypervisor Agnostic

vSphere, KVM, Hyper-V

Enterprise Storage

Clones, snapshots, replication,

compression, thin provisioning


NDFS Cluster Infinite Scalability •   Controller VMs create NDFS cluster •   Storage Pool created from all Disks (RF:2) •   NFS datastore presented to hypervisor (SMB for Hyper-V) •   I/O reads local, I/O writes synchronous


The great majority of Read/Write I/O is serviced locally •   vMotion or HA event triggers RF review and data locality •   Only accessed blocks are brought local (RF maintained) •   Background process (no flooding network)

© 2013 Scalar Decisions Inc. Not for distribution outside of intended audience 10

•   Replication Factor (RF) : 2 •   Synchronous replication of blocks between node OpLogs •   Fault Tolerance for Node failure •   All nodes participate in OpLog replication (no “hot nodes”) •   Asynchronous data drained to extent store (HDD)


Implications?




QUESTIONS?


THANK YOU.