Improving Cloud Storage Cost and Data Resiliency with ... · PDF fileImproving Cloud Storage...

Improving Cloud Storage Cost and Data Resiliency with Erasure Codes

Michael Penick

Commodity Storage

Hosting storage FTP backup Goals Inexpensive (use “commodity” hardware) Resilient to failures Highly available Customizable

MogileFS

Open source distributed filesystem Written by Brad Fitzpatrick No single point of failure Automatic/Asynchronous file replication Shared-Nothing design (disks) Local filesystem agnostic Flat namespace

Tracker

Storage Node

MetadataDB

MogileFS

Tracker

Storage Node

MetadataDB

Clients

“NebulaFS”

Large file support Offsite Replication Self-healing Data retention C++ client (PHP and Perl SWIG wrappers) Metadata Sharding Range GETs

“NebulaFS”

Tracker / Storage Node

Storage Node

MySQL Tracker /

Storage Node

Clients

FTP Backup

FTP Presentation (Net::FTPServer)

VFS DB

NebulaFS

Metadata DB Super Nodes Storage Nodes

NebulaFSAPI

Widely Applicable

“Storage service” (REST) layer New Product Integrations Online File Folder (videos and images) Website Builder/ Photo Album Go Daddy Cloud Servers (snapshots) Email …

Object Storage

RESTful Presentation (S3, GDCS)

VFS DB

NebulaFS

User DB

NebulaFSAPI

5.39% 8.51%

83.89%

1.20% 1.01% ~3.25 PB

Aries FTP WST/PA OFF VDC Other

1.80% 2.56%

38.44%

55.44%

~10.8 PB

Aries FTP WST/PA OFF VDC Email Other

The Problem

NebulaFS = Inexpensive, resilient, highly available storage

Problem: Disk drives fail...a lot. F = mean time to failure In a system of n devices our mean time failure

is: F/n Solution: Replicate the data

Replication

01010101011010101011010101010101010101101010110010101011010011010101010110100101010101101010101….

Success!

Duplicate Replicate

Copy 1

Copy 2

Copy 3

Copy 4

Disk 1 … Disk 2 Disk n

Replication

Simple and effective Durability:

99.99999% over 1 year (or 0.1 of 1 million objects) 99.99% over 3 years (or 100 of 1 million objects)

Problem: 100 % overhead per copy +300% overhead for 3 onsite and 1 offsite

copy There has to be a better way.

Erasure Codes

Forward error correction code Add redundant data (codes) to message so that it

can be recovered Where’s EC used?

Optical media Media streaming File Systems (RAID-6, several distributed FS, …)

Erasure code (write)

Divide 01010101011010101011010101010101010101101010110010101011010011010101010110100101010101101010101….

Encode

010101010110101010110101010101

010101011010101100101010110100

110101010101101001010101011010

101010000000000000000000000000

101010010101101010101010101010

101010001010101010100101101001

Disk 1

Disk 2

Disk n

Copy 1

Copy .75

101010001010101010100101101001

Erasure code (read)

010101010110101010110101010101

010101011010101100101010110100

110101010101101001010101011010

101010000000000000000000000000

101010010101101010101010101010

101010001010101010100101101001

Verify Decode

01010101011010101011010101010101010101101010110010101011010011010101010110100101010101101010101….

Disk 1

Disk 2

Disk n 101010001010101010100101101001

Erasure codes

What? k – number of original pieces m – number of redundant pieces (codes)

How? k = 4, m = 3: only 75% overhead (3 failures) k = 10, m = 6: only 60% overhead (6 failures) k = 9, m = 3: only 33% overhead (3 failures)

AKA: k = 10, m = 6 10 of 16

Trade-offs (positive)

Better resilience to failure Durability for 10 of 16:

99.9999999999% over 1 year (or 0.000001 of 1 million objects) 99.99999% over 3 years (or 0.1 of 1 million objects)

Durability for 9 of 12: 99.99999% over 1 year (or 0.1 of 1 million objects) 99.99% over 3 years (or 100 of 1 million objects)

Significant savings (includes a full offsite copy) 10 of 16: (4 – 2.60) / 4 = 35% savings (60 % w/o offsite) 9 of 12: (4 – 2.33) / 4 = 42% savings (67% w/o offsite)

Trade-offs (negative)

Computationally expensive Increased number of IOPS Complexity (additional metadata) More nodes and connections

Erasure Codes

Optimal erasure code Any k pieces of the message can recover the message Reed-Solomon (and Cauchy Reed-Solomon)

Libraries (Jerasure, Zfec, Luby, librs,…) Stability/Performance Evaluation

Paper – “A Performance Comparison of Open-Source Erasure Coding Libraries for Storage Applications” http://web.eecs.utk.edu/~plank/plank/papers/CS-08-

625.pdf

EC Libraries - zfec

Reed Solomon Written in C (Python and Haskell bindings) Download: http://pypi.python.org/pypi/zfec Documentation is the source code

EC Libraries – zfec Encoding

EC Libraries – zfec Decoding

EC Libraries – zfec Decoding contd.

EC Libraries – zfec Decoding cond.

k = 6, m =2, erasures = { 0, 2, -1 }

index = { 6, 1, 7, 3, 4, 5 }

inpkts

coding 0

data 1

coding 1

data 3

data 4

data 5

outpkts

data 0

data 2

EC Libraries - Jerasure

Reed Solomon, Cauchy Reed Solomon, and Minimal Density Codes

Written in C (no bindings) Download:

http://web.eecs.utk.edu/~plank/plank/papers/CS-08-627.html

Good documentation and examples

EC Libraries – Jerasure Encoding

EC Libraries – Jerasure Decoding

EC Libraries – Performance

w = 8 w = 16 w = 32

Encoding

Jerasure RS Jearsure CRS zfec

EC Libraries – Performance

0 100 200 300 400 500 600 700 800 900

w = 8 w = 16 w = 32

Decoding

Jerasure RS Jearsure CRS zfec

Integration – EC Library

EC library (Phase I) Read/Copy Write Repair

Inputs/Outputs abstracted boost::asio (HTTP) PHP/Perl bindings Random access reads (i.e. Range GET) Data validated/corrected on-the-fly

Integration – EC Library Writes

01010101011010101011010101010101010101101010110010101011010011010101010110100101010101101010101….

010101010110101010110101010101

010101011010101100101010110100

110101010101101001010101011010

101010000000000000000000000000

101010010101101010101010101010

101010001010101010100101101001

Disk 1

Disk 2

Disk n 101010001010101010100101101001

Integration – EC Library Failures

Integration – EC Library Reads

010101010110101010110101010101

010101011010101100101010110100

110101010101101001010101011010

101010000000000000000000000000

101010010101101010101010101010

101010001010101010100101101001

01010101011010101011010101010101010101101010110010101011010011010101010110100101010101101010101….

Disk 1

Disk 2

Disk n 101010001010101010100101101001

Integration – EC Library Copy

010101010110101010110101010101

010101011010101100101010110100

110101010101101001010101011010

101010000000000000000000000000

101010010101101010101010101010

101010001010101010100101101001

Disk 1

Disk 2

Disk n 101010001010101010100101101001

k Disk 1

Disk 2

Integration – EC Library Repair

010101010110101010110101010101

010101011010101100101010110100

110101010101101001010101011010

101010000000000000000000000000

101010010101101010101010101010

101010001010101010100101101001

Disk 1

Disk 2

Disk n 101010001010101010100101101001

Integration – Reads/Writes

Integration – Reads/Writes DB

Increased number of “file_device” entries Decreased number of “file” entries

Change the meaning of “class”

Integration – Reads/Writes DB

Integration – Write

Integration – Read

Integration – Recovery

Lessons Learned

CRC32 can be slow Intel’s Slicing-by-8 Algorithm

Block size can limit your smallest file size Lighttpd doesn’t support “Transfer-Encoding:

chunked”

Performance Test Setup

6 super nodes (tracker and storage node) 180 drives Drives not distributed i.e. not 30 drives per node EC strips maximally distributed

Performance Test Results

0 10 20 30 40 50 60 70 80

1KB 1MB 16MB 32MB 64MB 128MB 256MB

File Size

Writes

ec_1_of_2 ec_6_of_9 ec_9_of_12 ec_10_of_16 replication

Performance Test Results

0 20 40 60 80

100 120 140 160 180 200

1KB 1MB 16MB 32MB 64MB 128MB 256MB

File Size

ec_1_of_2 ec_6_of_9 ec_9_of_12 ec_10_of_16 replication

Migrations

RESTful Presentation (S3, GDCS)

VFS DB

NebulaFS

User DB

Migration Script

VFS NebulaFSAPI

Future

Finish Phase III Repairs Offsite copy

Net new growth Optimizations Open source

Questions

Thank You!

Improving Cloud Storage Cost and Data Resiliency with ... · PDF fileImproving Cloud Storage...

Documents

Regional Resiliency and Security - Department of … Resiliency and Security: Compressed Air Energy Storage Department of Energy EAC June 8, 2017 . 2 Magnum’s Renewable CAES Location

Dell PowerEdge R720xd 6,000 Mailbox Resiliency Microsoft Exchange 2013 … · 2020-01-22 · 7 Dell PowerEdgeR720xd 6,000 Mailbox Resiliency Exchange 2013 Storage Solution 3 Solution

CANADA–U.S. ENHANCED RESILIENCY EXPERIMENT SERIES … · CANADA–U.S. ENHANCED RESILIENCY EXPERIMENT SERIES “CAUSE RESILIENCY”: A Canada–U.S. Resiliency Experiment (CAUSE

IBM PRODUCT REFERENCE GUIDE - Broadcom Inc. · Business Resiliency, and Data Mobility solutions. GEN 4 ENTRY SWICTH IBM GEN 5 STORAGE NETWORKING PRODUCTS IBM GEN 6 STORAGE NETWORKING

EMC Celerra NX4 (500 User) Mailbox Resiliency Storage ... · PDF fileESRP Storage Program EMC Celerra NX4 (500 User) Mailbox Resiliency Storage Solution for Microsoft Exchange 2010

Actifio Resiliency Director Collector User Guidedocs.actifio.com/9.0/PDFs/ActifioResiliencyDirectorCollector.pdfActifio Resiliency Director Collector 8.0.0 Actifio Resiliency Director

WHITE PAPER Infinidat Storage Architecture · the true meaning Software Defined Storage (SDS) Performance Simplicity Reliability Cost Design for resiliency When designing for scale,

HHE Report No. HETA-87-0311-2087, Penick Corporation, Newark

Sun ZFS Storage 7320 Appliance 10,000 Mailbox Resiliency Exchange 2010 Storage Solution · Sun ZFS Storage 7320 Appliance 10,000 Mailbox Resiliency Exchange 2010 Storage Solution

Dell Compellent Storage Center 6.5 SC4020 10,000 Mailbox Exchange 2013 … · 2020-03-15 · 6 Dell Compellent Storage Center 6.5 SC4020 10,000 Mailbox Exchange 2013 Resiliency Storage

Sun ZFS Storage 7420 Appliance 32000 Mailbox Resiliency Exchange 2010 Storage Solution

Cyber Resiliency Engineering Aid - Cyber Resiliency Techniques

Summarydownload.microsoft.com/download/2/5/8/258E73C5-D851-4654... · Web view... storage performance and resiliency similar to a traditional SAN using Microsoft Storage Solution

Sun ZFS Storage 7120 Appliance 5,000 Mailbox Resiliency Exchange

Workload Resiliency with EMC VPLEX - Data Storage ... · increased workload resiliency to the data center. Best practice ... EMC VPLEX introduces a new architecture that ... Workload

Gridstore Proves Storage Resiliency - Paperless Productivity · 2017-10-05 · i Gridstore Proves Storage Resiliency About DeepStorage DeepStorage, LLC. is dedicated to revealing

Doing Resiliency Better - StarChapter · Doing Resiliency Better: ... • While there are many positive Resiliency initiatives nationwide, Resiliency remains reactive in nature, somewhat

DELLTM PowerEdge T610 500 Mailbox Resiliency Exchange 2010 ... · DELLTM PowerEdgeTM T610 500 Mailbox Resiliency Exchange 2010 Storage Solution . Tested with: ESRP – Storage Version

Sun ZFS Storage 7420 Appliance 32,000 Mailbox Resiliency Exchange 2010 Storage Solution · Sun ZFS Storage 7420 Appliance 32,000 Mailbox Resiliency Exchange 2010 Storage ... Storage

Minutes of March 20, 1969 Meeting Rjr-penick