Ceph vs Swift

8/9/2019 Ceph vs Swift

1/46

Ceph vs SwiftPerformance Evaluation on a Small Cluster

eduPERT monthly call

July, 24th 2014

Jul 24, 2014 GANT eduPERT meeting


2/46

About me

Vincenzo Pii

Researcher @

Leading research initiative on CloudStorage Under the theme IaaS

More on ICCLab: www.cloudcomp.ch

http://www.cloudcomp.ch/http://www.cloudcomp.ch/


3/46

About this work

Performance evaluation study on cloudstorage

Small installations

Hardware resources hosted at the ZHAWICCLab data center in Winterthur

Two OpenStack clouds (stable and experimental)

One cluster dedicated to storageresearch



4/46

INTRODUCTION



5/46

Cloud storage

Cloud storage Based on distributed, parallel, fault-tolerant file systems Distributed resources exposed through a single homogeneous

interface Typical requirements

Highly scalable Replication management Redundancy (no single point of failure) Data distribution

Object storage A way to manage/access data in a storage system Typical alternatives

Block storage File storage



6/46

Ceph and Swift

Ceph (ceph.com) Supported by Inktank

Recently purchased by RedHat (owners of GlusterFS)

Mostly developed in C++ Started as PhD thesis project in 2006

Block, file and object storage

Swift (launchpad.net/swift) OpenStack object storage

Completely written in Python

RESTful HTTP APIs

http://ceph.com/https://launchpad.net/swifthttps://launchpad.net/swifthttp://ceph.com/


7/46

Objectives of the study

1. Performance evaluation of Ceph and Swift on a smallcluster

Private storage Storage backend for own-apps with limited requirements

in size Experimental environments

2. Evaluate Ceph maturity and stability Swift already widely deployed and industry-proven

3. Hands-on experience Configuration Tooling



8/46

CONFIGURATION AND PERFORMANCEOF SINGLE COMPONENTS



9/46

Network configuration

Three servers on a dedicated VLAN

1 Gbps NICs

100BaseT cabling


10.0.5.x/24

Node 1(.2)

Node 2(.3)

Node 3(.4)


10/46

Servers configuration

Hardware Lynx CALLEO Application Server 1240

2x Intel Xeon E5620 (4 core)

8x 8 GB DDR3 SDRAM, 1333 MHz, registered, ECC 4x 1 TB Enterprise SATA-3 Hard Disk, 7200 RPM, 6

Gb/s (Seagate ST1000NM0011)

2x Gigabit Ethernet network interfaces

Operating system Ubuntu 14.04 Server Edition with Kernel 3.13.0-24-

generic



11/46

Disks performance

READ:

WRITE:


$ sudo hdparm -t --direct /dev/sdb1

/dev/sdb1:

Timing O_DIRECT disk reads: 430 MB in 3.00 seconds = 143.17 MB/sec

$ dd if=/dev/zero of=anof bs=1G count=1 oflag=direct1+0 records in1+0 records out

1073741824 bytes (1.1 GB) copied, 8.75321 s, 123 MB/s


12/46

Network performance


$ iperf -c ceph-osd0

------------------------------------------------------------Client connecting to ceph-osd0, TCP port 5001

TCP window size: 85.0 KByte (default)------------------------------------------------------------[ 3] local 10.0.5.2 port 41012 connected with 10.0.5.3 port 5001

[ ID] Interval Transfer Bandwidth[ 3] 0.0-10.0 sec 1.10 GBytes 942 Mbits/sec

942 Mbits/s117.5 MB/s


13/46

CLOUD STORAGE CONFIGURATION



14/46

Ceph OSDs


Monitor(mon0)

HDD1 (OS)

Not used

Not used

Not used

St. node 0

HDD1 (OS)

osd0XFS

osd1XFS

Journal

St. node 1

HDD1 (OS)

osd2XFS

osd3XFS

Journal

cluster-admin@ceph-mon0:~$ ceph status

cluster ff0baf2c-922c-4afc-8867-dee72b9325bbhealth HEALTH_OK

monmap e1: 1 mons at {ceph-mon0=10.0.5.2:6789/0}, election epoch 1,

quorum 0 ceph-mon0

osdmap e139: 4 osds: 4 up, 4 in

pgmap v17348: 632 pgs, 13 pools, 1834 bytes data, 52 objects

199 MB used, 3724 GB / 3724 GB avail

632 active+clean

cluster-admin@ceph-mon0:~$ ceph osd tree

# id weight type name up/down reweight-1 3.64 root default

-2 1.82 host ceph-osd0

0 0.91 osd.0 up 1

1 0.91 osd.1 up 1

-3 1.82 host ceph-osd1

2 0.91 osd.2 up 1

3 0.91 osd.3 up 1


15/46

Swift devices

Building rings on storage devices

(No separation of Accounts, Containers and Objects)


export ZONE= # set the zone number for that storage device

export STORAGE_LOCAL_NET_IP= # and the IP address

export WEIGHT=100 # relative weight (higher for bigger/faster disks)

export DEVICE=swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT

swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT

swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT

Swift

ProxyHDD1 (OS)

Not used

Not used

Not used

St. node 0

HDD1 (OS)

dev1XFS

dev2XFS

Not used

St. node 1

HDD1 (OS)

dev3XFS

dev4XFS

Not used


16/46

Highlighting a difference

LibRadosused to access Ceph Plain installation of a Ceph storage cluster Non ReST-ful interface This is the fundamental access layer in Ceph RadosGW (Swift/S3 APIs) is an additional component on top of

LibRados (as block and file storage clients) ReST-ful APIs over HTTP used to access Swift

Extra overhead in the communication Out-of-the box access method for Swift

This is part of the differences to be benchmarked, even if...

HTTP APIs for object-storage are interesting for many use cases This use case:

Unconstrained self-managed storage infrastructure for, e.g., own apps Control over infrastructureand applications



17/46

WORKLOADS



18/46

Tools

COSBench (v. 0.4.0.b2) - https://github.com/intel-cloud/cosbench Developed by Intel Benchmarking for Cloud Object Storage Supports both Swift and Ceph

Cool web interface to submit workloads and monitor current status Workloads defined as XML files

Very good level of abstractions applying to object storage Supported metrics

Op-Count (number of operations) Byte-Count (number of bytes) Response-Time (average response time for each successful request) Processing-Time (average processing time for each successful request) Throughput (operations per seconds)

Bandwidth (bytes per seconds) Success-Ratio (ratio of successful operations)

Outputs CSV data Graphs generated with cosbench-plot - https://github.com/icclab/cosbench-plot

Describe inter-workload charts in Python

https://github.com/intel-cloud/cosbenchhttps://github.com/icclab/cosbench-plothttps://github.com/icclab/cosbench-plothttps://github.com/icclab/cosbench-plothttps://github.com/icclab/cosbench-plothttps://github.com/icclab/cosbench-plothttps://github.com/intel-cloud/cosbenchhttps://github.com/intel-cloud/cosbenchhttps://github.com/intel-cloud/cosbenchhttps://github.com/intel-cloud/cosbench


19/46

Workloads gist



20/46

COSBench web interface



21/46

Workload matrix

Containers Objects size R/W/D Distr. (%) Workers

1 4 kB 80/15/5 1

20 128 kB 100/0/0 16

512 kB 0/100/0 64

1024 kB 1285 MB 256

10 MB 512



22/46

Workloads

216 workstages (all the combinations of thevalues of the workload matrix)

12 minutes per workstage

2 minutes warmup

10 minutes running time

1000 objects per container (pools in Ceph)

Uniformly distributed operations over theavailable objects (1000 or 20000)



23/46

READING

Performance Results



24/46

Read tptWorkstage AVGs



25/46

Read tpt1 cont4 KB



26/46

Read tpt1 cont128 KB


R d R Ti 1 t


27/46

Read ResponseTime1 cont128 KB



28/46

Read bdwWorkstage AVGs



29/46

Read tpt20 cont1024 KB


R ti 20 t


30/46

Response time20 cont1024 KB



31/46

WRITING

Performance Results



32/46

Write tptWorkstage AVGs



33/46

Write tpt1cont128 KB


Ceph write tpt 1 cont 128


34/46

Ceph write tpt1 cont128KBreplicas


Ceph write RT 1 cont 128


35/46

Ceph write RT1 cont128KBreplicas



36/46

Write bdwWorkstage AVGs


Write Response Time 20 cont


37/46

Write Response Time20 cont512 KB



38/46

READ/WRITE/DELETE

Performance Results



39/46

R/W/D tptWorkstage AVGs



40/46

Read R/W/D Response Time



41/46

CONCLUSIONS

General considerations and future works



42/46

Performance Analysis Recap

Ceph performs better when reading, Swift when writing Ceph librados Swift ReST APIs over HTTP

More remarkable difference with small objects

Less overhead for Ceph Librados CRUSH algorithm

Comparable performance with bigger objects Network bottleneck at 120 MB/s for read operations

Response time Swift: greedy behavior Ceph: fairness


General considerations:


43/46

General considerations:challenges

Equivalency Comparing two similar systems that are not exactly overlapping Creating fair setups (e.g., Journals on additional disks for Ceph) Transposing corresponding concepts

Configuration Choosing the right/best settings for the context (e.g., number of Swift

workers)

Identifying bottlenecks To be done in advance to create meaningful workloads

Workloads

Run many tests to identify saturating conditions Huge decision space

Keep up the pace Lot of developments going on (new versions, new features)


General considerations:


44/46

General considerations:lessons learnt

Publication medium (blog post) Excellent feedback (e.g., Rackspace developer) Immediate right of reply and real comments

Most important principles Openness

Share every bit of information Clear intents, clear justifications

Neutrality When analyzing the results When drawing conclusions

Very good suggestions coming from you could, youshould, you didnt comments



45/46

Future works

Performance evaluation necessary for cloudstorage

More object storage evaluations

Interesting because its very close to the applicationlevel

Block storage evaluations Very appropriate for IaaS

Provide storage resources to VMs Seagate Kinetic

Possible opportunity to work on a Kinetic setup



46/46

THANKS!QUESTIONS?

Vincenzo Pii: [email protected]

Documents

Ceph vs Swift