Ceph, Xen, and CloudStack: Semper Melior-XPUS13 McGarry

Ceph, Xen, and CloudStack:Semper Melior

Xen User Summit| New Orleans, LA | 18 SEP 2013

•Patrick McGarry•Community monkey•Inktank / Ceph•/. > ALU > P4•@scuttlemonkey•patrick@inktankcom

Accept no substitutes

C’est Moi2

•Ceph in <30s•Ceph, a little bit more•Ceph in the wild•Orchestration•Community status•What’s Next?•Questions

The plan, Stan

Welcome!3

On commodity hardware

Ceph can run on any infrastructure, metal or virtualized to provide a cheap and powerful storage cluster.

Object, block, and file

Low overhead doesn’t mean just hardware, it means people too!

Awesomesauce

Infrastructure-aware placement algorithm allows you to do really cool stuff.

Huge and beyond

Designed for exabyte, current implementations in the multi-petabyte. HPC, Big Data, Cloud, raw storage.

…besides wicked-awesome?

What is Ceph?

Software All-in-1 CRUSH Scale

4

Find out more!Ceph.com

…but you can find out more

Use it todayDreamhost.com/cloud/DreamObjects

Get SupportInktank.com

That WAS fast5

OBJECTS VIRTUAL DISKS FILES & DIRECTORIES

CEPHFILE SYSTEM

A distributed, scale-out filesystem with POSIX

semantics that provides storage for a legacy and

modern applications

CEPHGATEWAY

A powerful S3- and Swift-compatible gateway that brings the power of the Ceph Object Store to modern applications

CEPHBLOCK DEVICE

A distributed virtual block device that delivers high-

performance, cost-effective storage for virtual machines

and legacy applications

CEPH OBJECT STOREA reliable, easy to manage, next-generation distributed object

store that provides storage of unstructured data for applications

6

7

8

• CRUSH– Pseudo-random placement

algorithm– Ensures even distribution– Repeatable, deterministic– Rule-based configuration

• Replica count• Infrastructure topology• Weighting

9

10 10 01 01 10 10 01 11 01 10

10 10 01 01 10 10 01 11 01 10

hash(object name) % num pg

CRUSH(pg, cluster state, rule set)

10

10 10 01 01 10 10 01 11 01 10

10 10 01 01 10 10 01 11 01 10

11

CLIENT12

13

14

15

CLIENT

??

16

…with Marty Stouffer

Ceph in the Wild17

No incendiary devices please…

Linux Distros18

Object && BlockVia RBD and RGW (Swift API)

Our BFF

IdentityVia Keystone

More coming!Work continues with updates in Havana and Icehouse.

OpenStack19

BlockAlternate primary, and secondary

Community maintained

CommunityWido from 42on.com

More coming in 4.2!Snapshot & backup supportCloning (layering) supportNo NFS for system VMsSecondary/Backup storage (s3)

CloudStack20

A blatent ripoff!

Primary Storage Flow

•The mgmt server never talks to the Ceph cluster

•One mgmt server can manage 1000s of hypervisors

•Mgmt server can be clustered

•Multiple Ceph clusters/pools can be added to CloudStack cluster

21

A pretty package

A commercially packaged OpenStacksolution back by Ceph.

RADOS for Archipelago

Virtual server management software tool on top of Xen or KVM.

RBD backed

Complete virtualization management with KVM and containers.

BBC territory

Talk next week in Berlin

So many delicious flavors

Other Cloud

SUSE Cloud Ganeti Proxmox OpenNebula

22

Since 2.6.35

Kernel clients for RBD and CephFS. Active development as a Linux file system.

iSCSI ahoy!

One of the Linux iSCSItarget frameworks. Emulates: SBC (disk), SMC (jukebox), MMC (CD/DVD), SSC (tape), OSD.

Getting creative

Creative community member used Ceph to back their VMWareinfrastructure via fibre channel.

You can always use more friends

Project Intersection

Kernel STGT VMWareLove me!

Slightly out-of-date. Some work has been done, but could use some love.

Wireshark

23

CephFS

CephFS can serve as a drop-in replacement for HDFS.

Upstream

Ceph vfs module upstream samba.

CephFS or RBD

Reexporting CephFSor RBD for NFS/CIFS.

MOAR projects

Project Intersection

Hadoop Samba GaneshaRecently Open Source

Commercially supported product from Citrix. Recently Open Sourced. Still a bit of a tech preview.

XenServer

24

Support for libvirtXenServer can manipulate Ceph!

Don’t let the naming fool you, it’s easy

Blktap{2,3,asplode}Qemu; new boss, same as the old boss(but not really)

What’s in a name?Ceph :: XenServer :: LibvirtBlock device :: VDI :: storage volPool :: Storage Repo :: storage pool

Doing it with Xen*25

Thanks David Scott!

XenServer host arch

Xapi, XenAPI

xenopsd S M adapters

libvirtlibxl ceph ocfs2

libxenguest libxc qemu

xen

Client(CloudStack, OpenStack, XenDesktop)

26

Come for the blockStay for the object and file

No matter what you use!

Reduced OverheadEasier to manage one cluster

“Other Stuff”CephFS prototypesfast development profileceph-devellots of partner action

Gateway Drug27

Squash HotspotsMultiple hosts = parallel workload

But what does that mean?

Instant ClonesNo time to boot for many images

Live migrationShared storage allows you to move instances between compute nodes transparently.

Blocks are delicious28

Flexible APIsNative support for swift and s3

And less filling!

Secondary StorageComing with 4.2

Horizontal ScalingEasy with HAProxy or others

Objects can juggle29

Neat prototypesImage distribution to hypervisors

You can dress them up, but you can’t take them anywhere

Still earlyYou can fix that!

Outside usesGreat way to combine resources.

Files are tricksy30

Where the metal meets the…software

Deploying this stuff31

Procedural, Ruby

Written in Ruby, this is more of the dev-side of DevOps. Once you get past the learning curve it’s powerful though.

Model-driven

Aimed more at the sysadmin, this procedural tool has a very wide penetration (even on Windows!).

Agentless, whole stack

Using the built-in OpenSSH in your OS, this super easy tool goes further up the stack than most.

Fast, 0MQ

Using ZeroMQ this tool is designed for massive scale and fast, fast, fast. Unfortunately 0MQ has no built in encryption.

The new hotness

Orchestration

Chef Puppet Ansible Salt

32

Canonical Unleashed

Being language agnostic, this tool can completely encapsulate a service. Can also handle provisioning all the way down to hardware.

Dell has skin in the game

Complete operations platform that can dive all the way down to BIOS/RAID level.

Others are joining in

Custom provisioning and orchestration, just one example of how busy this corner of the market is.

Doing it w/o a tool

If you prefer not to use a tool, Ceph gives you an easy way to deploy your cluster by hand.

MOAR HOTNESS

Orchestration Cont’d

Juju Crowbar ComodIT Ceph-deploy

33

All your space are belong to us

Ceph Community34

35

Up and to the right!

Code Contributions36

Up and to the right!

Commits37

Up and to the right!

List Participation38

This Ceph thing sounds hot.

What’s Next?39

An ongoing process

While the first pass for disaster recovery is done, we want to get to built-in, world-wide replication.

Reception efficiency

Currently underway in the community!

Headed to dynamic

Can already do this in a static pool-based setup. Looking to get to a use-based migration.

Making it open-er

Been talking about it forever. The time is coming!

Hop on board!

The Ceph Train

Geo-Replication Erasure Coding Tiering Governance

40

Quarterly Online Summit

Online summit puts the core devs together with the Ceph community.

Not just for NYC

More planned, including Santa Clara and London. Keep an eye out: http://inktank.com/cephdays/

Geek-on-duty

During the week there are times when Ceph experts are available to help. Stop by oftc.net/ceph

Email makes the world go

Our mailing lists are very active, check out ceph.com for details on how to join in!

Open Source is Open!

Get Involved!

CDS Ceph Day IRC Lists

41

http://wiki.ceph.com/04Development/Project_Ideas

Lists, blueprints, sideboard, paper cuts, etc.

http://tracker.ceph.com/

All the things!

New #ceph-devel

Splitting off developer chatter to make it easier to filter discussions.

http://ceph.com/resources/mailing-list-irc/

Our mailing lists are very active, check out ceph.com for details on how to join in!

Patches welcome

Projects

Wiki Redmine IRC Lists

42

Comments? Anything for the good of the cause?

Questions?E-MAILpatrick@inktank.com

WEBSITECeph.com

SOCIAL@scuttlemonkey@cephFacebook.com/cephstorage

43