DreamObjects Cloud Object Storage

Powered by Ceph

About Me

•  Senior systems engineer at dreamhost •  Been with dreamhost since 2007 •  Systems lead for DreamObjects •  Excited about storage, distributed systems,

networking, linux internals, automation and monitoring

•  I built my first ceph cluster in april, 2010

twitter: @mmgaggle

Anatomy of DreamObjects

Nuts and Bolts - Storage Nodes

•  Intel nehalem or AMD lisbon •  32 GB of ram •  8-12 enterprise 3.5" sas disks •  LSI megaraid 1078/2108 •  5x Gb nics •  IPMI 2.0 BMC

Nuts and Bolts - Support Nodes

•  Intel westmere-ep •  8 GB ram •  3x Gb nics •  4x SAS drives •  IPMI 2.0 BMC

ceph monitors, radosgw, proxies, object sync

Tubes

•  IPv6 native •  Dual-stack application proxy / balancer •  Application, rados, and cluster networks •  One gig links, bonded •  LLDP for discovery

Traffic

•  IPv6 and IPv4 api endpoints •  IPv6 to radosgw nodes •  Stud for ssl termination •  HAproxy for load balancing •  10Gb backend and frontend

Ceph Numbers

•  Two plus petabytes of raw storage •  Triplicate replication •  900+ OSDs •  7 nines durability •  Single datacenter

CRUSH Configuration

•  Row, rack, node, and disk fault tolerant •  Weighted, even distribution of data •  Object striping •  Online expansion and contraction

OSD Configuration

•  Single disk RAID0 •  Two partitions per RAID0, data + journal •  Writeback enabled •  Disk cache disabled •  Using XFS for filesystem

Application Tuning

•  No keepalive on apache •  Increase apache thread pool •  Increase radosgw thread pool •  Raise radosgw ulimits

Argonaut

•  Lots of failure testing •  Tested on large clusters (~900 osds) •  Weight osds in/out of the cluster

Bobtail

•  Tunables for less aggressive remapping •  CNAME support •  OSD performance ++

Control and Management

•  chef •  pdsh •  runit/upstart •  jenkins •  sensu •  collectd •  graphite •  logstash

Chef

•  Bootstrap cluster, new nodes •  Packages and configuration •  Creates, destroys and encrypts osds •  Search node metadata, restful api •  Roles mapped to pdsh genders •  User and ssh key management •  Loads more..

pdsh

Parallel distributed shell •  Ceph monitors are a natural control point •  pdsh from monitor to other nodes •  ssh-agent + ssh agent forwarding •  Works when chef api doesn't

runit/upstart

Keep ceph daemons running •  ceph-osd •  ceph-mon •  radosgw

Jenkins

•  Continuous api functional tests •  Monitor chef convergence •  Generates alerts on failures

Sensu

•  Relatively new monitoring system •  Open source •  Written in ruby •  State stored in Redis •  Pub-sub with local agent •  Handlers monitor state and generate alerts

collectd

System metrics •  cpu •  memory •  disks •  entropy •  interfaces

graphite

•  bucky - collectd, statsd, metricsd api •  carbon cache •  whisper for metric storage (sequential writes) •  graphite webui

logstash

•  lumberjack agent to ship logs to logstash •  logstash graphite output •  logstash elasticsearch output •  elasticsearch restful api •  kibana webui sits on top of elasticsearch

Future

•  East coast region •  Pool replication to another datacenter •  Object expiration, root site support •  Content delivery network

..not necessarily in that order ;)

Thanks

Questions?

kyle.bader [at] dreamhost.com @mmgaggle

Inktank’s Professional Services Consulting Services:

•  Technical Overview •  Infrastructure Assessment •  Proof of Concept •  Implementation Support •  Performance Tuning

Support Subscriptions: •  Pre-Production Support •  Production Support

A full description of our services can be found at the following:

Consulting Services: http://www.inktank.com/consulting-services/

Support Subscriptions: http://www.inktank.com/support-services/

Check out our upcoming and on demand webinars from Inktank

Upcoming: Advanced Features of Ceph Distributed Storage (delivered by Sage Weil, creator of Ceph)

February 12, 2013 10:00AM PT, 12:00PM CT, 1:00PM ET http://www.inktank.com/news-events/webinars/

On Demand:

Getting Started with Ceph http://www.inktank.com/news-events/webinars/ Intro to Ceph with OpenStack http://www.inktank.com/news-events/webinars/

Contact Inktank

Info@inktank.com and 1-855-INKTANK Don’t forget to follow us on: Twitter: https://twitter.com/inktank Facebook: http://www.facebook.com/inktank YouTube: http://www.youtube.com/inktankstorage

THANK YOU!