Upload
udo-seidel
View
803
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Operational aspects of distributed Linux file systems like Ceph and/or GlusterFS
Citation preview
Cloud storage reloaded:Some aspects on operating distributed Linux file systems
like Ceph and GlusterFS
Udo Seidel
LinuxCon Europe 05-07/11 2012 2
Agenda
● Introduction/motivation● Distributed storage● Ceph and GlusterFS● Operational considerations● Summary
LinuxCon Europe 05-07/11 2012 3
Introduction
LinuxCon Europe 05-07/11 2012 4
Me ;-)
● Teacher of mathematics & physics● PhD in experimental physics● Started with Linux in 1996● Linux/UNIX trainer● Solution engineer in HPC and CAx environment● Head of a international team with people in
Erding, Miami and Sydney @Amadeus
LinuxCon Europe 05-07/11 2012 5
Shared File Systems
● Normal 'business' for Linux● Several implementation approaches
● Network based, e.g. NFS, CIFS● 'clustered'
– Shared disk, e.g. OCFS2, GFS2– Parallel/distributed, e.g. Ceph, GlusterFS
LinuxCon Europe 05-07/11 2012 6
Distributed File Systems
● 'Recent' attention on distributed storage● Cloud hype● Big Data
● Which solution to choose?● Just storage or file system too?
● Knock-on effects● Technical● Non-technical
LinuxCon Europe 05-07/11 2012 7
Operations
● Important part of the life cycle● Geek vs. Enterprise● Technical challenges and more
LinuxCon Europe 05-07/11 2012 8
Distributed Storage
LinuxCon Europe 05-07/11 2012 9
History
● Reviewing storage task responsibilities● Block allocation● Space management
● Extension of SCSI standard● Object based storage● Meta-Data handling separated from data
management
LinuxCon Europe 05-07/11 2012 10
Object based storage
● Storage objects quite general● Partition, file, ...● Unique identifier
● OSD (Object based Storage Device)● Hardware -> original trigger● Software -> common implementation
● Interfaces to existing technology● Part of the distributed storage solution● Additional development of existing approaches
LinuxCon Europe 05-07/11 2012 11
Distributed storage: Paradigm changes
● Block -> Object● Central -> Distributed
● Few -> Many● Big -> Small
LinuxCon Europe 05-07/11 2012 12
Distributed storage – Now what?!?
● Several implementations● Different functions● Support models● Storage vendors initiatives● Relation to Linux distributions
LinuxCon Europe 05-07/11 2012 13
Ceph and GlusterFS
LinuxCon Europe 05-07/11 2012 14
Ceph and GlusterFS
● Sufficient community presence● Picked up by Enterprise Linux vendors● Good amount of commonalities but still quite
different
LinuxCon Europe 05-07/11 2012 15
Ceph and GlusterFS – High level
● Server part● Block storage -> storage server (OSD)● (POSIX) file system -> Meta data● HA -> replication and distribution
● Client part● (POSIX) file system● Storage layer
LinuxCon Europe 05-07/11 2012 16
Ceph
LinuxCon Europe 05-07/11 2012 17
GlusterFS
LinuxCon Europe 05-07/11 2012 18
Ceph and GlusterFS – some detailsCEPH GLUSTERFS
Server implementation userspace userspace
Client implementation userspace/FUSE and native kernel
userspace/FUSE
File system access CEPH protocol suite GlusterFS and NFS protocol
Block device access yes RFP ongoing
QEMU integration yes Under development
Replication traffic OSD network Client-server network
Interconnect IP based IP based and/or Infiniband
Encryption no not really
Back-end file system XFS ..... BTRFS ... ext4 XFS ........ext4
Storage object CRUSH object file (except for striping)
GEO replication not really yes
LinuxCon Europe 05-07/11 2012 19
Ceph and GlusterFS – more detailsCEPH GLUSTERFS
Online growth/shrink yes yes
POSIX yes alike
Concurrent workload technically yes technically yes
NAS replacement technically yes technically yes
SAN replacment technically yes no
Hadoop integration yes yes
Openstack integration yes yes
Community version available
yes yes
Production readiness Block layer File system layer
Service provider Vendor and other Vendor and other
... ... ...
LinuxCon Europe 05-07/11 2012 20
Operational considerations
LinuxCon Europe 05-07/11 2012 21
Status quo “pre distributed storage”
● Server and Storage● Clear responsibilities● Known area: SCSI, NFS, FC● Strict border lines
– Technically– organizationally
● Storage hardware and topology● Less focus on local drives● Dedicated/separated storage network
LinuxCon Europe 05-07/11 2012 22
Changes with distributed storage
● New paradigm● Big overlap between server and storage
● Technical implication● Organizational structure
● Strong network dependency● Topology● physics
● Support and supported setups
LinuxCon Europe 05-07/11 2012 23
Challenge one: server vs. storage
● Need for● Server admin with strong storage skills● Storage admin with strong O/S skills
● Review of● Current organizational setup● Strategy for traditional storage● Operation procedures● HA architecture
LinuxCon Europe 05-07/11 2012 24
Challenge one: possible answers
● 'DevOps model' for O/S and storage admins● Best of both worlds● Better common understanding● Potential of process simplification● BUT organizational changes needed!
LinuxCon Europe 05-07/11 2012 25
Challenge one: possible answers
● 'Appliance model' for O/S and storage admins● Less internal changes● Potential of reusing existing processes● BUT less control and visibility
LinuxCon Europe 05-07/11 2012 26
Challenge two: infrastructure
● Dedicated vs. Shared● Responsibilities
● server vs. storage vs. network
● Transport medium● Local setup
● Spinning disks vs. SSD● RAID?● HA configuration of single server
LinuxCon Europe 05-07/11 2012 27
Challenge two: possible answers
● Different flavour of performance vs. cost topic● Dedicated vs. Shared● Transport medium● Spinning disks vs. SSD● HA on server level
LinuxCon Europe 05-07/11 2012 28
Challenge two: possible answers
● Team up with network organization● Single server full blown SPOF on purpose● Think big
LinuxCon Europe 05-07/11 2012 29
Challenge three: support
● Service provider● Application support
LinuxCon Europe 05-07/11 2012 30
Challenge three: possible answers
● Combine with Linux vendor support● Single point of contact ● Leverage vendor QA
● Hide the new technology● XaaS ● Appliance approach● File system level
● Reminder: Think big
LinuxCon Europe 05-07/11 2012 31
Summary ... kind of ...
LinuxCon Europe 05-07/11 2012 32
The message
● New world with several challenges ● Don't underestimate them● Don't overestimate them
● Technical revolution● requires non-technical changes● Don't underestimate the potential
● Think big! ;-)● Get started!
LinuxCon Europe 05-07/11 2012 33
References
● http://www.ceph.com● http://www.gluster.org● “your business/company”
LinuxCon Europe 05-07/11 2012 34
Thank you!