Reference Design: SwiftStack and Seagate Scale- Out Object ... · Reference Design: SwiftStack and Seagate Scale-Out Object Storage Solution June 2016. ... This white paper provides

ReferenceDesign:SwiftStackandSeagateScale-OutObjectStorageSolution

June2016

Copyright©2016SwiftStack,Inc.|swiftstack.comPage2of35

TABLE OF CONTENTS

.......................................................................................................................................................................1

TABLEOFCONTENTS......................................................................................................................................2

INTRODUCTION..............................................................................................................................................4

SOLUTIONOVERVIEW....................................................................................................................................5

USECASES......................................................................................................................................................5ENTERPRISEBACKUP...............................................................................................................................................5ACTIVEARCHIVE(E.G.,MEDIAINGEST/EDIT/BROADCAST/ARCHIVE,GENETICSEQUENCING)..............................................6WEBAPPLICATIONDATA.........................................................................................................................................6FILECOLLABORATION(I.E.,SYNC-AND-SHARE)............................................................................................................6PRIVATECLOUDSTORAGE........................................................................................................................................6

SEAGATEMODULARSTORAGEENCLOSURES..................................................................................................6SEAGATE’SCP-2584:ULTRA-DENSEMODULARSTORAGEENCLOSURE...........................................................................7SEAGATE’SAP-LS-1:EMBEDDEDCONTROLLER...........................................................................................................7

SEAGATESOLIDSTATEANDHARDDISKDRIVES..............................................................................................8

SWIFTSTACKOBJECTSTORAGESOFTWARE.....................................................................................................9...........................................................................................................................................................................9SWIFTSTACKOBJECTSYSTEMCOMPONENTS...............................................................................................................9GENERALSWIFTSTACKFEATURES............................................................................................................................10CONTROLLER.......................................................................................................................................................11OPENSTACKSWIFT...............................................................................................................................................11

USINGSWIFTSTACKANDSEAGATEHARDWARETOGETHER..........................................................................12HARDWARECONFIGURATION.................................................................................................................................12

ReferenceArchitecture#2:1-10PBforGeneralPurposeUseCases...........................................................13ReferenceArchitecture#3:1-10PBforBackup/ArchiveUseCases............................................................13

NETWORKING......................................................................................................................................................14ClusterNetworks.........................................................................................................................................14LoadBalancer(s)..........................................................................................................................................15

INSTALLATIONANDSOFTWARECONFIGURATION........................................................................................................16

VALIDATIONTESTING...................................................................................................................................17BENCHMARKINGLOCATIONANDSTRATEGY...............................................................................................................17BENCHMARKHARDWARECONFIGURATIONWITHANEXTERNALSERVERRUNNINGOBJECTSERVICES...................................17BENCHMARKHARDWARECONFIGURATIONWITHASEAGATEEMBEDDEDCONTROLLERRUNNINGOBJECTSERVICES..............20PREPARINGFORTESTING.......................................................................................................................................21TESTSPERFORMED...............................................................................................................................................21

ssbenchTool................................................................................................................................................21


Example:Testingreadsof1kBobjects........................................................................................................22Example:Testingwritesof1MBobjects......................................................................................................22

INTERPRETATIONOFRESULTS:MEETINGREAL-WORLDREQUIREMENTS........................................................................23RESULTS:RAWNUMBERS......................................................................................................................................25

ExternalServers:ThreeReplicas..................................................................................................................25WriteThroughput(MB/sec).........................................................................................................................................................25ReadThroughput(MB/sec)..........................................................................................................................................................27

ExternalServers:8+4ErasureCoding..........................................................................................................28WriteThroughput(MB/sec).........................................................................................................................................................28ReadThroughput(MB/sec)..........................................................................................................................................................29

AP-LS-1Controller:ThreeReplicas...............................................................................................................30WriteThroughput(MB/sec).........................................................................................................................................................30ReadThroughput(MB/sec)..........................................................................................................................................................31

AP-LS-1Controller:8+4ErasureCoding.......................................................................................................32WriteThroughput(MB/sec).........................................................................................................................................................32ReadThroughput(MB/sec)..........................................................................................................................................................33

UnderstandingBottlenecks..........................................................................................................................34

CONCLUSION................................................................................................................................................35


INTRODUCTION

ThispaperpresentsaprivatecloudstoragesolutionthatcombinesSwiftStackobjectstoragesoftwarewithSeagatestoragehardware.SwiftStackisdesignedtoempowerenterprisestoharnessthepowerofobjectstorageintheirowndatacenters.SwiftStackenablesenterpriseuserstomanage,deploy,scale,upgrade,andmonitorsingle-andmulti-siteclustersusingtheOpenStackSwiftobjectstorageengineplusadditionalcapabilities—includingLDAPandActiveDirectoryintegration,CIFS/NFSaccesswiththeSwiftStackFilesystemGateway,and24x7enterprisesupport.

Figure:BlockdiagramofthecomponentsinaSwiftStackdeployedobjectstoragesystem.TheSwiftStackObjectStorageSystemcanspanmultiplegeographicallydistributeddatacenterswithasinglenamespace—thusprovidingbuilt-indisasterrecoveryandflexibilityfordatastorage.AdministratorscanusetheSwiftStackController,whichprovidesaunifiedviewacrossallclusters,todynamicallytuneandoptimizeperformanceandnon-disruptivelyupgradetheentirestorageinfrastructure.Seagateisaworldleaderinstoragehardwarewithofferingsforbothstorageenclosuresaswellassolidstateandharddiskdrives.TheSeagatestoragehardwareisanindustryprovenofferingthatmeetscustomersneedsfromhighperformancecomputingtoobjectstorage.Seagateleadstheindustryforintegratingstoragedrivesintostoragesystems.Asamanufactureranddeveloperofstoragedrivesforthepast35years,Seagateknowshowtobestuseadriveinastoragesystemtodeliverhighperformanceandlonglastingreliability.


WithSwiftStackdeployedonSeagatestoragehardware,usershaveahigh-performancesolutionthatmeetstheirstorageneedsandissimpletodeployandeasytomanage.

SOLUTION OVERVIEW

BycombiningthepowerandflexibilityofSwiftStacksoftwarewiththedenseandmodularSeagateCP-2584storageenclosureandcomputeservers,customerscanachieveatrulyscalable,cost-effective,andeasy-to-deploy-and-manageobjectstoragesolution.ThiswhitepaperprovidesanoverviewofthejointSwiftStackandSeagatesoftware-definedobjectstoragesolution.Itprovidesbothdetailedreferencearchitecturesandsupportingperformanceinformation.TheintentofthisdocumentistoenablethereadertounderstandhowSwiftStacksoftwareisdeployedonSeagatestorageenclosuresandcomputeserversrunningcommondistributionsoftheLinuxoperatingsystem.ThedocumentdescribesthesetupandobservedperformanceonacommonconfigurationinSeagate’slab.Forloadgenerationandtesting,SwiftStackusedthefreelyavailablessbenchloadgenerator,anddetailsaboutthetestsetupandparametersusedcanbefoundintheperformancesection. ThiscombinedSwiftStackandSeagatesolutioncanbeusedformanyusecases—includingenterprisebackupandarchive,mediaandlarge-dataarchivesintheMedia&EntertainmentandLifeSciencesindustries,file-sync-and-share,privatecloudstorageforCloudStackandOpenStack,andmanywebapplicationsleveragingobjectstorage.

USE CASES

ThestoragesolutionofSwiftStacksoftwareandSeagatehardwaretogethercanbeappliedtoavarietyofusecasesintheobjectstoragemarket.Foralltheseusecases,theSeagatehardwareprovidesanextremelydensestoragecapabilitythatisdemonstratedtoshowgoodperformance.SwiftStackhassomebenefitsthatareuniquetoeachusecase;theseareoutlinedhere.Enterprise Backup SwiftStackprovidesaresilientandeasilyscaledstoragetargetforbackupsofVMs,files,anddatabasesusingpopularapplicationslikeCommVault,VeritasNetBackup,IBMSpectrumProtect,PerconaXtraBackup,TrilioData,andmore—manyofwhichhaveaddednativesupportfortheSwiftandS3APIs.ThesebackupscanbeconfiguredtobeautomaticallydistributedtomultipledatacentersbySwiftStacksoftwareforadditionalavailability.Asabackuptarget,SwiftStackcanprovidea


muchlower-costsolutionthanotheron-premisesstorageoptionsorpubliccloudstorage.Built-indatadurabilitythatisconfigurablebypolicyensuresthatbackupsremainavailableevenifaserver,site,ornetworkfails.Active Archive (e.g., Media Ingest/Edit/Broadcast/Archive, Genetic Sequencing) SwiftStackisusedtostoreandarchivelargefilesacrossseveralindustriesthatproducemassiveamountsofdata—includingmedia&entertainment,lifesciences,andotherresearchefforts.Throughoutthedatalifecycle,SwiftStackprovidesnecessaryflexibility—high-throughputdataingest,multi-regiondatadistribution,andcost-effectivereplicaanderasurecodingoptions—tobalancedataavailability,performance,andraw-to-usablespaceratios.Web Application Data SwiftStackisusedtobuildhigh-volumecontentrepositoriesformanywebandmobileapplications.High-concurrencycapabilitysupportstheworkloadthatmillionsofusersplaceonastoragesystem,andasingle,web-addressablenamespaceismuchsimplertousethanrequiringtheapplicationtohaveknowledgeofthestorageinfrastructure.TheseapplicationsaccessdatavianativeHTTPRESTfulAPIs(SwiftandS3),anddevelopersleveragebroadAPIsupportforprogramminglanguagessuchasJava,C,C#,Python,PHP,andRuby.File Collaboration (i.e., Sync-and-Share) SwiftStackmakesitsimpletostoreandshareunlimiteddocumentsandunstructureddataandaccessiteasilyviatheSwiftStackFilesystemGatewayorawiderangeofthird-partysync-and-sharedocumentmanagementapplications—includingOwnCloud,CloudBerry,StorageMadeEasy,Gladinet,andmore.Private Cloud Storage SwiftStackmakesitfastandsimpletorollouton-premisesobjectstorage,easilyconnectittoyourOpenStackorCloudStackcloud,integratewithyourAD,LDAP,orKeystoneauthenticationsystems,andmonitorallofyourmulti-sitestoragefromaneasy-to-usebrowserinterface.

SEAGATE MODULAR STORAGE ENCLOSURES

Seagateoffersamodular,standards-basedfamilyofenclosuresdesignedforcustomersandpartnersrequiringhighperformanceandscalabilityinadenseformfactor.Customersandpartnerscantakeadvantageofindustry-leading,enterprise-classenclosuresastheunderlyingplatformfortheirstoragesolutionsandfocustheireffortsontheirownvalue-addedsoftware.TheSeagatefamilyof


enclosuresofferaseriesofstoragebuildingblockstodelivermorepowerful,intelligentstoragesolutions,eachleveragingthefollowingbenefits:

● 6Gb/sarchitecture● Commonplatformmanagementsoftware● SBB2.0compatibility● CapacityexpansionwithOneStorModularEnclosures● Customizabletomeetspecificendproductrequirements

Seagate’s CP-2584: Ultra-Dense Modular Storage Enclosure Seagate’sCP-2584StoragePlatformdeliversamodularandultra-densestoragesolutiondesignedtoaccommodatetheproliferationofdatagrowth.Thishighdensitydesignhelpsreclaimpreciousdatacenterfloorspacewhiledecreasingpowerandcoolingcostresultinginexceptionallylowtotalcostofownership.TheCP-2584familyisperfectforenterprise-classapplicationswithsignificantstoragerequirementsincludingbigdataanalytics,HPC,digitalmedia,complianceretentionanddataarchival.

● 5Urackmountenclosure● Upto84SASandSATAHDDsorSSDs● Expansioncapabilityupto336drives● ConfigurablewithanembeddedserverorasaJBOD.

Figure:TheSeagatehighdensity5Uenclosurewhichholds84HDDsSeagateoffersservicepackagesforallofitshardwareofferings.ThispaperfocusesontheCP-2584-G2SATADD(directdockSATA)version.TheCP-2584-G2SATADDisavailablein6Gb/sSAStoaccommodate84SATAHDDs.AnotherversionisavailablethatincludessupportforSASHDDs.OnecouldalsoutilizeSeagate’s4Uenclosuresfor24HDDsifstoragedensityisnotatoppriority.Seagate’s AP-LS-1: Embedded Controller TheSeagateCP-2584storageenclosurecanbeconfiguredwithanembeddedcontroller(i.e.server).TheAP-LS-1leveragesIntel’sXeon®E5architecturecodenamedHaswellwithupto12physicalcores


and24threadsofprocessingcapabilityinasinglecontroller.ThisCPUclassalsoutilizesDDR4DIMM2133MHztechnology.Basedon128GBDDR4StandardHeightDIMMS,thiswillyieldupto512GBpercontroller.Forinternalcontrollerstorage,theAP-LS-1hasoptionsfor1or2M.2SSDsthatcandoavarietyofdutiesincludingoperatingsystemorapplication.TheAP-LS-1hasdualI/Oexpansionslots–atraditionallow-prole,half-heightPCIecardslotplusaPCIebasedcustomDaughterCardslot.ThisgivestheAP-LS-1multipleI/Ooptions,withbothslotsoperatingatPCIeGen3x16.TheAP-LS-1usedinthistestingisconfiguredwithanIntelE5-2648Lv3CPU,128GBRAM,dual256GBM.2SSDs,andanIntelX520-DA2toprovidetwoSFP+portsforethernet.ThebenefitsofusingSeagate’sAP-LS-1embeddedcontrollerincludereducedcomplexityandincreasedstoragedensity.OnedoesnothavetodealwithmorepowercablingastheHApowerfortheenclosurealsosuppliestheembeddedcontroller.Inaddition,onedoesnothavetodealwithSAScablingasthatisallinternaltotheenclosure.Lastly,densityofthesystemisincreasedasbothstorageandservercomputearecontainedwiththeentire5uoftheenclosure.

SEAGATE SOLID STATE AND HARD DISK DRIVES

TheSeagateSSDfamilyoffershigh-capacity,high-performance12Gb/sSASSSDsdesignedwithmultipleenduranceofferingsoptimizedfordemandingenterpriseapplicationsandmaximumTCOsavings.TheSeagateSSDusedinthistestingwastheST400FM0073.

Figure:ASeagate6TBenterpriseclassSATAHDDSeagateEnterpriseHardDrivesenablethehighestenterprisestoragecapacitiesandcostefficienciesfordatacenterandcloudstorageapplications.WiththeninthgenerationofSeagateEnterpriseHardDrivesyoucanConfidentlyStoreUpto10TBofdatainasingledrivethatwasdesignedandbuilttosupportenterprise-classnearlineworkloadsofup550TBperyear.Seagatedrivesalsoofferavariety


ofoptionstomeetcustomer's’securityneedssuchasselfencryptionandinstantsecureerase.TheSeagateHDDusedinthistestingforstoringenduserdatawastheST6000NM0024.

SWIFTSTACK OBJECT STORAGE SOFTWARE

Figure:BlockdiagramshowingthekeycomponentsinaSwiftStackobjectstoragesystem.ThissectiondescribesthecomponentsinaSwiftStackobjectstoragesystem.SwiftStack Object System Components

● SwiftStackController:EasilymanagetheentirestoragesystemviatheController,whichsitsout-of-bandfromthedatapathandenablesadministratorstoeasilydeploy,scale,upgrade,andmonitorthestoragesystemacrossmultiplesites.Thecontrollercanbedeployedon-premisesorAs-a-ServicefromcloudinfrastructuremanagedbySwiftStack.

● SwiftStackNodes:SwiftStackNodesoftwareisinstalledonindustry-standardx86hardwarewithstandardLinuxdistributions.Thesystemenablesadministratorstomix-and-matchhardwarefrommultiplevendors,matchingcapitalinvestmentsdirectlytostorageneeds.

● FilesystemGateway(Optional):ProvidesflexibilitytostoreandretrievefilesviaNFSorCIFSprotocols.UnlikemostotherfilesystemGateways,filesstoredviatheGatewaycanalsobeaccessedviaRESTfulHTTPAPIasobjectsandviceversa.


General SwiftStack Features ● CentralManagementInterface:TheSwiftStackControllerletsyoumanageallofyourobject

storageclustersfromasinglecentralizedmanagementinterface,reducingoperationalcomplexity,conservingITresources,andsavingcosts.

● SupportsExistingApplicationWorkflows:AppsaccessSwiftStackstorageviaaRESTfulHTTPAPIandabroadsetofclientlibrariesforPHP,Python,Ruby,Java,andotherlanguages.AndwealsosupporttheS3APIandfilesystemprotocolslikeNFSandCIFS.

● FlexibleDataProtection:Storagepoliciesletyoucreatemultipletiersofstoragewithinthesamenamespace,givingadminsandapplicationsfullcontroloverwheretheirdataislocatedandhowmanyreplicasorerasure-codedsegmentsarekeptforavailability.

● EnterpriseIntegrations:SwiftStackeasilyintegrateswithyourexistingenterpriseauthenticationinfrastructure,includingActiveDirectoryandLDAP,speedinguptheprovisioningandmanagementofstorageusers.

● Non-disruptiveUpgrades:TheSwiftStackControllerletsyouupdateyourstorageclusterwhileitisrunning,withnodisruptionordowntime.Oneclickandallofthestepsneededtoupdatenodeshappensautomaticallyinthebackground.

● UtilizationandReporting:Build-outyourstorage-as-a-service.Chargebackandreportonstorageutilizationtousers,departmentsand/orapplications.Utilizationistrackedonaper-storagepolicybasisforbothdatastoredandtransfersin/out.

● MonitoringandAlerts:SwiftStackproactivelymonitorsthehealthofeachstoragenodeanddrivetoensureavailabilityandperformance.AlertscanbeintegratedwithexternalmonitoringtoolssuchasNagiosandviaSNMP.

● AutomaticDataDistribution:ASwiftStackobjectstorageclustercanbespreadovermultiple,geographicallydistributeddatacenterswithasinglenamespace.Thismeansbuilt-indisasterrecoveryandmoreflexibilityforyourdata.

● 24x7EnterpriseSupport:SwiftStackprovides24x7enterprisesupport.


Controller

TheSwiftStackControllerprovidesasinglepaneofglassdashboardtoeasilymanageyourentireSwiftStackObjectStoragedeployment,whetherafewterabytesatonesiteorhundredsofpetabytesdistributedworld-wide.TheControllerenablesEnterprisestomanage,deploy,scale,upgrade,andmonitortheobjectstoragesystemviaAPIsandabrowserbaseddashboard.DeploymentoptionsfortheControllerinclude:on-premisesbehindyourfirewallorAs-a-ServicefromcloudinfrastructuremanagedbySwiftStack.Regardlessofdeploymentchoice,yourdatadoesnotleaveyourdatacenter(s).OpenStack Swift OpenStackObjectStorage(codenamed“Swift”)isanopensourceengineforobjectstorage,availableforuseundertheApache2opensourcelicensethroughtheOpenStackFoundation.Swiftistheenginethatrunstheworld’slargeststorageclouds,andisdesignedtooperateonindustry-standardx86serverhardwareandscaleouttostorebillionsoffilesandhundredsofpetabyteswithoutanysinglepointoffailure.


USING SWIFTSTACK AND SEAGATE HARDWARE TOGETHER

Hardware Configuration SwiftStack,atahighlevel,consistsofthefollowingprimaryservices:

● Proxyservices:Communicatewiththeexternalclientsandcoordinateread/writerequests● Accountservices:Providemetadatafortheindividualaccountsandalistofthecontainers

withinanaccount● Containerservices:Providemetadatafortheindividualcontainersandalistofobjectswithin

acontainer● Objectservices:Provideablobstorageservicethatcanstore,retrieve,anddeleteobjectson

thenode’sdrives● Consistencyservices:Findandcorrecterrorscausedbydatacorruptionorhardwarefailures

ForaSwiftStackclusterusingSeagatehardware,theProxy,Account,andContainerservicesaremostoftendeployedoncommonone-rack-unit(1u)serverswhichcanutilizeSeagateHDDsandSSDsforthebootdrivesandmetadatastorage.TheObjectservicesrunonseparateserversconnectedtotheCP-2584JBODs.Theseserverscouldbeanexternal(1u)serveroranAPL-LS-1SeagatecontrollerembeddedintheCP-2584enclosure.Asdetailedintheconfigurationsbelow,theratioofobject-servers-to-JBODs(i.e.,theratioofCPUs-to-HDDs)variesaccordingtousecase.ThefollowingthreereferencearchitecturesprovideagoodstartingpointforSwiftStackandSeagateOneStorehardware.Inadditiontothehardwarelisted,besuretoaccountfordatacenterracks,top-of-racknetworkswitches,andloadbalancers(ifnotalreadyinplace).ContactSwiftStackforassistancewiththesecomponentsand/ortoidentifyadditionalconfigurationsforsignificantlydifferentusecasesormuchlargercapacities.ReferenceArchitecture#1:<1PBforTest/Dev&GeneralPurposeUseCases

NodeModel(Role) CPU/RAM Drives Network

1userver+CP-2424(PACO)

● 2xCPU(e.g.,E5-2620v3)● 64GB

● 2xSeagatebootdrives● 12-24x3.5”SeagateHDDfordata

● Multiple1GbNICsoratleast1x10GbNIC

Notes:● Forlargeobjectcounts(e.g.,1-10millionobjectspercontainer),add1xSeagateSSDperserverforaccountand

containerdata.● ForCPUs,choosemodelssuchthattheratioof(sockets*cores*speed):dataHDDsisatleast2:3.● Testingenvironmentscommonlystartwiththreenodes;productionenvironmentscommonlystartwithnine

(threezonesofthreenodeseach)andexpandthreenodesatatime.


Reference Architecture #2: 1-10 PB for General Purpose Use Cases


1userver(PAC) ● 2xCPU(e.g.,E5-2620v3)● 128GB

● 2xSeagatebootdrives● SeagateSSD(4perCP-2584)

● Atleast2x10GbNICports

2x1userver+1xCP-2584(Object)

Perserver:● 2xCPU(e.g.,E5-2650v3)● 128GB

● 2xSeagatebootdrives(perserver)

● SeagateHDDsfordata (inCP-2584)

● Atleast2x10GbNICports(perserver)

Notes:● Include4xSSDperCP-2584inthe1userverPACnodesforaccountandcontainerdata.● ForCPUsintheobjectservers,choosemodelssuchthattheratioof(sockets*cores*speed):dataHDDsinthe

CP-2584isatleast2:3.● ConfigurationscommonlystartwithtwoorthreedatacenterrackscontainingtwoPACnodesandtwoor

threeCP-2584+serversobjectnodesineachrack;capacityisaddedbyaddingobjectnodesuntiltheracksarefullandthenaddingracks(eachrackcontainstwoPACnodes).

Reference Architecture #3: 1-10 PB for Backup/Archive Use Cases


1userver(PAC) ● 2xCPU(e.g.,E5-2620v3)● 64GB

● 2xSeagatebootdrives● SeagateSSD (4perCP-2584)


1x1userver+1xCP-2584(Object)

● 2xCPU(e.g.,E5-2620v3)● 128GB

● 2xSeagatebootdrives (inserver)

● SeagateHDDsfordata (inCP-2584)


Notes:● Include4xSeagateSSDperCP-2584inthe1userverPACnodesforaccountandcontainerdata.● ForCPUsintheobjectservers,choosemodelssuchthattheratioof(sockets*cores*speed):dataHDDsinthe

CP-2584isatleast1:3.● ConfigurationscommonlystartwithtwoorthreedatacenterrackscontainingonePACnodeandtwoor

threeobjectnodesineachrack;capacityisaddedbyaddingobjectnodesuntiltheracksarefullandthenaddingracks(eachrackcontainsonePACnode).


Networking

Cluster Networks

Note:DiagramsandadditionaldetailsaboutSwiftStacknetworkconfigurationsandintegrationwithDNS,load-balancing,andfirewallsareavailableathttps://swiftstack.com/solutions/guides/#network.

Figure:DiagramshowingthevariousnetworksconfiguredinatypicalSwiftStackdeployment.AtypicalSwiftStackdeploymentwillhavean“outward-facing”networkandaninternal“cluster-facing”network.WhendesigningthenetworkcapacityforaSwiftStackdeploymentusingthedefaultthree-replicastoragepolicy,keepinmindthatwritesfan-outintriplicateinthecluster-facingnetwork.Sincetherearethreecopiesofeachobject,anincomingwritesenttoaproxynodeisthen“forwarded”tothreeobjectnodes.Therefore,networkcapacityforwritesneedstobeconsideredinproportiontooverallworkload.Startingfromtheclient'sperspective,the“client-facing”IPistheIPaddress(orDNSrecord)thatclientsconnectto.Typically,thatwouldbeaWANIPonaloadbalanceroronafirewall.


The"outward-facing"IP(s)aretheIP(s)ontheproxynode(s).Itisimportanttonotethat“outward-facing”isnotthesameasapublicorWANIP.These“outward-facing”IPsaresimplytheIPaddressesoftheproxynodesfacingout—towardstheloadbalancer.Thus,outward-facingIPsaretheIPsthatshouldbeincludedintheloadbalancingpool.Allintra-clustertraffic(betweentheproxyandobjectnodes,forexample)occursthroughthe"cluster-facing"IP(s).Insummary,whileallnetworkinginaSwiftclusterisdoneviaLayer-3,aSwiftStackclusterwillhavethesenetworksegments:

● Outward-facingNetwork:Usedforclienttraffic(i.e.,APIaccess).Ifanexternalloadbalancerisused,itwillexistintheoutward-facingnetwork.

● Cluster-facingNetwork:Usedforintra-clusterSwiftStacknodecommunication.● ReplicationNetwork(optional):Usedtoseparateintra-clusterreplicationtraffic(which

occurswhenadrivefailsorcapacityisaddedtoorremovedfromthecluster)fromotherintra-clustertrafficfornormalclienttraffic.

● ManagementNetwork:UsedforIPMI,iLO,etc.,forhardwaremanagement.Ifdesired,Seagatecanprocurenetworkinghardwareasitoftendoesforavarietyofitscustomerbase.

Load Balancer(s)

ForaSwiftStackcluster,adedicatedload-balancingsolutioncanuseanymethodfromsimpleround-robinDNStolayer-7loadbalancing.Open-sourceloadbalancerssuchasHAProxyorcommercialsolutionsfromcompanieslikeF5andA10canbeused.ForSwift,youwillneedtoload-balanceacrossyourSwiftproxynodestodividetheclientrequestloadbetweenyourproxyservers.Theoutward-facingIPsoftheproxynodesneedtobeincludedintheload-balancingpool,andtheloadbalancermustbeassignedavirtualIPaddress(VIP).Loadbalancersshouldbesetuptorunhealthchecksagainsttheproxynodesintheirpoolssothataproxywillbeautomaticallyexcludedifitisunresponsive.Toremoveafailedproxynodefromtheloadbalancingpool,configuretheloadbalancertochecktheproxynode’shealthcheckURL.AhealthySwiftStackproxynodeshouldrespondonhttp://<outward-ip-address>/healthcheck,suchashttp://192.168.11.81/healthcheck.


Installation and Software Configuration Thefollowinghigh-levelstepsareinvolvedindeployingthesereferencearchitectures:

1. InstallaSwiftStackcontroller.ThiscanbedonebyleveragingthehostedSwiftStackManagementServiceorbydeployinganon-premisesinstanceofthiscontrolleronphysicalhardwareorinavirtualmachine.TosignupforanaccountontheSwiftStackManagementService,visithttps://swiftstack.com/try-it-now/,orforanon-premisesinstance,[email protected].

2. Installhardware—including1uservers,SeagateJBODs,SeagateSSDsandSeagateHDDs,networkswitches,andaloadbalancer(ifneeded).

3. InstallLinux.SwiftStacksupportsRHEL,Ubuntu,andCentOS.Installthestandardserveredition;assignahostname;andassignIPaddressestonetworkinterfaces.

4. ConfigureSwiftStacknodes.TheSwiftStackmanagementsoftwaregreatlysimplifiestheprocessofpreparingandallocatingserversanddrivesforuse.Onlyasinglecommandisrequiredoneachserver,andtherestoftheconfigurationisdoneinSwiftStack’ssimple-to-usebrowserinterface.Detailsandstep-by-stepscreenshotscanbefoundinSwiftStack’sQuick-StartGuide:https://swiftstack.com/docs/install/index.html

a. InstalltheSwiftStacksoftwareoneachnode(‘curl https://platform.swiftstack.com/install | bash’).

b. Claimthenewnode(s)toyourSwiftStackclusterinthebrowser(i.e.,copyandpasteonelinefromtheterminalwindowtoyourbrowserwindow).

c. CreateanewSwiftStackclusterifdesiredforyournewnode(s).

d. Allocatedrivesandnetworkinterfacesforthenewnode(s),andidentifyinwhichzoneandregioneachnodebelongs.

e. Createyourfirstusertoaccessthecluster(oralternatively,integratewithLDAP,AD,etc.).

f. Click“Deploy,”andbeginusingyournewSwiftStackcluster!


VALIDATION TESTING

Benchmarking Location and Strategy Itisoftenhelpfultounderstandthecapabilitiesofareferencearchitecturewhenplacedundersimulatedclientload.SwiftStackandSeagateperformedbenchmarktestinginSeagate’sLongmont,Coloradoengineeringlabontwodifferentconfigurationstosimulateaseriesofcommonobjectstorageworkloads.ThefirstconfigurationusedanIntel1uservertoruntheobjectservices.ThesecondconfigurationusedaSeagateembeddedcontrollertoruntheobjectservices.BothconfigurationsusedtheSeagateCP-2584enclosurealongwithIntel1userverstogeneratethetestworkloadsandasSwiftStackProxy(P)nodes.Thefulltestconfigurationisdescribedbelow.Benchmark Hardware Configuration with an External Server running object services

Component Qty Description Picture

SwiftStackController 0 ● UsedSwiftStack’sController-as-a-Service

Load-generationNodes 3

● Intel1uServer● 2xE5-2630v2CPUs● 64GBRAM● 2xSeagateSASbootHDDs● 2x10Gbnetworkports

ProxyNodes 3


Account/Container/Object(ACO)NodeServers

6

● Intel1uServer● 2xE5-2630v2CPUs● 128GBRAM● 2xSeagateSASbootHDDs● 4xSeagateSASSSDs● 2x10Gbnetworkports● 1xAvago9207-8eSASHBA


Account/Container/Object(ACO)NodeJBODs

6

● Seagate5U84EnclosurewithEBODSASSBBModule(CP-2584-G2-E6-EBD-2P2800-SATA-DD)

Account/Container/Object(ACO)NodeHDDs

6● 84x6TBSeagateSATAHDDs

ModelST6000NM0024

Top-of-RackSwitches 2● Arista7150S-24-R24-port

10GbESFP+switch

ManagementSwitch 1 ● BrocadeICX6610-2424-port1GbERJ45switch

Note:NetworkandSAScables,datacenterracks,andpowerdistributionunitsarenotincludedinthehardwarelistingabove.ThisconfigurationdiffersslightlyfromReferenceArchitecture#2inthatthecontainerandaccountservicesaredeployedontheobjectserver—nottheproxyserver.

Table:ThecomponentsoftheexternalserversystemtestedbySwiftStackandSeagate.


Figure:PictureshowinghowthehardwareisdeployedinarackforthisSwiftStackSeagatesystem.


Benchmark Hardware Configuration with a Seagate Embedded Controller Running Object Services

Component Qty Description Picture

SwiftStackController 0 ● UsedSwiftStack’sController-as-a-Service

Load-generationNodes 3


Proxy/Account/Container(PAC)NodeServers

3

● Intel1uServer● 2xE5-2630v2CPUs● 64GBRAM● 2xSeagateSASbootHDDs● 6xSeagateSASSSDs● 2x10Gbnetworkports

Object(O)NodeServers 6

● Seagate5U84EnclosurewithembeddedcontrollerModule(CP-2584-G2-SATA-APL-LS-STD128)

● 1xE5-2630v2CPUs● 128GBRAM● 2xbootSSDs● 2x10Gbnetworkports

Object(O)NodeHDDs 6● 84x6TBSeagateSATAHDDs

ModelST6000NM0024

Top-of-RackSwitches 2 ● Arista7150S-24-R24-port10GbESFP+switch

ManagementSwitch 1 ● BrocadeICX6610-2424-port1GbERJ45switch

Note:Networkcables,datacenterracks,andpowerdistributionunitsarenotincludedinthehardwarelistingabove.SAScablesarenotneeded,sincethecontrollerisembeddedintheenclosure.

Table:ThecomponentsoftheembeddedserversystemtestedbySwiftStackandSeagate.


Preparing for Testing Todemonstratethesupportofmultipleoperatingsystems,theexternalserverconfigurationwasinstalledwithCentOSv6.6,andtheembeddedserverconfigurationwasinstalledwithUbuntu14.04.3.Inbothcases,thedual10Gbportswerebondedusingmode4(IEEE802.3addynamiclinkaggregation),andtheSASHBAdriversusedwerefromthesamepackageastheSASHBAfirmware.FollowingOSinstallation,theSwiftStackobjectstoragesoftwarewasinstalledusingtheprocedureoutlinedearlier.OncetheSwiftStacksoftwarewasinstalled,minimalconfigurationandpreparationwasrequiredintheSwiftStackcontrollertoprepareforbenchmarktesting.(Note:Sincethistesting,SwiftStacksoftwarenowdefaultstoan“auto”settingfortheseparametersthatautomaticallyoptimizesthesesettingswithouttheneedformanualtuning.)Specifically,thefollowingtuningsettingswereappliedaccordingtothebestpracticesinSwiftStack’sdocumentationathttps://swiftstack.com/docs/admin/cluster_management/ss_tuning.html:

Setting Value

SwiftProxyWorkers 24

SwiftObjectWorkers 24

ObjectServersperPort 4

SwiftAccountWorkers 24

SwiftContainerWorkers 24

NetworkTransmitHashPolicy(onswitchesandLinuxnodes)

Layer3+4IEEE802.3adDynamicLinkAggregation

LoadBalancing Noloadbalancingwasused;eachload-generatorconnecteddirectlytooneproxynode

Table:ThevaluesofafewkeyparametersintheSwiftStackcontrollerusedduringtheSwiftStackSeagatetestingTests Performed ssbench Tool

SwiftStackhasenhancedaquickandeffectivetool—called“ssbench”—forbenchmarkingobjectstorageclustersthatsupporttheSwiftAPI;itisfreelyavailableathttps://github.com/swiftstack/ssbench.Priortoeachtestexecution,apredeterminednumberofcontainersandobjectswerepre-populatedinthecluster,thenthetestsperformedamaximum


numberofPUTStotheclusterorGETSfromtheclusteracrossarangeofobjectsizeswithvaryinglevelsofconcurrentclientconnections—eachfora120-secondperiodoftime.Twoexamplesof“scenariofiles”usedtodeterminehowabenchmarktestwillexecuteareprovidedhere:Example: Testing reads of 1kB objects

{ "name": "1KB-GET test scenario", "sizes": [{ "name": "1KB-GET", "size_min": 1024, "size_max": 1024 }], "initial_files": { "1KB-GET": 1000 }, "run_seconds": 120, "crud_profile": [0, 1, 0, 0], "user_count": 100, "container_base": "ssbench-1KB", "container_count": 100, "container_concurrency": 10}Example: Testing writes of 1MB objects

{ "name": "1MB-PUT test scenario", "sizes": [{ "name": "1MB-PUT", "size_min": 1048576, "size_max": 1048576 }], "initial_files": { "1MB-PUT": 100 }, "run_seconds": 120, "crud_profile": [1, 0, 0, 0], "user_count": 100, "container_base": "ssbench-1MB", "container_count": 100, "container_concurrency": 10}


Interpretation of Results: Meeting Real-World Requirements Forthesebenchmarks,PUTsandGETsofobjectswithsizesrangingfrom1KBto5GBweremadeusingarangeofconcurrentworkersfrom100to2,100clients.Thetwohardwareconfigurationsdescribedearlierwerebothtested—onewithexternalobjectserversandonewithinternalAP-LS-1controllers,andbothreplica-basedanderasure-coding-baseddataprotectionschemeswereused.Thebenchmarktestingcapturedresultsthatincludethetotalnumberofobjectswrittenorread,latency,throughput,andmore.The“ExternalObjectServer”resultswerecapturedusingexternal1uIntelserversrunningtheSwiftobjectservicesattachedtotheSeagate5u84JBODenclosures.The“AP-LS-1Controller”resultswerecapturedusinginternalAP-LS-1controllermodulesintheSeagate5u84enclosurestoruntheSwiftobjectservices(i.e.,noexternallyattachedIntelserver).Thedataprotectionpoliciesusedeitherthreereplicasorerasurecoding(seehttps://swiftstack.com/docs/admin/cluster_management/policies.html).Withthreereplicas,everyobjectwrittenbyaclienttoaproxynodeisthenphysicallystoredthreetimesintheclusterofobjectnodes,butreadsrequireaccesstoonlyonedrivewithacopyoftheobject.Typically,thisresultsinhigherdatadurabilityguaranteesandbettersmall-objectperformanceatthecostofphysicalspaceutilizationrelativetoerasurecoding.Theerasurecodingbenchmarksusedan8+4erasurecodingstoragepolicy,whichmeansthateveryobjectwrittenbyaclienttoaproxynodeisthenphysicallystoredaseightsegmentsplusanadditionalfourparitysegmentsintheclusterofobjectnodes,andreadsofthatobjectrequireaccesstoanyeightofthosetwelvetotalsegments.Typically,thisresultsinphysicalspacesavingsandbetterlarge-objectperformancerelativetoreplica-basedstoragepolicies.Manyreal-worldapplicationstendtofallingroupingsofobjectsizeandconcurrencyofclientconnections;forexample,combinationsoftenstohundredsofclientswithmid-to-largeobjectsizeswouldbetypicalofmanybackupandarchiveusecases.Tofindresultsrelevanttoaparticularusecaseofinterest,notetheroughdescriptionoftherelevantworkload,andobservetheresultsforthatcombinationofobjectsizeandclientconcurrencyinthefollowingsections.


CommonSwiftStackUseCases Concurrency(average)

ObjectSize(average)

Backup 10sto100s 10sofMB

Archive(Generic) 10sto100s 10sor100sofMB

ActiveArchiveofGeneticSequencingData 10sto100s 100sofMBor1sofGB

FileCollaboration(“Sync-and-Share”) 100sto1000s 100sofKBto10sofMB

WebApplication 1000s 1s/10s/100sofKB

MediaFileIngest 100s 10sto100sofMB

MediaBroadcast 10sto100s 100sofMB

MediaEditing 10sto100s 10sto100sofMB

ServiceProvider/PrivateCloud Varieswidely Varieswidely

Other Varieswidely Varieswidely

Table:Adescriptionoftypicalconcurrencyandobjectsizevaluesastheycorrespondtodifferentusecases. Asaspecificexample,aSwiftStackcustomer(whohappenstobeabackup-as-a-serviceprovider)recentlystatedthefollowingrequirement:“Oneofourcustomers,whoisintobackupandarchive,hasatargetcapacityof~1PB.At1PB,theyexpectadailychangerate(8hourwindow)of~1.5-3%,whichmeanstheyneedanObjectStorethatcanhandlebetween1.8TBand3.6TBperhourofsustainedwrites,i.e.,around1GB/s.”Forthisscenario,notethatthebackupusecaseisgoingtofallmostcommonlyintothegroupingofmid-to-largeobjectsizesandsmallclientconcurrency.Forthiscustomer’susecase,onecanobserveintheresultsthatfollowthewritesatconcurrencycountsof“99”and“300”aroundthe10MB,100MB,and500MBobjectsizes:Notethat,usingexternalobjectserversforexample,totalclientwritethroughputfortheparticularsetofparameterslistedrangedfromabout1500MB/secupto2600MB/sec(includingbothreplicaanderasurecodingresults)—allwellabovetherequired1GBytes/sec.


Results: Raw Numbers Thissectionshowstheresultsoftheperformancetestingforthetwohardwareconfigurationsforbothreplicationanderasurecodingstoragepolicies

External Servers: Three Replicas

ThissectionshowstheresultsoftheexternalserverconfigurationwithastoragepolicyofthreereplicaswithCentOSv6.6astheoperatingsystem.Recallthatforwrites,thisrequiresaproxytosendoutthreewritecommandstoACOserversforeachwritecommanditreceivesfromtheclients.

Write Throughput (MB/sec)

ConcurrentClients

1kBObjects

1MBObjects

10MBObjects

100MBObjects

500MBObjects

5GBObjects

99 0.8 603 1,496 1,650 1,473 1,291

300 1.87 1,020 1,608 1,654 1,655 1,666

600 2.92 996 1,692 1,734 1,553 1,507

900 3.36 1,109 1,550 1,414 1,359 888

1,200 3.42 1,142 1,353 1,420 1,444 905

1,500 3.49 977 1,406 1,312 988 938

1,800 3.48 1,095 1,314 1,450 996 1,239

2,100 3.54 1,190 1,363 1,226 741 1,354

Table:Writethroughputresultsfortheexternalserverconfiguration(Seagate5u84JBODsattachedtoexternalserversrunningSwiftStackobjectservices)withastoragepolicyofthreereplicas.Asthenumberofconcurrentclientsandthesizeofobjectswasvaried,writethroughputscaleduptofullavailablenetworkbandwidthutilizationresultingin1.5GB/sofobjectthroughput.


Figure:Agraphofwritethroughputresultsfortheexternalserverconfiguration(Seagate5u84JBODsattachedtoexternalserversrunningSwiftStackobjectservices)withastoragepolicyofthreereplicas.Asthenumberofconcurrentclientsandthesizeofobjectswasvaried,writethroughputscaleduptofullavailablenetworkbandwidthutilizationresultingin1.5GB/sofobjectthroughput.


Read Throughput (MB/sec)

ConcurrentClients 1kBObjects

1MBObjects

10MBObjects

100MBObjects

99 4.83 2,189 3,255 3,561

300 8.11 2,390 3,190 2,982

600 8.87 2,474 3,399 3,178

900 9.19 2,621 2,961 3,392

1200 9.07 2,300 3,016 3,245

1500 9.02 2,374 2,907 3,050

1800 8.97 2,344 2,946 3,250

2100 8.91 2,378 2,687 2,986

Table:Readthroughputresultsfortheexternalserverconfiguration(Seagate5u84JBODsattachedtoexternalserversrunningSwiftStackobjectservices)withastoragepolicyofthreereplicas.Asthenumberofconcurrentclientsandthesizeofobjectswasvaried,readthroughputscaleduptofullavailablenetworkbandwidthutilizationresultingin3GB/sofobjectthroughput.

Figure:Agraphofreadthroughputresultsfortheexternalserverconfiguration(Seagate5u84JBODsattachedtoexternalserversrunningSwiftStackobjectservices)withastoragepolicyofthreereplicas.Asthenumberofconcurrentclientsandthesizeofobjectswasvaried,readthroughputscaleduptofullavailablenetworkbandwidthutilizationresultingin3GB/sofobjectthroughput.


External Servers: 8+4 Erasure Coding

Thissectionshowstheresultsoftheexternalserverconfigurationwithastoragepolicyof8+4erasurecodingwithCentOSv6.6astheoperatingsystem.


ConcurrentClients

1kBObjects

1MBObjects

10MBObjects

100MBObjects

500MBObjects

5GBObjects

99 0.32 306 1,594 2,678 2,649 2,123

300 0.62 531 2,049 2,434 2,664 2,182

600 0.66 556 2,076 2,705 2,564 1,994

900 0.68 554 2,203 2,339 1,815 1,913

1200 0.68 560 2,028 2,659 2,123 1,196

1500 0.68 556 2,080 2,158 1,985 1,511

1800 0.68 554 2,088 2,133 1,418 1,391

2100 0.67 549 2,110 2,077 1,470 1,151

Table:Writethroughputresultsfortheexternalserverconfiguration(Seagate5u84JBODsattachedtoexternalserversrunningSwiftStackobjectservices)withastoragepolicyof8+4erasurecoding.Asthenumberofconcurrentclientsandthesizeofobjectswasvaried,writethroughputscaleduptofullavailablenetworkbandwidthutilizationresultinginnearly2.5GB/sofobjectthroughput.

Figure:Agraphofwritethroughputresultsfortheexternalserverconfiguration(Seagate5u84JBODsattachedtoexternalserversrunningSwiftStackobjectservices)withastoragepolicyof8+4erasurecoding.Asthenumberofconcurrentclientsandthesizeofobjectswasvaried,writethroughputscaleduptofullavailablenetworkbandwidthutilizationresultinginnearly2.5GB/sofobjectthroughput.



ConcurrentClients 1kBObjects 1MBObjects 10MBObjects 100MBObjects

99 2.17 1,225 2,593 3,835

300 2.56 1,439 2,687 3,808

600 2.63 1,461 2,631 3,326

900 2.67 1,443 2,452 3,246

1200 2.66 1,421 2,760 3,202

1500 0.93 1,384 2,407 2,941

1800 2.55 1,414 2,180 2,842

2100 2.56 1,415 2,331 3,154

Table:Readthroughputresultsfortheexternalserverconfiguration(Seagate5u84JBODsattachedtoexternalserversrunningSwiftStackobjectservices)withastoragepolicyof8+4erasurecoding.Asthenumberofconcurrentclientsandthesizeofobjectswasvaried,readthroughputscaleduptofullavailablenetworkbandwidthutilizationresultinginnearly4GB/sofobjectthroughput.

Figure:Agraphofreadthroughputresultsfortheexternalserverconfiguration(Seagate5u84JBODsattachedtoexternalserversrunningSwiftStackobjectservices)withastoragepolicyof8+4erasurecoding.Asthenumberofconcurrentclientsandthesizeofobjectswasvaried,readthroughputscaleduptofullavailablenetworkbandwidthutilizationresultinginnearly4GB/sofobjectthroughput.


AP-LS-1 Controller: Three Replicas

ThissectionshowstheresultsoftheembeddedserverconfigurationwithastoragepolicyofthreereplicaswithUbuntu14.04.3astheoperatingsystem.Recallthatforwrites,thisrequiresaproxytosendoutthreewritecommandstoACOserversforeachwritecommanditreceivesfromtheclients.


ConcurrentClients

1kBObjects

1MBObjects

10MBObjects

100MBObjects

500MBObjects

5GBObjects

99 1.08 668 1,091 1,409 1,380 1,393

300 2.32 484 832 1,064 1,098 981

600 2.93 471 475 994 1,014 880

900 2.98 400 490 872 755 722

1200 2.98 379 440 729 714 615

1500 3.00 367 472 715 584 413

1800 2.97 346 437 561 645 360

2100 2.98 339 393 529 436 467

Table:Writethroughputresultsfortheembeddedserverconfiguration(Seagate5u84JBODswithembeddedSeagateAP-LS-1controllersrunningSwiftStackobjectservices)withastoragepolicyofthreereplicas.Thebestperformancewasseenwithlargeobjectsizesandsmallnumbersofconcurrentclientconnections.

Figure:Agraphofwritethroughputresultsfortheembeddedserverconfiguration(Seagate5u84JBODswithembeddedSeagateAP-LS-1controllersrunningSwiftStackobjectservices)withastoragepolicyofthreereplicas.Thebestperformancewasseenwithlargeobjectsizesandsmallnumbersofconcurrentclientconnections.




99 5.86 2,115 2,890 3,517

300 8.46 1,601 1,882 2,465

600 9.45 1,196 1,419 1,708

900 9.63 1,010 1,106 1,417

1200 9.61 923 1,138 1,246

1500 9.65 762 1,115 1,181

1800 9.73 763 1,143 1,032

2100 9.73 748 1,091 1,040

Table:Readthroughputresultsfortheembeddedserverconfiguration(Seagate5u84JBODswithembeddedSeagateAP-LS-1controllersrunningSwiftStackobjectservices)withastoragepolicyofthreereplicas.Thebestperformancewasseenwithlargeobjectsizesandsmallnumbersofconcurrentclientconnections.

Figure:Agraphofreadthroughputresultsfortheembeddedserverconfiguration(Seagate5u84JBODswithembeddedSeagateAP-LS-1controllersrunningSwiftStackobjectservices)withastoragepolicyofthreereplicas.Thebestperformancewasseenwithlargeobjectsizesandsmallnumbersofconcurrentclientconnections.


AP-LS-1 Controller: 8+4 Erasure Coding

Thissectionshowstheresultsoftheembeddedserverconfigurationwithastoragepolicyof8+4erasurecodingwithUbuntu14.04.3astheoperatingsystem.



99 0.35 281 1,005 1,399

300 0.60 445 672 745

600 0.72 418 611 634

900 0.72 390 620 701

1200 0.73 376 420 631

1500 0.73 367 416 512

1800 0.70 433 308 466

2100 0.69 348 290 367

Table:Writethroughputresultsfortheembeddedserverconfiguration(Seagate5u84JBODswithembeddedSeagateAP-LS-1controllersrunningSwiftStackobjectservices)withastoragepolicyof8+4erasurecoding.Thebestperformancewasseenwithlargeobjectsizesandsmallnumbersofconcurrentclientconnections.

Figure:Agraphofwritethroughputresultsfortheembeddedserverconfiguration(Seagate5u84JBODswithembeddedSeagateAP-LS-1controllersrunningSwiftStackobjectservices)withastoragepolicyof8+4erasurecoding.Thebestperformancewasseenwithlargeobjectsizesandsmallnumbersofconcurrentclientconnections.




99 2.67 1,051 1,739 1,963

300 3.08 926 1,302 1,327

600 3.14 773 1,059 1,153

900 3.11 782 862 1,337

1200 2.04 712 742 1,003

1500 3.04 680 704 922

1800 2.96 637 708 843

2100 2.94 516 666 732

Table:Readthroughputresultsfortheembeddedserverconfiguration(Seagate5u84JBODswithembeddedSeagateAP-LS-1controllersrunningSwiftStackobjectservices)withastoragepolicyof8+4erasurecoding.Thebestperformancewasseenwithlargeobjectsizesandsmallnumbersofconcurrentclientconnections.

Figure:Agraphofreadthroughputresultsfortheembeddedserverconfiguration(Seagate5u84JBODswithembeddedSeagateAP-LS-1controllersrunningSwiftStackobjectservices)withastoragepolicyof8+4erasurecoding.Thebestperformancewasseenwithlargeobjectsizesandsmallnumbersofconcurrentclientconnections.


Understanding Bottlenecks

Wheneverperformanceismeasured,the“bottleneck”isthecomponent(orsometimescomponents)ofthesystemthatlimitperformancefrombeinganyhigher.Bottlenecksarenotnecessarilyabadthing;theyaresimplythelimitingfactorinagivensystemandtestscenario,andremovalofonebottleneckwillalwaysrelegatethebottlenecktosomeotherpartofthesystem.Toimproveperformance,therefore,itisimportanttoidentifybottlenecksandunderstandifandhowtheycanberemovedandhowmuchimprovementeachchangemightoffer.ForadistributedstoragesystemlikeSwiftStack,foragivenworkload,bottlenecksmightbefoundatthepointoftheclientsystems’CPUs,thenetworkconnectingclientstotheSwiftStackcluster,aloadbalancer,theSwiftStackproxynodes,theintra-clusternetwork,ortheCPUsand/ordrivesintheSwiftStackobjectnodes—amongotherthings.Inthebenchmarkconfigurationsdescribedabove,differentbottleneckswereobservedindifferenttestscenarios.Forexample,forsmall-objectwritesinthethree-replicascenarios,theCPUsontheACOnodesreachedsustained100%utilizationduringtestingandthuswerethebottleneck;byusingCPUswithmorecores,theclusterwouldbeabletodomoreofthenecessarymathtocalculatechecksumsofobjectsastheyarestored,whichwouldallowforprocessingmoreobjectwritespersecond—resultingingreateroverallthroughputfromtheclientintothecluster.Astheobjectsizeincreased,however,notethatthebottleneckshiftedfromtheACOnodeCPUstotheproxynodes’networkbandwidth:Eachofthethreeproxynodeshadabout20Gb/softotalavailablebandwidth(from2x10Gbbondednetworkports),andinthethree-replicawritescenario,eachobjectwasreceivedbyaproxynodeandthensentthreetimestoobjectnodes,so25%ofthatavailablebandwidthwasavailabletoclienttraffic,andtheremaining75%wasusedforproxy-node-to-object-nodetraffic,thuseachproxynodecontributedabout500MB/secoftotalclientthroughputtothecluster,whichalignswiththeobservedupperlimitofaround1,500MB/secofwritethroughputintothecluster.Toimprovethatresult,moreproxynodebandwidth(orperhapsmoreproxynodes)wouldberequired.


CONCLUSION

Inconclusion,bycombiningthepowerandflexibilityofSwiftStacksoftwarewiththedensityofSeagate’sCP-2584enclosure,customerscanachieveatrulyscalable,cost-effective,andeasy-to-deploy-and-manageobjectstoragesolutionformanyoftoday’sdata-heavyusecases—includingenterprisebackupandarchive,mediaandlarge-dataarchivesintheMedia&EntertainmentandLifeSciencesindustries,filecollaboration,privatecloudstorageforCloudStackandOpenStack,andmanywebapplicationsleveragingobjectstorage.Descriptionsofthecomponenttechnologieswerediscussed;detailsoffourreferencearchitectureswereprovided(twohardwareconfigurationswithtwostoragepolicies);andbenchmarkresultsweredescribed.MoreinformationcanbefoundfromSwiftStackandSeagateattheirrespectivewebsites:www.swiftstack.com|www.seagate.com

Documents

Reference Design: SwiftStack and Seagate Scale- Out Object ... · Reference Design: SwiftStack and Seagate Scale-Out Object Storage Solution June 2016. ... This white paper provides