DRIVESCALE-HDPREFERENCEARCHITECTURE

April2017

1

Contents1. ExecutiveSummary...........................................................................................................2

2. AudienceandScope..........................................................................................................3

3. GlossaryofTerms..............................................................................................................3

4. DriveScale|HortonworksDataPlatform-ApacheHadoopSolutionOverview..............5

5. DriveScaleComponentsOverview....................................................................................7

4.1Hardware:DriveScaleAdapterChassiswithDriveScaleControllers.....................................7

4.2Software................................................................................................................................7

4.3ConceptualdiagramofDriveScalesolution..........................................................................9

6. BenefitsoftheDriveScalesolution..................................................................................10

7. ReferenceArchitectureDetails........................................................................................11

6.1PhysicalClusterComponentsandConfigurationList.........................................................11

6.2LogicalClusterTopology.....................................................................................................12

6.3PhysicalClusterTopology...................................................................................................13

6.4ClusterManagement..........................................................................................................13

6.4.1SettingupDriveScalecluster...........................................................................................13

6.4.2SettingupHortonworksDataPlatformcluster................................................................16

6.5DiskandFilesystemLayout.................................................................................................17

6.6OSSupportability/CompatibilityMatrix..............................................................................17

8. RackScalability................................................................................................................17

9. References.......................................................................................................................18

10. BillofMaterials..............................................................................................................19

11. Conclusion.....................................................................................................................20

2

1. ExecutiveSummary

Thisdocumentisahigh-leveldesignreferencearchitectureguideforimplementingHortonworksDataPlatformonaDriveScalesolutionwithindustrystandardserversandJBODs.Thereferencearchitectureintroducesallthehigh-levelcomponents,hardware,andsoftwarethatareincludedinthestack.Eachhigh-levelcomponentisthendescribedindividually.ThereferencearchitecturedoesnotdescribetheHortonworksDataPlatformcomponentsortheirapplications.DriveScaleTechnologyOverviewDriveScale is leading the charge in bringing hyper scale computing capabilities tomainstream enterprises. Its composable data center architecture transforms rigid datacenters into flexible and responsive scale-out deployments.UsingDriveScale, data centeradministrators can deploy independent pools of commodity compute and storageresources, automatically discover available assets, and combine and recombine theseresourcesasneeded.Thesolutionisprovidedthroughasetofon-premisesandSaaStoolsthat coordinate between multiple levels of infrastructure. With DriveScale, HadooparchitectscanmoreeasilysupportHadoopdeploymentsofanysizeaswellasothermodernapplicationworkloads.DriveScaleprovideshardwareandsoftwaretechnologythatallowsseparatedeploymentofcomputeandstorageusingcommoditydisklessserversandJBODs(JustaBoxofDisks),withflexiblebindingofstorage-to-computeresourcesinanyratiorequiredbyanapplication.Asneeds change, these bindings can be dissolved and reconfigured on demand, all undersoftwarecontrol.DriveScale technology acquires a deep understanding of the physical infrastructure anddynamics of a data center,which it uses to provide an integrated set of intelligence andautomationtoolstoscale-outdatacenterinfrastructuretogreatlysimplifyandoptimizethedatacenter’soperations.HortonworksDataPlatformTechnologyOverviewHDPistheindustry'sonlytruesecure,enterprise-readyopensourceApache™Hadoop®distributionbasedonacentralizedarchitecture(YARN).YARN(YetAnotherResource

3

Negotiator)allocatesresourcesamongvariousapplicationsandmaximizesdataingestionbyenablingenterprisestoanalyzedatatosupportdiverseusecases.YARNcoordinatescluster-wideservicesforoperations,datagovernanceandsecurity.HDPisinteroperablewithabroadecosystemofdatacenterandcloudproviders,andprovidescentralizedmanagementandmonitoringofclusters.WithHDP,securityanddatagovernanceisbuiltintotheplatform.HDPaddressesthecompleteneedsofdata-at-rest,powersreal-timecustomerapplicationsanddeliversrobustanalyticsthatacceleratedecisionmakingandinnovation.

2. AudienceandScope

This referencearchitectureguide is forHadoopand IT architectswhoare responsible forthedesignanddeploymentofApacheHadoopsolutionsonpremises,aswellasforApacheHadoopadministratorsandarchitectsanddatacenterarchitects/engineerswhocollaboratewithspecialistsinthatspace.

3. GlossaryofTerms

Term Description

DataNode WorkernodesoftheclustertowhichtheHDFSdataiswritten.

HBA HostBusAdapter.AnI/Ocontrollerthatisusedtointerfaceahostwithstoragedevices.

HDD HardDiskDrive.

HDFS Apache™Hadoop®DistributedFileSystem.

HighAvailability

(HA)

Configurationthataddressesavailabilityissuesinacluster.Inastandardconfiguration,theNameNodeisasinglepointoffailure(SPOF).EachclusterhasasingleNameNode,andifthatmachineorprocessbecomesunavailable,theclusterasawholeisunavailableuntiltheNameNodeiseitherrestartedorbroughtuponanewhost.ThesecondaryNameNodedoesnotprovidefailovercapability.HighAvailabilityenablesrunningtwoNameNodesinthesamecluster:theactiveNameNodeandthestandbyNameNode.ThestandbyNameNodeallowsafastfailovertoanewNameNodeincaseofmachinecrashorplannedmaintenance.

4

JBOD JustaBunchofDisks.JBODisanalternativetousingaRAIDconfiguration.RatherthanconfiguringdrivestouseaRAIDlevel,thediskswithinthearrayareeitherspannedortreatedasindependentdisks.

JobHistoryServer

Processthatarchivesjobmetricsandmetadata.Onepercluster.

NameNode ThemetadatamasterofHDFSessentialfortheintegrityandproperfunctioningofthedistributedfilesystem.

NodeManager TheprocessthatstartsapplicationprocessesandmanagesresourcesontheDataNodes.

NIC NetworkInterfaceCard.

HDP™ Hortonworks®DataPlatform(thisincludestheHadoopDistributedFileSystemHDFS)

PDU PowerDistributionUnit.

NTP NetworkTimeProtocol

YARN YetAnotherResourceNegotiator,whichisasoftwarerewritethatdecouplesMapReduce'sresourcemanagementandschedulingcapabilitiesfromthedataprocessingcomponent,enablingHadooptosupportmorevariedprocessingapproachesandabroaderarrayofapplications.IncludedinHDP.

OS OperatingSystem

RM ResourceManager.TheresourcemanagementcomponentofYARN.ThisinitiatesapplicationstartupandcontrolsschedulingontheDataNodesofthecluster(oneinstancepercluster).

ToR TopofRack.

ZK ZooKeeper.Acentralizedserviceformaintainingconfigurationinformation,naming,andprovidingdistributedsynchronizationandgroupservices.

DSC DriveScaleCentral.Aweb-baseduserinterfacetotheDriveScalecloudthatperformsDriveScaleaccountmanagement.DSCiswhereyou

5

findanddownloadthekeystoenableinstallationoftheDriveScalesoftware,andthensetupyourDriveScaleManagementDomain(s)(DMDs).

DMD DriveScaleManagementDomain(s).Thisiswhereyoucreateyourdomain,selectandconfiguretheDMSnodesforthedomain,andselectachassis(withitsassociatedDriveScaleAdapters,DSAs)forthedomain.

DMS

DriveScaleManagementServer.Thisistheserverthatrunsthebundleofsoftware(service)thatmanagesasetofphysicalresourcestoenabletheDriveScaleservices.DriveScaleManageristheweb-baseduserinterfacetotheDMS.

DSAchassis DriveScaleAdapterchassisisa1RUchassisthathosts4EthernettoSAScontrollersservingasabridgebetween10GbpsEthernetconnectingcomputeresourcestoJBODsfullofcommoditydisks.

DSAcontroller DriveScaleAdaptercontrollerisanEthernettoSAScontrollersservingasabridgebetween10GbpsEthernetconnectingcomputeresourcestoJBODsfullofcommoditydisks.

MLAG Multi-chassisLinkAggregation.MLAGistheabilityoftwoormoreswitchestoactasasingleswitchwhenforminglinkbundles.

4. DriveScale|HortonworksDataPlatform-ApacheHadoopSolutionOverview

TheDriveScale|HortonworksDataPlatform(HDP)solutionisdesignedtoaddressthechangingrequirementsfromcustomersforamoreflexibleanddynamichardwareinfrastructurethatprovidessignificantcostandoperationalbenefits.Itisdesignedwithcomposabilityastheprimarygoal,savingmoney,improvingutilizationandgreatlysimplifyingthedeploymentofHadoopclusters.HadoopisanApacheprojectbeingdevelopedintheJavaprogramminglanguagebyaglobalcommunityofcontributors.Yahoo!,hasbeenthelargestcontributortothisproject,andusesApacheHadoopextensivelyacrossitsbusinesses.CorecommittersontheHadoopprojectincludeemployeesfromCloudera,eBay,Facebook,Getopt,Hortonworks,Huawei,IBM,InMobi,INRIA,LinkedIn,MapR,Microsoft,Pivotal,Twitter,UCBerkeley,VMware,WANdisco,andYahoo!,withcontributionsfrommanymoreindividualsandorganizations.

6

AlthoughHadoopispopularandwidelyused,installing,configuring,andrunningaproductionHadoopclusterinvolvesmanyconsiderations,including:

• ChoosingtheappropriateHadoopsoftwaredistributionandextensions• Installingmonitoringandmanagementsoftware• AllocationofHadoopservicestophysicalnodes• Selectionofappropriateserverhardware• Rightsizingthestorageconfiguration• Implementingdatalocality• Designofthenetworkfabric• Sizingandsystemscalability• Overallperformance

Thisiscomplicatedbytheneedtounderstandtheworkloadsthatwillberunningonthecluster,thefast-movingpaceofthecoreHadoopproject,andthechallengestomanagingasystemdesignedtoscaletothousandsofnodesinasingleinstance.

TheDriveScale|HortonworksDataPlatformsolutionstogetherembodiesallthehardware,software,resourcesandservicesneededtorunHadoopinaproductionenvironment.Thisend-to-endapproachmeansthatyoucanbeinproductionwithHadoopinashortertimethanistypicallypossiblewithhomegrownsolutions.Todeliver thecomputeandstoragepoweradatacenterneeds, this solution isbasedonHDP, a secure, enterprise-ready open source Apache Hadoop distribution built on acentralized YARN architecture, DriveScale hardware and software, industry standardservers,networksswitches,andJBODsbuiltfromcommoditydiskdrives.Thissolutionincludescomponentsthatspantheentiresolutionstack:

• Referencearchitectureandbestpractices• Optimizedstorageconfigurations• Optimizednetworkinfrastructure• HDP

ItisdesignedtoaddressthevastmajorityofApacheHadoopusecasesincluding,butnotlimitedto:

• Bigdataanalytics• ETLOffload• DataWarehouseOptimization• Bathprocessingofunstructureddata

7

• Bigdatavisualization• Searchandpredictiveanalysis

5. DriveScaleComponentsOverview DriveScalesystemiscomposedofonehardwarecomponentandfoursoftwarecomponents:

4.1 Hardware: DriveScale Adapter Chassis with DriveScale Controllers

Thisisa1Uappliancewithadaptersthatconnecttoserversvia10GbEthernetinterfacesandtoJBOD’sviaSASinterfaces.

Figure1:DriveScaleAdapter

4.2Software

TherearefourprincipalcomponentsoftheDriveScalesoftware:a) DriveScaleManagementServer(DMS)

• TheserverrunningtheDMSsoftwarebundleiscalledtheDMSnode.• AtypicaldeploymentconsistsofthreeDMSSystemsinaclusteredforhigh

availability(HA).• Thesoftwaremanagesandconfigureresourcesandcontainsthe

inventory/configurationinformationrepositoryanddatabase:ü Inventory:DMS’s,DSAdapters,switches,JBODchassis,disks,servernodesü Configuration:nodetemplates,clustertemplates,configuredclustersü DMSDatabase:usedasamessagebustocommunicatewiththeendpoints.

8

b) DriveScaleServerAgent• DriveScaleServerAgentdiscoveryactionprovidesinventoryforhardwareand

servers,andcreatesmappingsbetweenservernodesandthediskstheyconsume.

c) DriveScaleCentral(DSC)Cloud-basedsoftwaremanagementportalthatactsasthe:o softwaredistributionrepositoriesforsubscriberso DriveScalekeysrepositoryo centralizedlogfilerepositoryo userdocumentationrepositoryo licensemanager

d) DriveScaleAdapterFirmware

• AttheheartoftheclusterwheretheDMSisrunning,thefirmwareontheprocessorenablestheJBODstobemappedtotheserversandusedaslocaldrives.

9

4.3ConceptualdiagramofDriveScalesolution

Figure2:DriveScaleClustercomponentsoverview

10

6. BenefitsoftheDriveScalesolutionTheDriveScalesolutionfornext-generationScale-Outarchitecturedisaggregatesserversinclustersintopoolsofindependentcomputeandstorageresources.DriveScaledisaggregatesstoragefromcomputenodesattheracklevel,allowingdatacenterstobuycommoditydisklessservernodesandJBODsfromtheirpreferredvendorsandinstalltheminracks.DriveScale’sadvancedsoftwaremanagestheorchestrationofserversandclustersfromthepoolsofdrivesandcomputenodesthroughaGUIbuiltonaRESTfulAPI.Administratorscanprovision,decommission,andre-provisionserversandclustersdynamically,asneeded.WithDriveScale,theminimumclustersizeistheminimumsizeofaHadoopcluster.

• SoftwareDefinedorElasticinfrastructureforHadoopclusterWithDriveScalesolution,allthecomputeandstorageresourcesaresharedandcanbedeployedatwill.AdministratorscanprovisionclustersinminutesinsteadofdaysandgetsignificantlybetterutilizationoftheirhardwarebyusingDriveScale’sInfrastructureProvisioningandManagementtool.AdministratorscanbuildasingleresourcepoolormultiplesmallresourcepoolsforHadoopapplications.Theseresourcepoolscanbemodifiedondemandtoquicklyrespondtothechangingworkloadneeds.

• IntegrationDriveScale’ssolutionworksseamlesslywiththedatacenter’sexistingbest-in-classorcommodityserverandJBODtechnology.

• ScalabilityWithDriveScalesolution,thedatacentercanstartwithasmallclusterwithfewcomputenodesandasingleJBODandlaterscalestorageandcomputeseparatelyastheclusterstartstorunoutofresources,withoutcausinganydisruption.

• ImprovedutilizationWithDriveScalesolution,adatacentercanreplacetheserversanddrivesseparately,therebymaximizingthelifetimeofthehardwareinfrastructure.

• SimplyeverythingCustomerscanchoosetokeeptheirexistingbest-in-classequipment,oraddcommodityhardwaretobuildacomposableHadoopinfrastructurethatcaneasilybemodified.Addingorremovingcomputeorstoragecapacityisdonewithjustafewclicks.

11

7. ReferenceArchitectureDetails

6.1PhysicalClusterComponentsandConfigurationList

Thefollowingtableliststhephysicalcomponentsforthecluster.

Component Configuration Description QuantityDriveScaleAdapter

ChassisDHCP,Jumboframeenabled

1UappliancewithadaptersthatconnecttoserversviaEthernet,andtoJBOD’sviaSAS.

1

DriveScaleAdaptercontroller

DHCP,Jumboframeenabled

Providesthedatanetwork. 4foreachchassis

DriveScaleManagementServer(DMS)

DMSrunningonaVM ManagesandconfiguresthenodesandDriveScaleclusterandalsostorestheinventory/configurationrepositoryofeveryhardwareinthecluster.

Min1,forHA3DMS’sshouldbeconfiguredasmasterandslave

Servers 2socketCPUandmemorypertheindividualHadoopclusterrequirements

Commodityx86serversthathousealltheNodeManager,computeinstancesandDriveScaleagents.

Min1Namenodes+3Datanodes

HDDforServers 2drivesconfiguredinRAID1

TheinternaldrivesareusedforOSinstall.

2foreachserver

NICs Dual-port10GbpsEthernetNICs.Theconnectortypedependsonthenetworkdesign;couldbeSFP+orTwinax.

Providesthedatanetwork Min1foreachserver

JBOD Defaultconfiguration HousesthedrivewithdualIOcontrollers.

Min1

HDDforJBOD Defaultconfiguration Drivestohousethedataforthecluster.

Dependingontheclusterrequirements

ToR10Gswitch LLDP,MLAG,JumboFrame9Kconfigured

Providesdatanetworkconnectivity.

2foreachrack

ToR1Gswitch Defaultconfiguration Providesmanagementnetworkconnectivity.

1foreachrack

AmbariServer AmbariserverrunningonaVM

Manages,configuresandmonitorstheHDPHadoopcluster.

1foreachenvironment

12

6.2LogicalClusterTopology

Theminimumrequirementstobuildouttheclusterare:

● 1NameNode● 4DataNodes● 1DriveScaleAdapterChassis● 1DriveScaleManagementServer● 210GSwitches● 11GSwitch● 1JBODchassiswithdrives● 1DMS● 1Ambariserver

Thisreferencearchitectureisbuilton1namenodeand4datanodeswith1JBODand60drivesof1or2or3TBHDD.Thefollowingtableliststheconfigurationsoftheserversandnumberofdrivesused.

Component Configuration Description QuantityNamenode 2socket20coreCPU,

256GBRAM,10GbEIntelNICwith2internalHDDforOSand4highcapacityHDDmountedfromtheJBOD.

NamenodehoststheHDPnameservicesandDriveScaleagents.

1

Datanodes 2socket16coreCPU,256GBRAM,10GbEIntelNICwith2internalHDDforOSand8highcapacityHDDmountedfromtheJBOD.

DatanodeshousetheHDFSDataNodesandYARNNodemanagers,anyadditionalrequiredservicesandDriveScaleagents.

4

Notes:-Customerswithhigher(orlower)computeneedscanacquirebigger(orsmaller)datanodesconfiguredwithCPUandmemorythatfitsthespecificrequirementsoftheirapplications.-Similarly,dependingonthedatarequirements,customerscanaddorremovediskdrivestomatchthespecificneedsoftheirapplications.

13

6.3PhysicalClusterTopology

Figure3:DriveScalelabArchitecturewith1xDSAChassis(4xAdaptersinuse),1xJBOD,1NameNodeand4DataNodes

6.4ClusterManagement

ThissectiondetailsthestepsforsettingupaDriveScaleenabledHadoopclusterusingAmbariServer.

6.4.1SettingupDriveScaleclusterBeforeinstallingAmbariServerorusinganexistinginstallofAmbariServer,youmustcompletethefollowingtasksforsettinguptheDriveScalesolution:

1. RackandinstalltheDriveScaleAdapterchassisandcontrollers(DSAs)usingthedocumentationprovidedbyDriveScale.

2. RackandinstalltheJBODusingthedocumentationprovidedbythevendor.3. Rackandinstalltheserversusingthedocumentationprovidedbythevendor.4. CreateaRAID1configurationfortheinternalHDDontheserverandinstalltheOS

onalltheotherservers.5. InstallandconfigureDriveScaleManagementServer(DMS)eitherasaVMorona

standaloneserver.6. SetupDSAconfigurationfromtheDMS.

14

7. InstallandconfigureDriveScaleagentsonthemasteranddatanodes.8. Createmaster/datanodeandclustertemplatewithrequireddrivesusingDMS.9. CreatetheclusterfromthetemplateusingDMS.10. EnsurethatDriveScaleclusterisupandrunningbeforeproceedingahead.

Figure4:PhysicalcomponentsoverviewfromDMSUI

Figure5:LogicalClusterstatusfromDMSUI

15

Figure6:LogicalclusterdetailsoverviewfromDMSUI

Figure7:LogicalclusterserverdetailsfromDMSUI

16

6.4.2SettingupHortonworksDataPlatformcluster1. Afterthesuccessfulcompletionofthestepsabove,installAmbariServerusingthe

HDPgettingreadytoinstallandinstallationguide.2. Thefollowingservicesweresetupforthisreferencearchitecture.

Figure8:InstalledHadoopservicesdetailsfromAmbariUIDashboard

3. Ensurethatthenameanddatanodesareupandrunningwiththerightassignedrolesandstorage.

Figure9:HostsandrolesoverviewfromAmbariUIHostssection

17

6.5DiskandFilesystemLayout

Node/Role DiskandFilesystemLayout Description

Management/Master Ext4 1/2/3TBdrivesaremountedfromtheJBOD’s

YARNNodeManagernodes

Ext4 1/2/3TBdrivesaremountedfromtheJBOD’s

6.6OSSupportability/CompatibilityMatrix

DMS ServerNodes

CentOS/RHEL6.x X X

CentOS/RHEL7.x X X

Ubuntu14.04 X X

8. RackScalability

Customerscanscalebeyondonerackinastraightforwardmannertoexpandtheircomputeandstorageresourcesdependingasapplicationneedsgrow.Customerscanchangeormaintainthecompute-to-storageratioforthenewracksoranexistingrack.ForeverynewJBODaddition,anewDriveScaleAdapterwithfourcontrollersmustbeaddedaswell.Sincedrivesareassignedfromwithintheracktoserversintherack,scalingisachievedbysimplyaddingmorerackswithServers,DriveScaleAdapters,SwitchesandJBODs.

18

Figure10:HostsandrolesoverviewfromAmbariUIHostssection

9. References

1. HDPInstallationprerequisiteshttps://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ch_getting_ready_chapter.html

2. AutomatedInstallwithAmbarihttps://docs.hortonworks.com/HDPDocuments/Ambari-2.1.1.0/bk_Installing_HDP_AMB/content/index.html

3. DriveScaledocumentationforrackingandinstallationwhichareprovidedbyDriveScale

4. YARNDefinitionhttp://searchdatamanagement.techtarget.com

19

10. BillofMaterials

ServerComponents Quantity

IntelE5-E5-2665,2.40GHz,16CCPU 2

16GBDIMMS 16

Intel10GbESFP+NIC 1

JBODComponents Quantity

QUANTAJB4602JBOD 1

IOControllers 2

SeagateHDD 60

Switch Quantity

CISCONEXUS554810GbEswitch 2

D-LinkDGs-1518-281GbEswitch 1

DriveScaleComponents Quantity

DriveScaleAdapterChassis 1

DriveScaleAdapter 4

Software Version

CentOS 6.7

DriveScaleAdapter 1.2.0.1

HDP 2.3

20

11. Conclusion

TheDriveScale-HDPsolutionreferencearchitectureguideisdesignedtoprovideanoverviewofthecombinedsolutionsandthekeycomponentsemployed.ThereferencearchitecturealsooutlinestheadvantagesofcomputeandstoragedisaggregationwithDrivescale-HDPsolution.AboutDriveScaleDriveScaleisleadingthechargeinbringinghyperscalecomputingcapabilitiestomainstreamenterprises.Itscomposabledatacenterarchitecturetransformsrigiddatacentersintoflexibleandresponsivescale-outdeployments.UsingDriveScale,datacenteradministratorscandeployindependentpoolsofcommoditycomputeandstorageresources,automaticallydiscoveravailableassets,andcombineandrecombinetheseresourcesasneeded.Thesolutionisprovidedviaasetofon-premisesandSaaStoolsthatcoordinatebetweenmultiplelevelsofinfrastructure.WithDriveScale,companiescanmoreeasilysupportHadoopdeploymentsofanysizeaswellasothermodernapplicationworkloads.DriveScaleisfoundedbyateamwithdeeprootsinITarchitectureandthathasbuiltenterprise-classsystemssuchasCiscoUCSandSunUltraSparc.BasedinSunnyvale,California,thecompanywasfoundedin2013.InvestorsincludePelionVenturePartners,NautilusVenturePartnersandIngrasys,awhollyownedsubsidiaryofFoxconn.Formoreinformation,visitwww.drivescale.comorfollowusonTwitterat@DriveScale_Inc.

AboutHortonworksHortonworksisaleadinginnovatorintheindustry,creating,distributingandsupportingenterprise-readyopendataplatformsandmoderndataapplications.Ourmissionistomanagetheworld’sdata.Wehaveasingle-mindedfocusondrivinginnovationinopensourcecommunitiessuchasApacheHadoop,NiFi,andSpark.Wealongwithour1600+partnersprovidetheexpertise,trainingandservicesthatallowourcustomerstounlocktransformationalvaluefortheirorganizationsacrossanylineofbusiness.Ourconnecteddataplatformspowersmoderndataapplicationsthatdeliveractionableintelligencefromalldata:data-in-motionanddata-at-rest.Visitusathortonworks.com.WearePoweringtheFutureofData™.