Characterizing and Contrasting Container Orchestrators

CharacterizingandContrastingContainer

Orchestrators

LeeCalcote

http://calcotestudios.com/ccco

LeeCalcoteclouds,containers,infrastructure,applications

andtheirmanagement

linkedin.com/in/leecalcote

@lcalcote

blog.gingergeek.com

lee@calcotestudios.com

[kuhn-tey-ner][awr-kuh-streyt-or]

Definition:

FleetNomadSwarm

KubernetesMesos+Marathon

(Staytunedforupdatestopresentation)

CategoricallySpeaking

CategoricallySpeakingGenesis&Purpose

CategoricallySpeakingGenesis&PurposeSupport&Momentum

CategoricallySpeakingGenesis&PurposeSupport&MomentumHost&ServiceDiscovery

CategoricallySpeaking

Scheduling

Genesis&PurposeSupport&MomentumHost&ServiceDiscovery

CategoricallySpeaking

Scheduling

Genesis&PurposeSupport&MomentumHost&ServiceDiscovery

Modularity&Extensibility

CategoricallySpeaking

Scheduling

Genesis&PurposeSupport&MomentumHost&ServiceDiscovery

Modularity&ExtensibilityUpdates&Maintenance

CategoricallySpeaking

Scheduling

Genesis&PurposeSupport&MomentumHost&ServiceDiscovery

Modularity&ExtensibilityUpdates&MaintenanceHealthMonitoring

CategoricallySpeaking

Scheduling

Genesis&PurposeSupport&MomentumHost&ServiceDiscovery

Modularity&ExtensibilityUpdates&MaintenanceHealthMonitoringNetworking&Load-Balancing

CategoricallySpeaking

Scheduling

Genesis&PurposeSupport&MomentumHost&ServiceDiscovery

Modularity&ExtensibilityUpdates&MaintenanceHealthMonitoringNetworking&Load-BalancingHighAvailability&Scale

HypervisorManagerElements

ComputeNetworkStorage

ContainerOrchestratorElements

Host(Node)ContainerServiceVolumeApplications

CoreCapabilities

Scheduling

ClusterManagement

HostDiscovery

HostHealthMonitoring

OrchestratorUpdatesandHost

Maintenance

ServiceDiscovery

NetworkingandLoad-Balancing

AdditionalKeyCapabilitiesApplicationHealthMonitoring

ApplicationDeployments

ApplicationPerformanceMonitoring

DockerSwarm

Genesis&PurposeSwarmissimpleandeasytosetup.Animperativesystem,Swarmisresponsiblefortheclusteringandschedulingaspectsoforchestration.Swarm’sarchitectureisnotcomplexasthoseofKubernetesandMesos

hasManager(s)andAgent(s)

WritteninGolang,Swarmislightweight,modular,portable,andextensible

Support&Momentum~3,000commitsby12coremaintainers(130contributorsinall),andforDockerCompose,thenumbersareapproximatelythesameSwarmleveragesthesameAPIforDockerenginewhichover23,000commitsand1,350contributors

~250DockermeetupsworldwideAnnouncedasproduction-ready5monthsago(Nov2015)

image:digitaltrends

Host&ServiceDiscoveryHostDiscovery

usedintheformationofclustersbytheManagertodiscoverforNodes(hosts).

ServiceDiscovery

SwarmhasaconceptofservicesthroughnetworkaliasesandroundrobinDNS

image:iStock

SchedulingSwarm’sschedulerispluggableSwarmschedulingisacombinationofstrategiesandfilters/constraint:Strategies

RandomBinpackSpread*

Filterscontainerconstraints(affinity,dependency,port)aredefinedas

environmentvariablesinthespecificationfile

nodeconstraints(health,constraint)mustbespecifiedwhen

startingthedockerdaemonanddefinewhichnodesacontainer

maybescheduledon.image:pickywallpapers

Modularity&ExtensibilityAbilitytoremovebatteriesisastrengthforSwarm:

PluggableschedulerPluggablenetworkdriverPluggabledistributedK/VstoreDockercontainerengineruntime-only

image: AlanChia

Updates&Maintenance

Nodes

Manualswarmmanagerandagentupdates

Applications

Nofacilitiesforupdatingapplication

image:123RF

HealthMonitoringNodes

Swarmmonitorstheavailabilityandresourceusageofnodeswithinthecluster

Applications

NofacilitiesformonitoringapplicationsUsethird-partysoftware

Networking&Load-Balancing

DockerSwarmisfullycompatiblewithDocker’snetworkingDockermulti-hostnetworking(requiresaK/Vstore)providesforuser-definedoverlaynetworksthataremicro-segmentable

usesagossipprotocolforquickconvergenceofneighbortable

facilitatescontainernameresolutionviaembeddedDNSserver

(previouslyviaetc/hosts)

YoumaybringyourownnetworkdriverLoad-balancingisnewin1.11.0

-nodocumentationavailable

HighAvailability&ScaleManagersmaybedeployedinahighly-availableconfiguration(outofexperimentalin1.11.0)

Active/Standby-anactivemanagerandreplicasconfiguration

Reschedulinguponnodefailure(experimentalin1.11.0)andsimple.Rebalancingisnotavailable

Rescheduledcontainersloseaccesstovolumesmountedontheformerhost.

usevolumepluginslikeFlockertoavoidthis

Doesnotsupportmultiplefailureisolationregionsorfederation(although,withcaveats, ).

thisispossible

Scalingswarmto1,000AWSnodesand50,000containers

Suitablefororchestratingacombinationofinfrastructurecontainers

Lessbuilt-infunctionalityforapplicationcontainers

Swarmisayoungprojectandlacksadvancedfeatures.

HighAvailabilityoutofexperimentalasoflatestrelease(1.11.0)

Reschedulingisexperimentalin1.11.0(noRebalancing,yet)

Load-balancingin1.11.0?Swarmneedstobeusedwiththird-partysoftware.

OnlyschedulesDockercontainers,notcontainersusingotherspecifications.

Whiledependencyandaffinityfiltersareavailable,Swarmdoesnotprovide

theabilitytoenforceschedulingoftwocontainersontothesamehostornot

atall.

Currentfiltersdonotfacilitatesidecarpattern.No“pod”concept.

Swarmworks.Swarmissimpleandeasyto

deploy.

IfyoualreadyknowDockerCLI,usingSwarm

isstraight-forward

facilitatesearlierstagesofadoptionby

organizationsviewingcontainersasfasterVMs

Lessbuilt-infunctionalityforapplications

Swarmiseasytoextend,ifcanalready

knowDockerAPIs,youcancustomizeSwarm

Highlymodular:

Pluggablescheduler

PluggableK/Vstoreforbothnodeandmulti-

hostnetworking

Kubernetes

Genesis&Purposeanopinionatedframeworkforbuildingdistributedsystems

orasitstaglinestates"anopensourcesystemforautomating

deployment,scaling,andoperationsofapplications."

WritteninGolang,Kubernetesislightweight,modular,portable,andextensibleconsideredathirdgenerationcontainerorchestratorledbyGoogle,RedHatandothers.

bakesinload-balancing,scale,volumes,deploymentsandsecret

managementamongotherfeatures.

Declaratively,opinionatedwithmanykeyfeaturesincluded

Support&MomentumKubernetesisyoung(abouttwoyearsold)

Announcedasproduction-ready10monthsago(July2015)

Projectcurrentlyhasover1,000commitspermonth(~23,000total)

madebyabout100(650total)Kubernauts(Kubernetesenthusiasts).~5,000commitsmadeinthelatestrelease-1.2.

UnderthegovernanceoftheCloudNativeComputingFoundationRobustsetofdocumentationand~90meetups

Host&ServiceDiscoveryHostDiscovery

bydefault,thenodeagent(kubelet)isconfiguredtoregisteritselfwiththemaster(APIserver)

automatingthejoiningofnewhoststothecluster.

ServiceDiscoveryTwoprimarymodesoffindingaService

environmentvariables

environmentvariablesareusedasasimplewayofprovidingcompatibilitywithDockerlinks-style

networking

DNS

whenenabledisdeployedasaclusteradd-on image:iStock

Scheduling

Bydefault,schedulingishandledbykube-scheduler.PluggableSelectioncriteriausedbykube-schedulertoidentifythebest-fitnodeisdefinedbypolicy:

Predicates(noderesourcesandcharacteristics):

PodFitPorts,PodFitsResources,NoDiskConflict

,MatchNodeSelector,HostName,ServiceAffinit,LabelsPresence

Priorities(weightedstrategiesusedtoidentify“bestfit”node):

LeastRequestedPriority,BalancedResourceAllocation,ServiceSpreadingPriority,

EqualPriority

Modularity&Extensibility

OneofKubernetesstrengthsitspluggablearchitectureChoiceof:

databaseforservicediscoveryornetworkdrivercontainerruntime

usersmaychoosetorunDockerwithRocketcontainers

Clusteradd-onsoptionalsystemcomponentsthatimplementaclusterfeature(e.g.DNS,logging,etc.)shippedwiththeKubernetesbinariesandareconsideredaninherentpartoftheKubernetesclusters

Updates&MaintenanceApplications

Deploymentobjectsautomatedeployingandrollingupdatingapplications.

KubernetesComponents

UpgradingtheKubernetescomponentsandhostsisdoneviashellscriptHostmaintenance-markthenodeasunschedulable.

existingpodsarenotvacatedfromthenodepreventsnewpodsfrombeingscheduledonthenode

image:123RF

HealthMonitoringNodes

Failures-activelymonitorsthehealthofnodeswithinthecluster

viaNodeController

Resources-usagemonitoringleveragesacombinationofopensourcecomponents:

cAdvisor,Heapster,InfluxDB,Grafana

Applications

threetypesofuser-definedapplicationhealth-checksandusestheKubeletagentasthethehealthcheckmonitor

HTTPHealthChecks,ContainerExec,TCPSocket

Cluster-levelLogging

collectlogswhichpersistbeyondthelifetimeofthepod’scontainerimagesorthelifetimeofthepodorevencluster

standardoutputandstandarderroroutputofeachcontainercanbeingestedusinga

Networking&Load-Balancing

…enterthePod

atomicunitofschedulingflatnetworkingwitheachpodreceivinganIPaddressnoNATrequired,portconflictslocalizedintra-podcommunicationvialocalhost

Load-Balancing

Servicesprovideinherentload-balancingviakube-proxy:

runsoneachnodeofaKubernetescluster

reflectsservicesasdefinedintheKubernetesAPI

supportssimpleTCP/UDPforwardingandround-robinandDocker-links-

basedserviceIP:PORTmapping.

HighAvailability&Scale

Eachmastercomponentmaybedeployedinahighly-availableconfiguration.

Active/Standbyconfiguration

Intermsofscale,v1.2bringssupportfor1,000nodeclustersandasteptowardfully-federatedclusters(Ubernetes)Application-levelauto-scalingissupportedwithinKubernetesviaReplicationControllers

Onlyrunscloud-nativeapplications

ForthosefamiliarwithDocker-only,

Kubernetesrequiresunderstandingofnew

concepts

Powerfulframeworkswithmoremovingpiecesbeget

complicatedclusterdeploymentandmanagement.

Lightweightgraphicaluserinterface

Doesnotprovideassophisticatedtechniques

forresourceutilizationasMesos

Kubernetescanscheduledockerorrkt

containers

Inherentlyopinionatedwithfunctionalitybuilt-

in.

nothird-partysoftwareneededtorunservices,load-

balancing,

buildsinmanyapplication-levelconceptsandservices

(secrets,petsets,jobsets,daemonsets,rollingupdates,etc.)

Kubernetesarguablymovingthequickest

Relativelythoroughprojectdocumentation

Multi-master,Robustlogging&metrics

aggregation

Mesos+

Marathon

Genesis&PurposeMesosisadistributedsystemskernel

stitchestogethermanydifferentmachinesintoalogicalcomputer

Mesoshasbeenaroundthelongest(launchedin2009)andisarguablythemoststable,withhighest(proven)scalecurrently

MessiswritteninC++withJava,PythonandC++APIs

FrameworksMarathonisoneofanumberofframeworks(ChronosandAuroraother

examples)thatmayberunontopofMesos

Frameworkshaveaschedulerandexecutor.Schedulersgetresource

offers.Executorsruntasks.

MarathoniswritteninScala

Support&Momentum

MesosCon2015inSeattlehad700attendeesupfrom262attendeesin2014

78contributorsinthelastyear

UnderthegovernanceofApacheFoundationUsedbyTwitter,AirBnb,eBay,Apple,Cisco,Yodle

Host&ServiceDiscovery

Mesos-DNSgeneratesanSRVrecordforeachMesostask

includingMarathonapplicationinstances

MarathonwillensurethatalldynamicallyassignedserviceportsareuniqueMesos-DNSisparticularlyusefulwhen:

appsarelaunchedthroughmultipleframeworks(notjustMarathon)

youareusinganIP-per-containersolutionlike

youuserandomhostportassignmentsinMarathon

ProjectCalico

image:iStock

Scheduling

TwolevelschedulerFirstlevelschedulinghappensatmesosmasterbasedonallocationpolicy,whichdecideswhichframeworkgetresourcesSecondlevelschedulinghappensatFrameworkscheduler,whichdecideswhattaskstoexecute.

Providereservationsandoversubscriptions

Modularity&ExtensibilityFrameworks

multipleavailablemayrunmultipleframeworks

Modules

hasthreetypesofModulesReplacement,Hook,Anonymous

Updates&Maintenance

Nodes-MesoshasmaintenancemodeApplications

Marathoncanbeinstructedtodeploycontainersbasedonthatcomponentusingablue/greenstrategy

whereoldandnewversionsco-existfora

time.

image:123RF

HealthMonitoringNodes

Mastertracksasetofstatisticsandmetricstomonitorresourceusage

CountersandGauges

Applications

supportforhealthchecks(HTTPandTCP)aneventstreamthatcanbeintegratedwithload-balancersorforanalyzingmetrics

Networking&Load-Balancing

Networking

AnIPperContainerNolongersharethenode'sIP

Helpsremoveportconflicts

Enables3rdpartynetworkdrivers

Load-Balancing

MarathonofferstwoTCP/HTTPproxy.Asimpleshellscriptandamorecomplexonecalledmarathon-lbthat

hasmorefeatures.

Pluggable(e.g.Traeficforload-balancing)

HighAvailability&ScaleAstrengthofMesos’sarchitecture

requiresmasterstoformaquorumusingZooKeeper

onlyoneActive(Leader)Marathonmasterat-a-time

ScaleisastrongsuitforMesos.UsedatTwitter,AirBnB...TBDforMarathon

Greatatasynchronousjobs.Highavailabilitybuilt-in.Referredtoasthe“goldenstandard”bySolomonHykes,DockerCTO.

UniversalContainerizer

abstractawayfromDockerEngine,(runc,appc)

Canrunmultipleframeworks,includingKubernetesandSwarm.

Onlyofthecontainerorchestratorsthatsupportsmulti-tenancy

GoodforBigDatahouseandjob-orientedortask-orientedworkloads.

Goodformixedworkloadsandwithdata-localitypolicies

Powerfulandscalable,Battle-tested

Goodformultiplelargethingsyouneedtodo10,000+nodeclustersystem

MarathonUIisyoung,butpromising

Stillneeds3rdpartytools

MarathoninterfacecouldbemoreDockerfriendly

(hardtogetatvolumesandregistry)

MayneedadedicatedinfrastructureITteam

anoverlycomplexsolutionforsmalldeployments

Summary

Ahigh-levelperspectiveofthecontainerorchestratorspectrum.

LeeCalcote

linkedin.com/in/leecalcote

@lcalcote

blog.gingergeek.com

lee@calcotestudios.com

Thankyou.Questions?