Upload
buingoc
View
239
Download
2
Embed Size (px)
Citation preview
TableofContents
MonitoringDocker
Credits
AbouttheAuthor
AbouttheReviewer
www.PacktPub.com
Supportfiles,eBooks,discountoffers,andmore
Whysubscribe?
FreeaccessforPacktaccountholders
Preface
Whatthisbookcovers
Whatyouneedforthisbook
Whothisbookisfor
Conventions
Readerfeedback
Customersupport
Downloadingtheexamplecode
Downloadingthecolorimagesofthisbook
Errata
Piracy
Questions
1.IntroductiontoDockerMonitoring
Pets,Cattle,Chickens,andSnowflakes
Pets
Cattle
Chickens
Snowflakes
Sowhatdoesthisallmean?
Docker
Launchingalocalenvironment
Cloningtheenvironment
Runningavirtualserver
Haltingthevirtualserver
Summary
2.UsingtheBuilt-inTools
Dockerstats
RunningDockerstats
Whatjusthappened?
Whataboutprocesses?
Dockertop
Dockerexec
Summary
3.AdvancedContainerResourceAnalysis
WhatiscAdvisor?
RunningcAdvisorusingacontainer
CompilingcAdvisorfromsource
Collectingmetrics
TheWebinterface
Overview
Processes
CPU
Memory
Network
Filesystem
Viewingcontainerstats
Subcontainers
Driverstatus
Images
Thisisallgreat,what’sthecatch?
Prometheus
LaunchingPrometheus
QueryingPrometheus
Dashboard
Thenextsteps
Alternatives?
Summary
4.ATraditionalApproachtoMonitoringContainers
Zabbix
InstallingZabbix
Usingcontainers
Usingvagrant
Preparingourhostmachine
TheZabbixwebinterface
Dockermetrics
Createcustomgraphs
Comparecontainerstoyourhostmachine
Triggers
Summary
5.QueryingwithSysdig
WhatisSysdig?
InstallingSysdig
UsingSysdig
Thebasics
Capturingdata
Containers
Furtherreading
UsingCsysdig
Summary
6.ExploringThirdPartyOptions
Awordaboutexternallyhostedservices
DeployingDockerinthecloud
WhyuseaSaaSservice?
SysdigCloud
Installingtheagent
Exploringyourcontainers
Summaryandfurtherreading
Datadog
Installingtheagent
Exploringthewebinterface
Summaryandfurtherreading
NewRelic
Installingtheagent
Exploringthewebinterface
Summaryandfurtherreading
Summary
7.CollectingApplicationLogsfromwithintheContainer
Viewingcontainerlogs
ELKStack
Startingthestack
Logspout
Reviewingthelogs
Whataboutproduction?
Lookingatthirdpartyoptions
Summary
8.WhatAretheNextSteps?
Somescenarios
Pets,Cattle,Chickens,andSnowflakes
Pets
Cattle
Chickens
Snowflakes
Scenarioone
Scenariotwo
MonitoringDockerCopyright©2015PacktPublishing
Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.
Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyoftheinformationpresented.However,theinformationcontainedinthisbookissoldwithoutwarranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishing,anditsdealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecauseddirectlyorindirectlybythisbook.
PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.
Firstpublished:December2015
Productionreference:1041215
PublishedbyPacktPublishingLtd.
LiveryPlace
35LiveryStreet
BirminghamB32PB,UK.
ISBN978-1-78588-275-3
www.packtpub.com
CreditsAuthor
RussMcKendrick
Reviewer
MarceloCorreiaPinheiro
CommissioningEditor
VeenaPagare
AcquisitionEditor
RahulNair
ContentDevelopmentEditor
AnishSukumaran
TechnicalEditor
SaurabhMalhotra
CopyEditor
TrishyaHajare
ProjectCoordinator
IzzatContractor
Proofreader
SafisEditing
Indexers
MariammalChettiyar
PriyaSane
ProductionCoordinator
ShantanuN.Zagade
CoverWork
ShantanuN.Zagade
AbouttheAuthorRussMcKendrickisanexperiencedsolutionsarchitectwhohasbeenworkinginITandIT-relatedindustriesforthebetterpartof23years.Duringhiscareer,hehashadvariedresponsibilitiesinanumberofindustries,rangingfromlookingafterentireITinfrastructurestoprovidingfirstline,secondline,andseniorsupportinclientfacing,andinternalteamsforcorporateorganizations.
HeworksalmostexclusivelywithLinux,usingopensourcesystemsandtoolsonvariousplatformsrangingfromdedicatedhardwareandvirtualmachinestopublicclouds.
AbouttheReviewerMarceloCorreiaPinheiroisaBraziliansoftwareengineerfromPortoAlegre.Hestartedtoworkasawebdesignerandprogrammerin2000withASPandPHP,naturallygettingintouchwiththeMicrosoft.NETframeworkandJavarunningrespectivedatabasesofchoiceforwebapplications.Since2003,hehasusedLinuxandUNIX-relatedoperationalsystems,fromSlackwaretoGoboLinux,Archlinux,CentOS,Debian,andtodayOSX,havingsomecontactwithBSDdistributionstoo.HehaslostsomenightscompilingandapplyingpatchestotheLinuxkerneltomakeitsdesktopwork.Sincethebeginning,hehasbeenactingasaproblemsolver,nomatterwhattheprogramminglanguage,database,orplatformis—opensourceenthusiast.
Afterafewyears,hedecidedtoliveinSãoPaulotoworkwithnewertechnologiessuchasNoSQL,cloudcomputing,andRuby,wherehestartedtoconducttechtalkswiththislanguageinLocaweb.HecreatedsometoolstostandardizedevelopmentusingtoolssuchasvagrantandRubygems—someoftheseintheirGitHub—inLocawebtoensurefastapplicationpackagingandreduceddeploymentrollbacks.In2013,hechangedhiscareertobeafull-stackdeveloperfollowingtheDevOpsmovement.Since2012,hehasattended,asaspeaker,someofthebiggestsoftwareconferencesinBrazil—RSonRails,QConSP,TheDeveloper’sConference,andRubyConfBrazil—talkingnotonlyaboutRuby,butalsoaboutsomeofthewell-knownDevOpstoolssuchasTerraform,Packer,Ansible,andDocker.Today,heworksasaDevOpsconsultantintheircompany.
Inhisfreetime,helovesplayingtheguitar,havingsomefunwithcats,traveling,anddrinkingbeer.Hecanbefoundonhisblog(http://salizzar.net),Twitter(https://twitter.com/salizzar),GitHub(https://github.com/salizzar)andLinkedin(https://www.linkedin.com/in/salizzar).
HehasworkedasareviewerforVagrantVirtualDevelopmentEnvironmentCookbook,aPacktPublishingbookwithusefulrecipesusingvagrantwithconfigurationmanagementtoolssuchasPuppet,Chef,Ansible,andSaltStack.
Iwanttothankallmyfriends,whobelievedinmypotentialsincethebeginningandwhostillfollowmedespitethedistance.Iwouldalsoliketothankmymentors,GleiconMoraes,RobertoGaiser,andRodrigoCampos,whogavemetheincentiveandtipstobeabettersoftwareengineerandperson.
Supportfiles,eBooks,discountoffers,andmoreForsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.
DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusat<[email protected]>formoredetails.
Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signupforarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooksandeBooks.
https://www2.packtpub.com/books/subscription/packtlib
DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigitalbooklibrary.Here,youcansearch,access,andreadPackt’sentirelibraryofbooks.
Whysubscribe?FullysearchableacrosseverybookpublishedbyPacktCopyandpaste,print,andbookmarkcontentOndemandandaccessibleviaawebbrowser
FreeaccessforPacktaccountholdersIfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccessPacktLibtodayandview9entirelyfreebooks.Simplyuseyourlogincredentialsforimmediateaccess.
PrefaceWiththeincreaseintheadoptionofDockercontainers,theneedtomonitorwhichcontainersarerunning,whatresourcestheyareconsuming,andhowitaffectstheoverallperformanceofthesystem,hasbecomeatime-relatedneed.MonitoringDockerwillteachyouhowmonitoringcontainersandkeepingakeeneyeontheworkingofapplicationshelptoimprovetheoverallperformanceoftheapplicationsthatrunonDocker.
ThisbookwillcovermonitoringcontainersusingDocker’snativemonitoringfunctions,variousplugins,andalsothird-partytoolsthathelpinmonitoring.Thebookwillfirstcoverhowtoobtaindetailedstatsfortheactivecontainers,resourcesconsumed,andcontainerbehavior.Thisbookwillalsoshowthereadershowtousethesestatstoimprovetheoverallperformanceofthesystem.
WhatthisbookcoversChapter1,IntroductiontoDockerMonitoring,discusseshowdifferentitistomonitorcontainerscomparedtomoretraditionalserverssuchasvirtualmachines,baremetalmachines,andcloudinstances(PetsversusCattleandChickensversusSnowflakes).Thischapteralsodetailstheoperatingsystemscoveredintheexampleslaterinthisbookandalsogivesalittleinformationonhowtogetalocaltestenvironmentupandrunningusingvagrant,sothatinstallationinstructionsandpracticalexamplescanbeeasilyfollowed.
Chapter2,UsingtheBuilt-inTools,helpsyoulearnaboutthebasicmetricsyoucangetoutofthevanillaDockerinstallationandhowyoucanusethem.Also,wewillunderstandhowtogetreal-timestatisticsonourrunningcontainers,howtousecommandsthatarefamiliartous,andhowtogetinformationontheprocessesthatarelaunchedaspartofeachcontainer.
Chapter3,AdvancedContainerResourceAnalysis,introducescAdvisorfromGoogle,whichaddsalotmoreprecisiontothebasictoolsprovidedbyDocker.YouwillalsolearnhowtoinstallcAdvisorandstartcollectingmetrics.
Chapter4,ATraditionalApproachtoMonitoringContainers,looksatatraditionaltoolformonitoringservices.Bytheendofthischapter,youshouldknowyourwayaroundZabbixandthevariouswaysyoucanmonitoryourcontainers.
Chapter5,QueryingwithSysdig,describesSysdigas“anopensource,system-levelexplorationtooltocapturesystemstateandactivityfromarunningLinuxinstance,thensave,filter,andanalyzeit.”Inthischapter,youwilllearnhowtouseSysdigtobothviewyourcontainers’performancemetricsinrealtimeandalsorecordsessionstoquerylater.
Chapter6,ExploringThirdPartyOptions,walksyouthroughafewoftheSoftwareasaService(SaaS)optionsthatareavailable,whyyouwouldusethem,andhowtoinstalltheirclientsonthehostserver.
Chapter7,CollectingApplicationLogsfromwithintheContainer,looksathowwecangetthecontentofthelogfilesfortheapplicationsrunningwithinourcontainerstoacentrallocationsothattheyareavailableevenifyouhavetodestroyandreplaceacontainer.
Chapter8,WhatAretheNextSteps?,looksatthenextstepsyoucantakeinmonitoringyourcontainersbytalkingaboutthebenefitsofaddingalertingtoyourmonitoring.Also,wewillcoversomedifferentscenariosandlookatwhichtypeofmonitoringisappropriateforeachofthem.
WhatyouneedforthisbookToensuretheexperienceisasconsistentaspossible,wewillbeinstallingvagrantandVirtualBoxtorunthevirtualmachinethatwillactasahosttorunourcontainers.VagrantisavailableforLinux,OSX,andWindows;fordetailsonhowtoinstallthis,seethevagrantwebsiteathttps://www.vagrantup.com/.ThedetailsofhowtodownloadandinstallVirtualBoxcanbefoundathttps://www.virtualbox.org/;again,VirtualBoxcanbeinstalledonLinux,OSX,andWindows.
WhothisbookisforThisbookisforDevOpsengineersandsystemadministratorswhowanttomanageDockercontainers,bettermanagethesecontainersusingexperttechniquesandmethods,andbettermaintainapplicationsbuiltonDocker.
ConventionsInthisbook,youwillfindanumberoftextstylesthatdistinguishbetweendifferentkindsofinformation.Herearesomeexamplesofthesestylesandanexplanationoftheirmeaning.
Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:“Wecanincludeothercontextsthroughtheuseoftheincludedirective.”
Ablockofcodeissetasfollows:
{
"fields":{
"@timestamp":[
1444567706641
]
},
"sort":[
1444567706641
]
}
Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevantlinesoritemsaresetinbold:
{
"fields":{
"@timestamp":[
1444567706641
]
},
"sort":[
1444567706641
]
}
Anycommand-lineinputoroutputiswrittenasfollows:
cd~/Documents/Projects/monitoring-docker/vagrant-ubuntu
vagrantup
Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,forexample,inmenusordialogboxes,appearinthetextlikethis:“ClickingtheNextbuttonmovesyoutothenextscreen.”
NoteWarningsorimportantnotesappearinaboxlikethis.
TipTipsandtricksappearlikethis.
ReaderfeedbackFeedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthisbook—whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsusdeveloptitlesthatyouwillreallygetthemostoutof.
Tosendusgeneralfeedback,simplye-mail<[email protected]>,andmentionthebook’stitleinthesubjectofyourmessage.
Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingorcontributingtoabook,seeourauthorguideatwww.packtpub.com/authors.
CustomersupportNowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelpyoutogetthemostfromyourpurchase.
DownloadingtheexamplecodeYoucandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.
DownloadingthecolorimagesofthisbookWealsoprovideyouwithaPDFfilethathascolorimagesofthescreenshots/diagramsusedinthisbook.Thecolorimageswillhelpyoubetterunderstandthechangesintheoutput.Youcandownloadthisfilefrom:http://www.packtpub.com/sites/default/files/downloads/Monitoring_Docker_ColorImages.pdf
ErrataAlthoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdohappen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthecode—wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveotherreadersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufindanyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthedetailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedandtheerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataundertheErratasectionofthattitle.
Toviewthepreviouslysubmittederrata,gotohttps://www.packtpub.com/books/content/supportandenterthenameofthebookinthesearchfield.TherequiredinformationwillappearundertheErratasection.
PiracyPiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.AtPackt,wetaketheprotectionofourcopyrightandlicensesveryseriously.IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswiththelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.
Pleasecontactusat<[email protected]>withalinktothesuspectedpiratedmaterial.
Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluablecontent.
QuestionsIfyouhaveaproblemwithanyaspectofthisbook,youcancontactusat<[email protected]>,andwewilldoourbesttoaddresstheproblem.
Chapter1.IntroductiontoDockerMonitoringDockerhasbeenarecentbutveryimportantadditiontoaSysAdminstoolbox.
Dockerdescribesitselfasanopenplatformforbuilding,shipping,andrunningdistributedapplications.Thismeansthatdeveloperscanbundletheircodeandpassittotheiroperationsteam.Fromhere,theycandeploysafeintheknowledgethatitwillbedonesoinawaythatintroducesconsistencywiththeenvironmentinwhichthecodeisrunning.
Whenthisprocessisfollowed,itshouldmaketheage-olddevelopersversusoperationsargumentof“itworkedonmylocaldevelopmentserver”—athingofthepast.Sincebeforeits“productionready”1.0releasebackinJune2014,therehadbeenover10,000Dockerizedapplicationsavailable.Bytheendof2014,thatnumberhadrisentoover71,000.YoucanseehowDockergrewin2014bylookingattheinfographicthatwaspublishedbyDockerinearly2015,whichcanbefoundathttps://blog.docker.com/2015/01/docker-project-2014-a-whirlwind-year-in-review/.
Whilethedebateisstillragingabouthowproductionreadythetechnologyis,Dockerhasgainedanimpressivelistoftechnologypartners,includingRedHat,Canonical,HP,andevenMicrosoft.
CompaniessuchasGoogle,Spotify,Soundcloud,andCenturyLink,haveallopensourcedtoolsthatsupportDockerinsomeway,shape,orformandtherehasalsobeennumerousindependentdeveloperswhohavereleasedappsthatprovideadditionalfunctionalitytothecoreDockerproductset.Also,allthecompanieshavesprunguparoundtheDockerecosystem.
Thisbookassumesthatyouhavehadsomelevelofexperiencebuilding,running,andmanagingDockercontainers,andthatyouwouldnowliketostarttometricsfromyourrunningapplicationstofurthertunethem,orthatyouwouldliketoknowwhenaproblemoccurswithacontainersothatyoucandebuganyongoingissues.
IfyouhaveneverusedDockerbefore,youmaywanttotryoneoftheexcellentbooksthatserveandintroduceyoutoallthethingsthatDockerprovides,bookssuchasLearningDocker,PacktPublishing,orDocker’sownintroductiontocontainers,whichcanbefoundattheirdocumentationpages,asfollows:
LearningDocker:https://www.packtpub.com/virtualization-and-cloud/learning-dockerOfficialDockerdocs:https://docs.docker.com/
Now,wehaveabroughtourselvesuptospeedwithwhatDockeris;therestofthischapterwillcoverthefollowingtopics:
Howdifferentisittomonitorcontainersversusmoretraditionalserverssuchasvirtualmachines,baremetalmachine,andcloudinstances(Pets,Cattle,Chickens,andSnowflakes).
WhataretheminimumversionsofDockeryoushouldberunning?HowtofollowinstructionsonbringingupanenvironmentlocallyusingVagrantinordertofollowthepracticalexercisesinthisbook
Pets,Cattle,Chickens,andSnowflakesBeforewestartdiscussingthevariouswaysinwhichyoucanmonitoryourcontainers,weshouldgetanunderstandingofwhataSysAdminsworldlookslikethesedaysandalsowherecontainersfitintoit.
AtypicalSysAdminwillprobablybelookingafteranestateofserversthatarehostedineitheranon-siteorthird-partydatacenter,somemayevenmanageinstanceshostedinapubliccloudsuchasAmazonWebServicesorMicrosoftAzure,andsomeSysAdminsmayjugglealltheirserverestatesacrossmultiplehostingenvironments.
Eachofthesedifferentenvironmentshasitsownwayofdoingthings,aswellasperformingbestpractices.BackinFebruary2012,RandyBiasgaveatalkatCloudscalingthatdiscussedarchitecturesforopenandscalableclouds.Towardstheendoftheslidedeck,RandyintroducedtheconceptofPetsversusCattle(whichheattributestoBillBaker,whowasthenanengineeratMicrosoft).
Youcanviewtheoriginalslidedeckathttp://www.slideshare.net/randybias/architectures-for-open-and-scalable-clouds.
PetsversusCattleisnowwidelyacceptedasagoodanalogytodescribemodernhostingpractices.
PetsPetsareakintotraditionalphysicalserversorvirtualmachines,asfollows:
Eachpethasaname;forexample,myserver.domain.com.Whenthey’renotwell,youtakethemtothevettohelpthemgetbetter.YouemploySysAdminstolookafterthem.Youpaycloseattentiontothem,sometimesforyears.Youtakebackups,patchthem,andensurethattheyarefullydocumented.
CattleCattle,ontheotherhand,representmoremoderncloudcomputinginstances,asfollows:
You’vegottoomanytoname,soyougivethemnumbers;forexample,theURLcouldlooksomethinglikeip123123123123.eu.public-cloud.com.Whentheygetsick,youshootthemandifyourherdrequiresit,youreplaceanythingyou’vekilled:Aservercrashesorshowssignsthatitishavingproblems,youterminateitandyourconfigurationautomaticallyreplacesitwithanexactreplica.Youputtheminafieldandwatchthemfromfarandyoudon’texpectthemtolivelong.Ratherthanmonitoringtheindividualinstances,youmonitorthecluster.Whenmoreresourcesareneeded,youaddmoreinstancesandoncetheresourceisnolongerrequired,youterminatetheinstancestogetyoubacktoyourbaseconfiguration.
ChickensNextupisatermthatisagoodwayofdescribinghowcontainersfitintothePetsversusCattleworld;inablogposttitle“CloudComputing:Pets,Cattleand…Chickens?”onActiveState,BernardGoldendescribescontainersasChickens:
They’remoreefficientthancattlewhenitcomestoresourceuse.Acontainercanbootinsecondswhereainstanceorservercantakeminutes;italsouseslessCPUpowerthanatypicalvirtualmachineorcloudinstance.Therearemanymorechickensthancattle.Youcanquitedenselypackcontainersontoyourinstancesorservers.Chickenstendtohaveashorterlifespanthancattleandpets.Containerslendthemselvestorunningmicros-services;thesecontainersmayonlybeactiveforafewminutes.
Theoriginalblogpostcanbefoundathttp://www.activestate.com/blog/2015/02/cloud-computing-pets-cattle-and-chickens.
SnowflakesThefinaltermisnotanimal-relatedanditdescribesatypeofserverthatyoudefiantlydon’twanttohaveinyourserverestate,aSnowflake.ThistermwaspennedbyMartinFowlerinablogposttitled“SnowflakeServer”.Snowflakesisatermappliedto“legacy”or“inherited”servers:
Snowflakesaredelicateandaretreatedwithkidgloves.Typically,theserverhasbeeninthedatacentersinceyoustarted.Nooneknowswhooriginallyconfigureditandthereisnodocumentationofit;allyouknowisthatitisimportant.Eachoneisuniqueandisimpossibletoexactlyreproduce.EventhemosthardenedSysAdminfearstorebootthemachineincaseitdoesn’tbootafterwards,asitisrunningend-of-lifesoftwarethatcannoteasilybereinstalled.
Martin’spostcanbefoundathttp://martinfowler.com/bliki/SnowflakeServer.html.
Sowhatdoesthisallmean?Dependingonyourrequirementsandtheapplicationyouwanttodeploy,yourcontainerscanbelaunchedontoeitherpetorcattlestyleservers.Youcanalsocreateaclutchofchickensandhaveyourcontainersrunmicro-services.
Also,intheory,youcanreplaceyourfearedsnowflakeserverswithacontainer-basedapplicationthatmeetsalltheend-of-lifesoftwarerequirementswhileremainingdeployableonamodernsupportableplatform.
Eachofthedifferentstylesofserverhasdifferentmonitoringrequirements,inthefinalchapterwewilllookatPets,Cattle,Chickens,andSnowflakesagainanddiscussthetoolswehavecoveredinthecomingchapters.Wewillalsocoverbestpracticesyoushouldtakeintoconsiderationwhenplanningyourmonitoring.
DockerWhileDockerhititsversion1.0milestoneoverayearago,itisstillinit’sinfancy;witheachnewreleasecomesnewfeatures,bugfixes,andevensupportforsomeearlyfunctionalitythatisbeingdepreciated.
Dockeritselfisnowacollectionofseveralsmallerprojects;theseincludethefollowing:
DockerEngineDockerMachineDockerComposeDockerSwarmDockerHubDockerRegistryKitmatic
Inthisbook,wewillbeusingDockerEngine,DockerCompose,andtheDockerHub.
DockerEngineisthecorecomponentoftheDockerprojectanditprovidesthemainbulkoftheDockerfunctionality.WheneverDockerorthedockercommandismentionedinthisbook,IwillbereferringtoDockerEngine.
ThebookassumesyouhaveDockerEngineversion1.71orlaterinstalled;olderversionsofDockerEnginemaynotcontainthenecessaryfunctionalityrequiredtorunthecommandsandsoftwarecoveredintheupcomingchapters.
DockerComposestarteditslifeasathird-partyorchestrationtoolcalledFigbeforebeingpurchasedbyDockerin2014.Itisdescribedasawayofdefiningamulti-containerapplicationusingYAML(http://yaml.org).Simplyput,thismeansthatyouquicklydeploycomplexapplicationsusingasinglecommandthatcallsahumanreadableconfigurationfile.
WeassumethatyouhaveDockerCompose1.3.3orlaterinstalled;thedocker-compose.ymlfilesmentionedinthisbookhavebeenwrittenwiththisversioninmind.
Finally,themajorityoftheimageswewillbedeployingduringthisbookwillbesourcedfromtheDockerHub(https://hub.docker.com/),whichnotonlyhousesapublicregistrycontainingover40,000publicimagesbutalso100officialimages.ThefollowingscreenshotshowstheofficialrepositorieslistingontheDockerHubwebsite:
LaunchingalocalenvironmentWhereverpossible,Iwilltrytoensurethatthepracticalexercisesinthisbookwillbeabletoberunonalocalmachinesuchasyourdesktoporlaptop.Forthepurposesofthisbook,IwillassumethatyourlocalmachineisrunningeitherarecentversionOSXoranup-to-dateLinuxdistributionandhasahighenoughspecificationtorunthesoftwarementionedinthischapter.
ThetwotoolswewillbeusingtolaunchourDockerinstanceswillalsorunonWindows;therefore,itshouldbepossibletofollowtheinstructionswithinthis,althoughyoumayhavetorefertheusageguidesforanychangestothesyntax.
DuetothewayinwhichDockerisarchitected,alotofthecontentofthisbookwillhaveyourunningcommandsandinteractingwiththecommandlineonthevirtualserverthatisactingasthehostmachine,ratherthanthecontainersthemselves.Becauseofthis,wewillnotbeusingeitherDockerMachineorKitematic.
BothofthesearetoolsprovidedbyDockertoquicklybootstrapaDocker-enabledvirtualserveronyourlocalmachine,asunfortunatelythehostmachinesdeployedbythesetoolscontainastrippeddownoperatingsystemthatisoptimizedforrunningDockerwiththesmallestfootprintaspossible.
Aswewillbeinstallingadditionalpackagesonthehostmachines,astrippeddown“Dockeronly”operatingsystemmaynothavethecomponentsavailabletomeettheprerequisitesofthesoftwarethatwewillberunninginthelaterchapters;therefore,toensurethattherearenoproblemsfurtheron,weberunningafulloperatingsystem.
Personally,IpreferaRPM-basedoperatingsystemsuchasRedHatEnterpriseLinux,Fedora,orCentOS,asIhavebeenusingthemprettymuchsincethedayIfirstloggedintoaLinuxserver.
However,asalotofreaderswillbefamiliarwiththeDebian-basedUbuntu,Iwillbeprovidingpracticalexamplesforbothoperatingsystems.
Toensuretheexperienceisasconsistentaspossible,wewillbeinstallingVagrantandVirtualBoxtorunthevirtualmachinethatwillactasahosttorunourcontainers.
Vagrant,writtenbyMitchellHashimoto,isacommandlinetoolforcreatingandconfiguringreproducibleandportablevirtualmachineenvironments.TherehavebeennumerousblogpostsandarticlesthatactuallypitchDockeragainstVagrant;however,inourcase,thetwotechnologiesworkquitewelltogetherinprovidingarepeatableandconsistentenvironment.
VagrantisavailableforLinux,OSX,andWindows.Fordetailsonhowtoinstall,gototheVagrantwebsiteathttps://www.vagrantup.com/.
VirtualBoxisagreatallroundopensourcevirtualizationplatformoriginallydevelopedbySunandnowmaintainedbyOracle.Itallowsyoutorunboth32-bitand64-bitguestoperatingsystemsonyourlocalmachine.DetailsonhowtodownloadandinstallVirtualBoxcanbefoundathttps://www.virtualbox.org/;again,VirtualBoxcanbeinstalled
CloningtheenvironmentThesourcefortheenvironmentalongwiththepracticalexamplescanbefoundonGitHubintheMonitoringDockerrepositoryathttps://github.com/russmckendrick/monitoring-docker.
Toclonetherepositoryonaterminalonyourlocalmachine,runthefollowingcommands(replacingthefilepathasneeded):
mkdir~/Documents/Projects
cd~/Documents/Projects/
gitclonehttps://github.com/russmckendrick/monitoring-docker.git
Oncecloned,youshouldseeadirectorycalledmonitoring-dockerandthenenterthatdirectory,asfollows:
cd~/Documents/Projects/monitoring-docker
RunningavirtualserverIntherepository,youwillfindtwofolderscontainingthenecessaryVagrantfiletolauncheitheraCentOS7oraUbuntu14.04virtualserver.
IfyouwouldliketousetheCentOS7vagrantbox,changethedirectorytovagrant-centos:
cdvagrant-centos
Onceyouareinthevagrant-centosdirectory,youwillseethatthereisaVagrantfile;thisfileisallyouneedtolaunchaCentOS7virtualserver.Afterthevirtualserverhasbeenbooted,thelatestversionofdockeranddocker-composewillbeinstalledandthemonitoring-dockerdirectorywillalsobemountedinsidethevirtualmachineusingthemountpoint/monitoring-docker.
Tolaunchthevirtualserver,simplytypethefollowingcommand:
vagrantup
Thiswilldownloadthelatestversionofthevagrantboxfromhttps://atlas.hashicorp.com/russmckendrick/boxes/centos71andthenbootthevirtualserver;it’sa450MBdownloadsoitmaytakeseveralminutestodownload;itonlyhastodothisonce.
Ifallgoeswell,youshouldseesomethingsimilartothefollowingoutput:
Nowthatyouhavebootedthevirtualserver,youcanconnecttoitusingthefollowingcommand:
vagrantssh
Onceloggedin,youshouldverifythatdockeranddocker-composearebothavailable:
Finally,youcantryrunningthehello-worldcontainerusingthefollowingcommand:
dockerrunhello-world
Ifeverythinggoesasexpected,youshouldseethefollowingoutput:
Totrysomethingmoreambitious,youcanrunanUbuntucontainerwiththefollowingcommand:
dockerrun-itubuntubash
BeforewelaunchandentertheUbuntucontainer,letsconfirmthatwearerunningtheCentOShostmachinebycheckingthereleasefilethatcanbefoundin/etc:
Now,wecanlaunchtheUbuntucontainer.Usingthesamecommand,wecanconfirmthatweareinsidetheUbuntucontainerbyviewingitsreleasefile:
Toexitthecontainerjusttypeinexit.Thiswillstopthecontainerfromrunning,asithasterminatedtheonlyrunningprocesswithinthecontainer,whichwasbash,andreturnedyoutothehostCentOSmachine.
AsyoucanseeherefromourCentOS7host,wehavelaunchedandremovedanUbuntucontainer.
BoththeCentOS7andUbuntuVagrantfileswillconfigureastaticIPaddressonyourvirtualmachine.Itis192.168.33.10;also,thereisaDNSrecordforthisIPaddress
availableatdocker.media-glass.es.Thesewillallowyoutoaccessanycontainersthatexposethemselvestoabrowserateitherhttp://192.168.33.10/orhttp://docker.media-glass.es/.
TipTheURLhttp://docker.media-glass.es/willonlyworkwhilethevagrantboxisup,andyouhaveacontainerrunningwhichservesWebpages.
Youcanseethisinactionbyrunningthefollowingcommand:
dockerrun-d-p80:80russmckendrick/nginx-php
TipDownloadingtheexamplecode
Youcandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.comforallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbookelsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilese-maileddirectlytoyou.
ThiswilldownloadandlaunchacontainerrunningNGINX.Youcanthengotohttp://192.168.33.10/orhttp://docker.media-glass.es/inyourbrowser;youshouldseeaforbiddenpage.ThisisbecausewehavenotyetgivenNGINXanycontenttoserve(moreonthiswillbecoveredlaterinthebook):
Formoreexamplesandideas,gotothewebsiteathttp://docs.docker.com/userguide/.
HaltingthevirtualserverTologoutofthevirtualserverandreturntoyourlocalmachine,youtypeexit.
Youshouldnowseeyourlocalmachine’sterminalprompt;however,thevirtualserveryoubootedwillstillberunninginthebackgroundhappily,usingresources,untilyoueitherpoweritdownusingthefollowingcommand:
vagranthalt
Terminatethevirtualserveraltogetherusingvagrantdestroy:
vagrantdestroy
Tocheckthecurrentstatusofthevirtualserver,youcanrunthefollowingcommand:
vagrantstatus
Theresultoftheprecedingcommandisgiveninthefollowingoutput:
Eitherpoweringthevirtualserverbackonorcreatingitfromscratchagain,canbeachievedbyissuingthevagrantupcommandagain.
TheprecedingdetailsshowhowtousetheCentOS7vagrantbox.IfyouwouldprefertolaunchanUbuntu14.04virtualserver,youcandownloadandinstallthevagrantboxbygoingintothevagrant-ubuntudirectoryusingthefollowingcommand:
cd~/Documents/Projects/monitoring-docker/vagrant-ubuntu
vagrantup
Fromhere,youwillbeablerunvagrantupandfollowthesameinstructionsusedtobootandinteractwiththeCentOS7virtualserver.
SummaryInthischapter,wetalkedaboutdifferenttypesofserverandalsodiscussedhowyourcontainerizedapplicationscanfitintoeachofthecategories.WehavealsoinstalledVirtualBoxandusedVagranttolauncheitheraCentOS7orUbuntu14.04virtualserver,withdockeranddocker-composeinstalled.
Ournewvirtualserverenvironmentwillbeusedthroughouttheupcomingchapterstotestthevariousdifferenttypesofmonitoring.Inthenextchapter,wewillstartourjourneybyusingDocker’sin-builtfunctionalitytoexploremetricsaboutourrunningcontainers.
Chapter2.UsingtheBuilt-inToolsInthelaterchaptersofthisbook,wewillexplorethemonitoringpartsofthelargeeco-systemthathasstartedtoflourisharoundDockeroverthelast24months.However,beforewepressaheadwiththat,weshouldtakealookatwhatispossiblewithavanillainstallationofDocker.Inthischapter,wewillcoverthefollowingtopics:
UsingthetoolsbuiltintoDockertogetreal-timemetricsoncontainerperformanceUsingstandardoperatingsystemcommandstogetmetricsonwhatDockerisdoingGeneratingatestloadsoyoucanviewthemetricschanging
DockerstatsSinceversion1.5,therehasbeenabasicstatisticcommandbuiltintoDocker:
dockerstats--help
Usage:dockerstats[OPTIONS]CONTAINER[CONTAINER…]
Displayalivestreamofoneormorecontainers'resourceusagestatistics
--help=falsePrintusage
--no-stream=falseDisablestreamingstatsandonlypullthefirst
result
Thiscommandwillstreamdetailsoftheresourceutilizationofyourcontainersinrealtime.Thebestwaytofindoutaboutthecommandistoseeitinaction.
RunningDockerstatsLet’slaunchacontainerusingthevagrantenvironment,whichwecoveredinthelastchapter:
[russ@mac~]$cd~/Documents/Projects/monitoring-docker/vagrant-centos/
[russ@mac~]$vagrantup
Bringingmachine'default'upwith'virtualbox'provider…
==>default:Importingbasebox'russmckendrick/centos71'...
==>default:MatchingMACaddressforNATnetworking…
==>default:Checkingifbox'russmckendrick/centos71'isuptodate…
.....
==>default:=>Installingdocker-engine…
==>default:=>Configuringvagrantuser…
==>default:=>Startingdocker-engine…
==>default:=>Installingdocker-compose…
==>default:=>FinishedinstallationofDocker
[russ@mac~]$vagrantssh
Nowthatyouareconnectedtothevagrantserver,launchthecontainerusingtheDockercomposefilein/monitoring_docker/Chapter01/01-basic/:
[vagrant@centos7~]$cd/monitoring_docker/Chapter01/01-basic/
[vagrant@centos701-basic]$docker-composeup-d
Creating01basic_web_1…
Youhavenowpulleddownandlaunchedacontainerinthebackground.Thecontaineriscalled01basic_web_1anditrunsNGINXandPHPservingasinglePHPinformationpage(http://php.net/manual/en/function.phpinfo.php).
Tocheckwhethereverythinghasbeenlaunchedasexpected,rundocker-composeps.YoushouldseeyoursinglecontainerwithStateofUp:
[vagrant@centos701-basic]$docker-composeps
NameCommandStatePorts
---------------------------------------------------------------
01basic_web_1/usr/local/bin/runUp0.0.0.0:80->80/tcp
Finally,youshouldbeabletoseethepagecontainingtheoutputofthePHPinformationathttp://192.168.33.10/(thisIPaddressishardcodedintothevagrantconfiguration),ifyouputitinyourlocalbrowser:
Now,youhaveacontainerupandrunning;let’slookatsomeofthebasicstats.Weknowfromtheoutputofdocker-composethatourcontaineriscalled01basic_web_1,soenterthefollowingcommandtostartstreamingstatisticsinyourterminal:
dockerstats01basic_web_1
Itwilltakeasecondtoinitiate;afterthisisdone,youshouldseeyourcontainerlistedalongwiththestatisticsforthefollowing:
CPU%:ThisshowsyouhowmuchoftheavailableCPUresourcethecontaineriscurrentlyusing.MEMUSEAGE/LIMIT:ThistellsyouhowmuchRAMthecontainerisutilizing;italsodisplayshowmuchallowancethecontainerhas.Ifyouhaven’texplicitlysetalimit,itwillshowthetotalamountofRAMonthehostmachine.MEM%:ThisshowsyouwhatpercentageoftheRAMallowancethecontainerisusing.
NETI/O:Thisgivesarunningtotalofhowmuchbandwidthhasbeentransferredinandoutofthecontainer.
Ifyougobacktoyourbrowserwindowandstarttorefreshhttp://192.168.33.10/,youwillseethatthevaluesineachofthecolumnsstarttochange.Tostopstreamingthestatistics,pressCtrl+c.
Ratherthankeepingonhittingrefreshoverandoveragain,let’sgeneratealotoftrafficto01basic_web_1,whichshouldputthecontainerunderaheavyload.
Here,wewilllaunchacontainerthatwillsend10,000requeststo01basic_web_1usingApacheBench(https://httpd.apache.org/docs/2.2/programs/ab.html).Althoughitwilltakeaminuteortwotoexecute,weshouldrundockerstatsassoonaspossible:
dockerrun-d--name=01basic_load--link=01basic_web_1russmckendrick/abab
-k-n10000-c5http://01basic_web_1/&&dockerstats01basic_web_1
01basic_load
AftertheApacheBenchimagehasbeendownloadedandthecontainerthatwillbecalled01basic_loadstarts,youshouldseethestatisticsforboth01basic_web_1and01basic_loadbegintostreaminyourterminal:
CONTAINERCPU%MEMUSAGE/LIMITMEM%NETI/O
01basic_load18.11%12.71MB/1.905GB0.67%335.2MB/5.27MB
01basic_web_1139.62%96.49MB/1.905GB5.07%5.27MB/335.2MB
Afterawhile,youwillnoticethatmostofthestatisticsfor01basic_loadwilldropofftozero;thismeansthatthetesthascompletedandthatthecontainerrunningthetesthasexited.Thedockerstatscommandcanonlystreamstatisticsfortherunningcontainers;onesthathaveexitedarenolongerrunningand,therefore,donotproduceoutputwhenrunningdockerstats.
ExitfromdockerstatsusingCtrl+c;toseetheresultsoftheApacheBenchcommand,youcantypedockerlogs01basic_load;youshouldseesomethinglikethefollowingscreenshot:
Youshouldn’tworryifyouseeanyfailureslikeintheprecedingoutput.ThisexercisewaspurelytodemonstratehowtoviewthestatisticsoftherunningcontainersandnottotuneawebservertohandletheamountoftrafficwesenttoitusingApacheBench.
Toremovethecontainersthatwelaunched,runthefollowingcommands:
[vagrant@centos701-basic]$docker-composestop
Stopping01basic_web_1…
[vagrant@centos701-basic]$docker-composerm
Goingtoremove01basic_web_1
Areyousure?[yN]y
Removing01basic_web_1…
[vagrant@centos701-basic]$dockerrm01basic_load
01basic_load
Tocheckwhethereverythinghasbeenremovedsuccessfully,rundockerps-aandyoushouldnotbeabletoseeanyrunningorexitedcontainersthathave01basic_intheirnames.
Whatjusthappened?WhilerunningtheApacheBenchtest,youmayhavenoticedthattheCPUutilizationonthecontainerrunningNGINXandPHPwashigh;intheexampleintheprevioussection,itwasusing139.62percentoftheavailableCPUresource.
Aswedidnotattachanyresourcelimitstothecontainerswelaunched,itwaseasyforourtesttousealloftheavailableresourcesonthehostVirtualMachine(VM).IfthisVMwasbeingusedbyseveralusers,allrunningtheirowncontainers,theymayhavestartedtonoticethattheirapplicationshadstartedtoslowdownor,evenworse,theapplicationshadstartedshowingerrors.
Ifyoueverfindyourselfinthissituation,youcanusedockerstatstohelptrackdowntheculprit.
Runningdockerstats$(dockerps-q)willstreamthestatisticsforallthecurrentlyrunningcontainers:
CONTAINERCPU%MEMUSAGE/LIMITMEM%NETI/O
361040b7b33e0.07%86.98MB/1.905GB4.57%2.514kB/738B
56b459ae9092120.06%87.05MB/1.905GB4.57%2.772kB/738B
a3de616f84ba0.04%87.03MB/1.905GB4.57%2.244kB/828B
abdbee7b52070.08%86.61MB/1.905GB4.55%3.69kB/738B
b85c49cf740c0.07%86.15MB/1.905GB4.52%2.952kB/738B
Asyoumayhavenoticed,thisdisplaysthecontainerIDratherthanthename;thisinformationshould,however,beenoughtospottheresourcehogsothatyoucanquicklystopit:
[vagrant@centos701-basic]$dockerstop56b459ae9092
56b459ae9092
Oncestopped,youcanthengetthenameoftheroguecontainerbyrunningthefollowingcommand:
[vagrant@centos701-basic]$dockerps-a|grep56b459ae9092
56b459ae9092russmckendrick/nginx-php"/usr/local/bin/run"9
minutesagoExited(0)26secondsagomy_bad_container
Alternatively,formoredetailedinformation,youcanrundockerinspect56b459ae9092,whichwillgiveyoualltheinformationyouneedonthecontainer.
Whataboutprocesses?OneofthegreatthingsaboutDockeristhatitisn’treallyvirtualization;asmentionedinthepreviouschapter,itisagreatwayofisolatingprocessesratherthanrunninganentireoperatingsystem.
Thiscangetconfusingwhenrunningtoolssuchastoporps.Togetanideajusthowconfusingthiscanget,letslaunchseveralcontainersusingdocker-composeandseeforourselves:
[vagrant@centos7~]$cd/monitoring_docker/Chapter01/02-multiple
[vagrant@centos702-multiple]$docker-composeup-d
Creating02multiple_web_1…
[vagrant@centos702-multiple]$docker-composescaleweb=5
Creating02multiple_web_2…
Creating02multiple_web_3…
Creating02multiple_web_4…
Creating02multiple_web_5…
Starting02multiple_web_2…
Starting02multiple_web_3…
Starting02multiple_web_4…
Starting02multiple_web_5…
Now,wehavefivewebserversthathaveallbeenlaunchedfromthesameimageusingthesameconfiguration.OneofthefirstthingsIdowhenloggingintoaservertotroubleshootaproblemisrunps-aux;thiswillshowalltherunningprocesses.Asyoucansee,whenrunningthecommand,therearealotprocesseslisted.
EvenjusttryingtolookattheprocessesforNGINXisconfusing,asthereisnothingtodifferentiatetheprocessesfromonecontainertoanother,asshowninthefollowingoutput:
DockertopThiscommandlistsalltheprocessesthatarerunningwithinacontainer;thinkofitasawayoffilteringtheoutputofthepsauxcommandweranonthehostmachine:
Asdockertopisanimplementationofthestandardpscommand,anyflagsyouwouldnormallypasstopsshouldworkasfollows:
[vagrant@centos702-multiple]$dockertop02multiple_web_3–aux
[vagrant@centos702-multiple]$dockertop02multiple_web_3-faux
DockerexecAnotherwaytoviewwhatisgoingonwithinacontaineristoenterit.Toenableyoutodothis,Dockerintroducedthedockerexeccommand.Thisallowsyoutospawnanadditionalprocesswithinanalreadyrunningcontainerandthenattachtotheprocess;so,ifwewantedtolookatwhatiscurrentlyrunningon02multiple_web_3,weshouldusethefollowingcommandspawnabashshellwithinanalreadyrunningcontainer:
dockerexec-t-i02multiple_web_3bash
Onceyouhaveanactiveshellonthecontainer,youwillnoticethatyourprompthaschangedtothecontainer’sID.Yoursessionisnowisolatedtothecontainer’senvironment,meaningthatyouwillonlybeabletointeractwiththeprocessesbelongingtothecontaineryouentered.
Fromhere,youcanrunthepsauxortopcommandasyouwoulddoonthehostmachine,andonlyseetheprocessesassociatedwiththecontaineryouareinterestedin:
Toleavethecontainer,typeinexit,youshouldseeyourpromptchangebackinyourhostmachine.
Finally,youcanstopandremovethecontainersbyrunningdocker-composestopanddocker-composekill.
SummaryInthischapter,wesawhowwecangetreal-timestatisticsonourrunningcontainersandhowwecanusecommandsthatarefamiliartous,togetinformationontheprocessesthatarelaunchedaspartofeachcontainer.
Onthefaceofit,dockerstatsseemslikeareallybasicpieceoffunctionalitythatisn’treallyanythingmorethanatooltohelpyouidentifywhichcontainerisusingalltheresourceswhileaproblemisoccurring.However,theDockercommandisactuallypullingtheinformationfromaquitepowerfulAPI.
ThisAPIformsthebasisforalotofthemonitoringtoolswewillbelookingatinthenextfewchapters.
Chapter3.AdvancedContainerResourceAnalysisInthelastchapter,welookedathowyoucanusetheAPIbuiltintoDockertogainaninsighttowhatresourcesyourcontainersarerunning.Now,wearetoseehowwecantakeittothenextlevelbyusingcAdvisorfromGoogle.Inthischapter,youwillcoverthefollowingtopics:
HowtoinstallcAdvisorandstartcollectingmetricsLearnallaboutthewebinterfaceandreal-timemonitoringWhatyouroptionsareforshippingmetricstoaremotePrometheusdatabaseforlong-termstorageandtrendanalysis
WhatiscAdvisor?GoogledescribescAdvisorasfollows:
“cAdvisor(ContainerAdvisor)providescontainerusersanunderstandingoftheresourceusageandperformancecharacteristicsoftheirrunningcontainers.Itisarunningdaemonthatcollects,aggregates,processes,andexportsinformationaboutrunningcontainers.Specifically,foreachcontainer,itkeepsresourceisolationparameters,historicalresourceusage,histogramsofcompletehistoricalresourceusage,andnetworkstatistics.Thisdataisexportedbyacontainerandismachine-wide.”
TheprojectstartedofflifeasaninternaltoolatGoogleforgaininganinsightintocontainersthathadbeenlaunchedusingtheirowncontainerstack.
NoteGoogle’sowncontainerstackwascalled“LetMeContainThatForYou”orlmctfyforshort.TheworkonlmctfyhasbeeninstalledasaGoogleportfunctionalityovertolibcontainerthatispartoftheOpenContainerInitiative.Furtherdetailsonlmctfycanbefoundathttps://github.com/google/lmctfy/.
cAdvisoriswritteninGo(https://golang.org);youcaneithercompileyourownbinaryoryoucanusethepre-compiledbinarythataresuppliedviaacontainer,whichisavailablefromGoogle’sownDockerHubaccount.Youcanfindthisathttp://hub.docker.com/u/google/.
Onceinstalled,cAdvisorwillsitinthebackgroundandcapturemetricsthataresimilartothatofthedockerstatscommand.Wewillgothroughthesestatsandunderstandwhattheymeanlaterinthischapter.
cAdvisortakesthesemetricsalongwiththoseforthehostmachineandexposesthemviaasimpleandeasy-to-usebuilt-inwebinterface.
RunningcAdvisorusingacontainerThereareanumberofwaystoinstallcAdvisor;theeasiestwaytogetstartedistodownloadandrunthecontainerimagethatcontainsacopyofaprecompiledcAdvisorbinary.
BeforerunningcAdvisor,let’slaunchafreshvagranthost:
[russ@mac~]$cd~/Documents/Projects/monitoring-docker/vagrant-centos/
[russ@mac~]$vagrantup
Bringingmachine'default'upwith'virtualbox'provider…
==>default:Importingbasebox'russmckendrick/centos71'...
==>default:MatchingMACaddressforNATnetworking…
==>default:Checkingifbox'russmckendrick/centos71'isuptodate…
.....
==>default:=>Installingdocker-engine…
==>default:=>Configuringvagrantuser…
==>default:=>Startingdocker-engine…
==>default:=>Installingdocker-compose…
==>default:=>FinishedinstallationofDocker
[russ@mac~]$vagrantssh
TipUsingabackslash
Aswehavealotoptionstopasstothedockerruncommand,weareusing\tosplitthecommandovermultiplelinessoit’seasiertofollowwhatisgoingon.
Onceyouhaveaccesstothehostmachine,runthefollowingcommand:
dockerrun\
--detach=true\
--volume=/:/rootfs:ro\
--volume=/var/run:/var/run:rw\
--volume=/sys:/sys:ro\
--volume=/var/lib/docker/:/var/lib/docker:ro\
--publish=8080:8080\
--privileged=true\
--name=cadvisor\
google/cadvisor:latest
YoushouldnowhaveacAdvisorcontainerupandrunningonyourhostmachine.Beforewestart,let’slookatcAdvisorinmoredetailbydiscussingwhywehavepassedalltheoptionstothecontainer.
ThecAdvisorbinaryisdesignedtorunonthehostmachinealongsidetheDockerbinary,sobylaunchingcAdvisorinacontainer,weareactuallyisolatingthebinaryinitsdownenvironment.TogivecAdvisoraccesstotheresourcesitrequiresonthehostmachine,wehavetomountseveralpartitionsandalsogivethecontainerprivilegedaccesstoletthecAdvisorbinarythinkitisbeingexecutedonthehostmachine.
NoteWhenacontainerislaunchedwith--privileged,Dockerwillenablefullaccesstodevicesonthehostmachine;also,DockerwillconfigurebothAppArmororSELinuxtoallowyourcontainerthesameaccesstothehostmachineasaprocessrunningoutsidethecontainerwillhave.Forinformationonthe--privilegedflag,seethispostontheDockerblogathttp://blog.docker.com/2013/09/docker-can-now-run-within-docker/.
CompilingcAdvisorfromsourceAsmentionedintheprevioussection,cAdvisorreallyoughttobeexecutedonthehostmachine;thismeans,youmayhavetouseacasetocompileyourowncAdvisorbinaryandrunitdirectlyonthehost.
TocompilecAdvisor,youwillneedtoperformthefollowingsteps:
1. InstallGoandMercurialonthehostmachine—version1.3orhigherofGoisneededtocompilecAdvisor.
2. SetthepathforGotoworkfrom.3. GrabthesourcecodeforcAdvisorandgodep.4. SetthepathforyourGobinaries.5. BuildthecAdvisorbinaryusinggodeptosourcethedependenciesforus.6. Copythebinaryto/usr/local/bin/.7. DownloadeitheranUpstartorSystemdscriptandlaunchtheprocess.
Ifyoufollowedtheinstructionsintheprevioussection,youwillalreadyhaveacAdvisorprocessrunning.Beforecompilingfromsource,youshouldstartwithacleanhost;let’slogoutofthehostandlaunchafreshcopy:
[vagrant@centos7~]$exit
logout
Connectionto127.0.0.1closed.
[russ@mac~]$vagrantdestroy
default:Areyousureyouwanttodestroythe'default'VM?[y/N]y
==>default:ForcingshutdownofVM…
==>default:DestroyingVMandassociateddrives…
==>default:Runningcleanuptasksfor'shell'provisioner…
[russ@mac~]$vagrantup
Bringingmachine'default'upwith'virtualbox'provider…
==>default:Importingbasebox'russmckendrick/centos71'...
==>default:MatchingMACaddressforNATnetworking…
==>default:Checkingifbox'russmckendrick/centos71'isuptodate…
.....
==>default:=>Installingdocker-engine…
==>default:=>Configuringvagrantuser…
==>default:=>Startingdocker-engine…
==>default:=>Installingdocker-compose…
==>default:=>FinishedinstallationofDocker
[russ@mac~]$vagrantssh
TobuildcAdvisorontheCentOS7host,runthefollowingcommand:
sudoyuminstall-ygolanggitmercurial
exportGOPATH=$HOME/go
goget-dgithub.com/google/cadvisor
gogetgithub.com/tools/godep
exportPATH=$PATH:$GOPATH/bin
cd$GOPATH/src/github.com/google/cadvisor
godepgobuild.
sudocpcadvisor/usr/local/bin/
sudowgethttps://gist.githubusercontent.com/russmckendrick/f647b2faad5d92c96
771/raw/86b01a044006f85eebbe395d3857de1185ce4701/cadvisor.service-O
/lib/systemd/system/cadvisor.service
sudosystemctlenablecadvisor.service
sudosystemctlstartcadvisor
OntheUbuntu14.04LTShost,runthefollowingcommand:
sudoapt-get-yinstallsoftware-properties-common
sudoadd-apt-repositoryppa:evarlast/golang1.4
sudoapt-getupdate
sudoapt-get-yinstallgolangmercurial
exportGOPATH=$HOME/go
goget-dgithub.com/google/cadvisor
gogetgithub.com/tools/godep
exportPATH=$PATH:$GOPATH/bin
cd$GOPATH/src/github.com/google/cadvisor
godepgobuild.
sudocpcadvisor/usr/local/bin/
sudowgethttps://gist.githubusercontent.com/russmckendrick/f647b2faad5d92c96
771/raw/e12c100d220d30c1637bedd0ce1c18fb84beff77/cadvisor.conf-O
/etc/init/cadvisor.conf
sudostartcadvisor
YoushouldnowhavearunningcAdvisorprocess.Youcancheckthisbyrunningpsaux|grepcadvisorandyoushouldseeaprocesswithapathof/usr/local/bin/cadvisorrunning.
CollectingmetricsNow,youhavecAdvisorrunning;whatdoyouneedtodotoconfiguretheserviceinordertostartcollectingmetrics?Theshortansweris,nothingatall.WhenyoustartedthecAdvisorprocess,itinstantlystartedpollingyourhostmachinetofindoutwhatcontainersarerunningandgatheredinformationonboththerunningcontainersandyourhostmachine.
TheWebinterfacecAdvisorshouldberunningonthe8080port;ifyouopenhttp://192.168.33.10:8080/,youshouldbegreetedwiththecAdvisorlogoandanoverviewofyourhostmachine:
Thisinitialpagestreamslivestatsaboutthehostmachine,thougheachsectionisrepeatedwhenyoustarttodrilldownandviewthecontainers.Tostartwith,let’slookateachsectionusingthehostinformation.
OverviewThisoverviewsectiongivesyouabird’s-eyeviewofyoursystem;itusesgaugessoyoucanquicklygetanideaofwhichresourcesarereachingtheirlimits.Inthefollowingscreenshot,thereisverylittleinthewayofCPUutilizationandthefilesystemusageisrelativelylow;however,weareusing64%oftheavailableRAM:
ProcessesThefollowingscreenshotdisplaysacombinedviewoftheoutputofthepsaux,dockerpsandtopcommandsweusedinthepreviouschapter:
Hereiswhateachcolumnheadingmeans:
User:ThisshowswhichuserisrunningtheprocessPID:ThisistheuniqueprocessIDPPID:ThisisthePIDoftheparentprocessStartTime:ThisshowswhattimetheprocessstartedCPU%:ThisisthepercentageoftheCPUtheprocessiscurrentlyconsumingMEM%:ThisisthepercentageoftheRAMtheprocessiscurrentlyconsumingRSS:ThisshowshowmuchofthemainmemorytheprocessisusingVirtualSize:ThisshowshowmuchofthevirtualmemorytheprocessisusingStatus:Thisshowsthecurrentstatusoftheprocess;thisarethestandardLinuxprocessstatecodesRunningTime:ThisshowshowlongtheprocesshasbeenrunningCommand:ThisshowswhichcommandtheprocessisrunningContainer:Thisshowswhichcontainertheprocessisattachedto;thecontainerlistedas/isthehostmachine
Astherecouldbeseveralhundredprocessesactive,thissectionissplitintopages;youcannavigatetothesewiththebuttonsonthebottom-left.Also,youcansorttheprocessesbyclickingonanyoftheheadings.
CPUThefollowinggraphshowstheCPUutilizationoverthelastminute:
Hereiswhateachtermmeans:
TotalUsage:ThisshowsanaggregateusageacrossallcoresUsageperCore:ThisgraphbreaksdowntheusagepercoreUsageBreakdown(notshownintheprecedingscreenshot):Thisshowsaggregateusageacrossallcores,butbreaksitdowntowhatisbeingusedbythekernelandwhatisbeingusedbytheuser-ownedprocesses
MemoryTheMemorysectionissplitintotwoparts.Thegraphtellsyouthetotalamountofmemoryusedbyalltheprocessesforthehostorcontainer;thisisthetotalofthehotandcoldmemory.TheHotmemoryisthecurrentworkingset:pagesthathavebeentouchedbythekernelrecently.TheColdmemoryisthepagethathasn’tbeentouchedforawhileandcouldbereclaimedifneeded.
TheUsageBreakdowngivesavisualrepresentationofthetotalmemoryinthehostmachine,orallowanceinthecontainer,alongsidethetotalandhotusage:
NetworkThissectionshowstheincomingandoutgoingtrafficoverthelastminute.Youcanchangetheinterfaceusingthedrop-downboxonthetop-left.Thereisalsoagraphthatshowsanynetworkingerrors.Typically,thisgraphshouldbeflat.Ifitisn’t,thenyouwillbeseeingperformanceissueswithyourhostmachineorcontainer:
FilesystemThefinalsectiongivesabreakdownofthefilesystemusage.Inthefollowingscreenshot,/dev/sda1isthebootpartition,/dev/sda3isthemainfilesystem,and/dev/mapper/docker-8…isanaggregateofthewritefilesystemsofyourrunningcontainers:
ViewingcontainerstatsAtthetopofthepage,thereisalinkofyourrunningcontainers;youcaneitherclickonthelinkorgodirectlytohttp://192.168.33.10:8080/docker/.Oncethepageloads,youshouldseealistofallyourrunningcontainers,andalsoadetailedoverviewofyourDockerprocess,andfinallyalistoftheimagesyouhavedownloaded.
SubcontainersSubcontainersshowsalistofyourcontainers;eachentryisaclickablelinkthatwilltakeyoutoapagethatwillgiveyouthefollowingdetails:
Isolation:
CPU:ThisshowsyoutheCPUallowancesofthecontainer;ifyouhavenotsetanyresourcelimits,youwillseethehost’sCPUinformationMemory:Thisshowsyouthememoryallowancesofthecontainer;ifyouhavenotsetanyresourcelimits,yourcontainerwillshowanunlimitedallowance
Usage:
Overview:ThisshowsgaugessoyoucanquicklyseehowclosetoanyresourcelimitsyouareProcesses:ThisshowstheprocessesforjustyourselectedcontainerCPU:ThisshowstheCPUutilizationgraphsisolatedtojustyourcontainerMemory:Thisshowsthememoryutilizationofyourcontainer
DriverstatusThedrivergivesthebasicstatsonyourmainDockerprocess,alongwiththeinformationonthehostmachine’skernel,hostname,andalsotheunderlyingoperatingsystem.
Italsogivesinformationonthetotalnumberofcontainersandimages.Youmaynoticethatthetotalnumberofimagesisamuchlargerfigurethanyouexpectedtosee;thisisbecauseitiscountingeachfilesystemasanindividualimage.
NoteFormoredetailsonDockerimages,seetheDockeruserguideathttps://docs.docker.com/userguide/dockerimages/.
Italsogivesyouadetailedbreakdownofyourstorageconfiguration.
ImagesFinally,yougetalistoftheDockerimageswhichareavailableonthehostmachine.ItliststheRepository,Tag,Size,andwhentheimagewascreated,alongwiththeimages’uniqueID.Thisletsyouknowwheretheimageoriginatedfrom(Repository),whichversionoftheimageyouhavedownloaded(Tag)andhowbigtheimageis(Size).
Thisisallgreat,what’sthecatch?Soyouaremaybethinkingtoyourselfthatallofthisinformationavailableinyourbrowserisreallyuseful;beingabletoseereal-timeperformancemetricsinaneasilyreadableformatisareallyplus.
ThebiggestdrawbackofusingthewebinterfaceforcAdvisor,asyoumayhavenoticed,isthatitonlyshowsyouoneminute’sworthofmetrics;youcanquiteliterallyseetheinformationdisappearinginrealtime.
Asapaneofglassgivesareal-timeviewintoyourcontainers,cAdvisorisabrillianttool;ifyouwanttoreviewanymetricsthatareolderthanoneminute,youareoutofluck.
Thatis,unlessyouconfiguresomewheretostoreallofyourdata;thisiswherePrometheuscomesin.
PrometheusSowhat’sPrometheus?Itsdevelopersdescribeitasfollows:
Prometheusisanopen-sourcesystem’smonitoringandalertingtoolkitbuiltatSoundCloud.Sinceitsinceptionin2012,ithasbecomethestandardforinstrumentingnewservicesatSoundCloudandisseeinggrowingexternalusageandcontributions.
OK,butwhatdoesthathavetodowithcAdvisor?Well,Prometheushasquiteapowerfuldatabasebackendthatstoresthedataitimportsasatimeseriesofevents.
Wikipediadescribesatimeseriesasfollows:
“Atimeseriesisasequenceofdatapoints,typicallyconsistingofsuccessivemeasurementsmadeoveratimeinterval.Examplesoftimeseriesareoceantides,countsofsunspots,andthedailyclosingvalueoftheDowJonesIndustrialAverage.Timeseriesareveryfrequentlyplottedvialinecharts.”
https://en.wikipedia.org/wiki/Time_series
OneofthethingscAdvisordoes,bydefault,isexposeallthemetricsitiscapturingonasinglepageat/metrics;youcanseethisathttp://192.168.33.10:8080/metricsonourcAdvisorinstallation.Themetricsareupdatedeachtimethepageisloaded:
Asyoucanseeintheprecedingscreenshot,thisisjustasinglelongpageofrawtext.ThewayPrometheusworksisthatyouconfigureittoscrapethe/metricsURLatauser-definedinterval,let’ssayeveryfiveseconds;thetextisinaformatthatPrometheusunderstandsanditisingestedintothePrometheus’stimeseriesdatabase.
Whatthismeansisthat,usingPrometheus’spowerfulbuilt-inquerylanguage,youcanstarttodrilldownintoyourdata.Let’slookatgettingPrometheusupandrunning.
LaunchingPrometheusLikecAdvisorthereareseveralwaysyoucanlaunchPrometheus.Tostartwith,wewilllaunchacontainerandinjectourownconfigurationfilesothatPrometheusknowswhereourcAdvisorendpointis:
dockerrun\
--detach=true\
--
volume=/monitoring_docker/Chapter03/prometheus.yml:/etc/prometheus/promethe
us.yml\
--publish=9090:9090\
--name=prometheus\
prom/prometheus:latest
Onceyouhavelaunchedthecontainer,PrometheuswillbeaccessibleonthefollowingURL:http://192.168.33.10:9090.WhenyoufirstloadtheURL,youwillbetakentoastatuspage;thisgivessomebasicinformationonthePrometheusinstallation.Theimportantpartofthispageisthelistoftargets.ThisliststheURLthatPrometheuswillbescrappingtocapturemetrics;youshouldseeyourcAdvisorURLlistedwithastateofHEALTHY,asshowninthefollowingscreenshot:
Anotherinformationpagecontainsthefollowing:
Runtimeinformation:ThisdisplayshowlongPrometheushasbeenupandpollingdata,ifyouhaveconfiguredanendpointBuildinformation:ThiscontainsthedetailsoftheversionofPrometheusthatyouhavebeenrunningConfiguration:ThisisacopyoftheconfigurationfileweinjectedintothecontainerwhenitwaslaunchedRules:Thisisacopyofanyrulesweinjected;thesewillbeusedforalertingStartupflags:Thisshowsalltheruntimevariablesandtheirvalues
QueryingPrometheusAsweonlyhaveafewcontainersupandrunningatthemoment,let’slaunchonethatrunsRedissowecanstarttolookatthequerylanguagebuiltintoPrometheus.
WewillusetheofficialRedisimageforthisandasweareonlygoingtousethisasanexamplewewon’tneedtopassitanyuservariables:
dockerrun--namemy-redis-server-dredis
Wenowhaveacontainercalledmy-redis-serverrunning.cAdvisorshouldalreadybeexposingmetricsaboutthecontainertoPrometheus;let’sgoaheadandsee.InthePrometheuswebinterface,gototheGraphlinkinthemenuatthetopofthepage.Here,youwillbepresentedwithatextboxintowhichyoucanenteryourquery.Tostartwith,let’slookattheCPUusageoftheRediscontainer.
Inthebox,enterthefollowing:
container_cpu_usage_seconds_total{job="cadvisor",name="my-redis-server"}
Then,afterclickingonExecute,youshouldhavetworesultsreturned,listedintheConsoletabofthepage.Ifyouremember,cAdvisorrecordstheCPUusageofeachoftheCPUcoresthatthecontainerhasaccessto,whichiswhywehavetwovaluesreturned,onefor“cpu00”andonefor“cpu01”.ClickingontheGraphlinkwillshowyouresultsoveraperiodoftime:
Asyoucanseeintheprecedingscreenshot,wenowhaveaccesstotheusagegraphsforthelast25minutes,whichisabouthowlongagoIlaunchedtheRedisinstancebefore
DashboardAlso,whencreatingoneofthegraphsusingthequerytoolinthemainapplication,youcaninstallaseparateDashboardapplication.ThisrunsinasecondcontainerthatconnectstoyourmainPrometheuscontainerusingtheAPIasadatasource.
BeforewestarttheDashboardcontainer,weshouldinitializeaSQLite3databasetostoreourconfiguration.Toensurethatthedatabaseispersistent,wewillstorethisonthehostmachinein/tmp/prom/file.sqlite3:
dockerrun\
--volume=/tmp/prom:/tmp/prom\
-eDATABASE_URL=sqlite3:/tmp/prom/file.sqlite3\
prom/promdash./bin/rakedb:migrate
Oncewehaveinitializedthedatabase,wecanlaunchtheDashboardapplicationproperly:
dockerrun\
--detach=true\
--volume=/tmp/prom:/tmp/prom\
-eDATABASE_URL=sqlite3:/tmp/prom/file.sqlite3\
--publish=3000:3000\
--name=promdash\
prom/promdash
Theapplicationshouldnowbeaccessibleathttp://192.168.33.10:3000/.Thefirstthingweneedtodoissetupthedatasource.Todothis,clickontheServerslinkatthetopofthescreenandthenclickonNewServer.Here,youwillbeaskedtoprovidethedetailsofyourPrometheusserver.NametheserverandenterthefollowingURL:
Name:cAdvisorURL:http://192.168.33.10:9090ServerType:Prometheus
OnceyouclickonCreateServer,youshouldreceiveamessagesayingServerwassuccessfullycreated.Nextup,youneedtocreateadirectory;thisiswhereyourdashboardswillbestored.
ClickontheDashboardslinkinthetopmenuandthenclickonNewdirectoryandcreateonecalledTestdirectory.Now,youarereadytostartcreatingDashboards.ClickonNewDashboard,callitMyDashboard,placeitinTestdirectory.OnceyouclickonCreateDashboard,youwillbetakentothepreviewscreen.
Fromhere,youcanbuildupdashboardsusingthecontrolinthetopright-handsideofeachsection.Toadddata,yousimplyenterthequeryyouwouldliketoseeinthedashboardsection:
NoteFordetailedinformationonhowtocreateDashboards,seethePROMDASHsectionofthePrometheusdocumentationathttp://prometheus.io/docs/visualization/promdash/.
ThenextstepsAtthemoment,wearerunningPrometheusinasinglecontaineranditsdataisbeingstoredwithinthatsamecontainer.Thismeans,ifforanyreasonthecontaineristerminated,ourdataislost;italsomeansthatwecan’tupgradewithoutloosingoutdata.Togetaroundthisproblem,wecancreateadatavolumecontainer.
NoteAdatavolumecontainerisaspecialtypeofcontainerthatonlyexistsasstorageforothercontainers.Formoredetails,seetheDockeruserguideathttps://docs.docker.com/userguide/dockervolumes/#creating-and-mounting-a-data-volume-container.
Firstofall,let’smakesurewehaveremovedalltherunningPrometheuscontainers:
dockerstopprometheus&&dockerrmPrometheus
Nextup,let’screateadatacontainercalledpromdata:
dockercreate\
--volume=/promdata\
--name=promdata\
prom/prometheus/bin/true
Finally,launchPrometheusagain,thistime,usingthedatacontainer:
dockerrun\
--detach=true\
--volumes-frompromdata\
--
volume=/monitoring_docker/Chapter03/prometheus.yml:/etc/prometheus/promethe
us.yml\
--publish=9090:9090\
--name=prometheus\
prom/prometheus
Thiswillensurethat,ifyouhavetoupgradeorrelaunchyourcontainer,themetricsyouhavebeencapturingaresafeandsound.
WehaveonlytouchedonthebasicsofusingPrometheusinthissectionofthebook;forfurtherinformationontheapplication,Irecommendthefollowinglinksasagoodstartingpoint:
Documentation:http://prometheus.io/docs/introduction/overview/Twitter:https://twitter.com/PrometheusIOProjectpage:https://github.com/prometheus/prometheusGooglegroups:https://groups.google.com/forum/#!forum/prometheus-developers
Alternatives?TherearesomealternativestoPrometheus.OnesuchalternativeisInfluxDBthatdescribesitselfasfollows:
Anopen-sourcedistributedtimeseriesdatabasewithnoexternaldependencies.
However,atthetimeofwriting,cAdvisorisnotcurrentlycompatiblewiththelatestversionofInfluxDB.TherearepatchesinthecodebaseforcAdvisor;however,theseareyettomakeitthroughtotheGoogle-maintainedDockerImage.
FormoredetailsonInfluxDBandit’snewvisualizationcomplainapplicationChronograf,seetheprojectwebsiteathttps://influxdb.com/andformoredetailsonhowtoexportcAdvisorstatisticstoInfluxDB,seethesupportingdocumentationforcAdvisorathttps://github.com/google/cadvisor/tree/master/docs.
SummaryInthischapter,welearnedhowtotaketheviewingreal-timestatisticsofourcontainersoffthecommandlineandintothewebbrowser.WeexploredsomedifferentmethodstoinstallGoogle’scAdvisorapplicationandalsohowtouseitswebinterfacetokeepaneyeonourrunningcontainers.WealsolearnedhowtocapturemetricsfromcAdvisorandstorethemusingPrometheus,amoderntimeseriesdatabase.
Thetwomaintechnologieswehavecoveredinthischapterhaveonlybeenpublicallyavailableforlessthantwelvemonths.Inthenextchapter,wewilllookatusingamonitoringtoolthathasbeeninaSysAdminstoolboxforover10years—Zabbix.
Chapter4.ATraditionalApproachtoMonitoringContainersSofar,wehavelookedatonlyafewtechnologiestomonitorourcontainers,sointhischapter,wewillbelookingmoreatatraditionaltoolformonitoringservices.Bytheendofthischapter,youshouldknowyourwayaroundZabbixandthevariouswaysyoucanmonitoryourcontainers.Wewillcoverthefollowingtopicsinthischapter:
HowtorunaZabbixServerusingcontainersHowtolaunchaZabbixServeronavagrantmachineHowtoprepareourhostsystemformonitoringcontainersusingtheZabbixagentHowtofindyourwayaroundtheZabbixwebinterface
ZabbixFirstthingsfirst,whatisZabbixandwhyuseit?
Ihavepersonallybeenusingitsinceversion1.2;theZabbixsitedescribesitasfollows:
“WithZabbix,itispossibletogathervirtuallylimitlesstypesofdatafromthenetwork.High-performancereal-timemonitoringmeansthattensofthousandsofservers,virtualmachines,andnetworkdevicescanbemonitoredsimultaneously.Alongwithstoringthedata,visualizationfeaturesareavailable(overviews,maps,graphs,screens,andsoon),aswellasveryflexiblewaysofanalyzingthedataforthepurposeofalerting.
Zabbixoffersgreatperformancefordatagatheringandcanbescaledtoverylargeenvironments.DistributedmonitoringoptionsareavailablewiththeuseofZabbixproxies.Zabbixcomeswithaweb-basedinterface,secureuserauthentication,andaflexibleuserpermissionschema.Pollingandtrappingissupported,withnativehigh-performanceagentsgatheringdatafromvirtuallyanypopularoperatingsystem;agent-lessmonitoringmethodsareavailableaswell.”
AtthetimeIstartedusingZabbix,theonlyrealviableoptionswereasfollows:
Nagios:https://www.nagios.org/Zabbix:http://www.zabbix.com/Zenoss:http://www.zenoss.org/
Outofthethesethreeoptions,Zabbixseemedtobethemoststraightforwardoneatthetime.ItwasdoingenoughworktomanagetheseveralhundredserversIwasgoingtomonitorwithouthavingtohavetheextraworkoflearningthecomplexitiesofsettingupNagiosorZenoss;afterall,giventhetaskthesoftwarehad,IneededtobeabletotrustthatIhadsetitupcorrectly.
Inthischapter,whileIamgoingtogointosomedetailaboutthesetupandthebasicsofusingZabbix,wewillonlybetouchingonsomeofthefunctionalities,whichcandoalotmorethanjustmonitoryourcontainers.Formoreinformation,Iwouldrecommendthefollowingasagoodstartingpoint:
Zabbixblog:http://blog.zabbix.comZabbix2.4manual:https://www.zabbix.com/documentation/2.4/manualFurtherreading:https://www.packtpub.com/all/?search=zabbix
InstallingZabbixAsyoumayhavenoticedfromthelinksintheprevioussection,therearealotofmovingpartsinZabbix.Itleveragesseveralopensourcetechnologies,andaproduction-readyinstallationneedsalittlemoreplanningthanwecangointointhischapter.BecauseofthiswearegoingtolookattwowaysofinstallingZabbixquicklyrathergointotoomuchdetail.
UsingcontainersAtthetimeofwriting,thereareoverahundredDockerimagesavailableontheDockerHub(https://hub.docker.com)thatmentionsZabbix.Theserangefromfullserverinstallationstojustthevariousparts,suchastheZabbixagentorproxyservices.
Outoftheoneslisted,thereisonethatisrecommendbyZabbixitself.So,wewilllookatthisone;itcanbefoundatthefollowingURLs:
DockerHub:https://hub.docker.com/u/zabbix/Projectpage:https://github.com/zabbix/zabbix-community-docker
TogettheZabbixServercontainerupandrunning,wemustfirstlaunchadatabasecontainer.Let’sstartafreshwithourvagrantinstancebyrunningthefollowingcommand:
[russ@mac~]$cd~/Documents/Projects/monitoring-docker/vagrant-centos/
[russ@mac~]$vagrantdestroy
default:Areyousureyouwanttodestroythe'default'VM?[y/N]y
==>default:ForcingshutdownofVM…
==>default:DestroyingVMandassociateddrives…
==>default:Runningcleanuptasksfor'shell'provisioner…
[russ@mac~]$vagrantup
Bringingmachine'default'upwith'virtualbox'provider…
==>default:Importingbasebox'russmckendrick/centos71'...
==>default:MatchingMACaddressforNATnetworking…
==>default:Checkingifbox'russmckendrick/centos71'isuptodate…
.....
==>default:=>Installingdocker-engine…
==>default:=>Configuringvagrantuser…
==>default:=>Startingdocker-engine…
==>default:=>Installingdocker-compose…
==>default:=>FinishedinstallationofDocker
[russ@mac~]$vagrantssh
Now,wehaveacleanenvironmentandit’stimetolaunchourdatabasecontainer,asfollows:
dockerrun\
--detach=true\
--publish=3306\
--env="MARIADB_USER=zabbix"\
--env="MARIADB_PASS=zabbix_password"\
--name=zabbix-db\
million12/mariadb
Thiswilldownloadthemillion12/mariadbimagefromhttps://hub.docker.com/r/million12/mariadb/andlaunchacontainercalledzabbix-db,runningMariaDB10(https://mariadb.org)withausercalledzabbixwhohasapasswordzabbix_password.WehavealsoopenedtheMariaDBport3306uponthecontainer,butaswewillbeconnectingtoitfromalinkedcontainer,thereisnoneedtoexposethatportonthehostmachine.
Now,wehavethedatabasecontainerupandrunning,wenowneedtolaunchourZabbixServercontainer:
dockerrun\
--detach=true\
--publish=80:80\
--publish=10051:10051\
--link=zabbix-db:db\
--env="DB_ADDRESS=db"\
--env="DB_USER=zabbix"\
--env="DB_PASS=zabbix_password"\
--name=zabbix\
zabbix/zabbix-server-2.4
Thisdownloadstheimage,whichatthetimeofwritingisover1GBsothisprocesscouldtakeseveralminutesdependingonyourconnection,andlaunchesacontainercalledzabbix.Itmapsthewebserver(port80)andtheZabbixServerprocess(port10051)onthehosttothecontainer,createsalinktoourdatabasecontainer,setsupthealiasdb,andinjectsthedatabasecredentialsasenvironmentvariablessothatthescriptsthatlaunchwhenthecontainerbootscanpopulatethedatabase.
Youcanverifythateverythingworkedasexpectedbycheckingthelogsonthecontainer.Todothis,enterdockerlogszabbix.Thiswillprintdetailsofwhathappenedwhenthecontainerlaunchedonscreen:
Now,oncewehavethecontainerupandrunning,itistimetomovetothebrowserforourfirsttasteofthewebinterface.Gotohttp://192.168.33.10/inyourbrowserandyouwillbegreetedbyawelcomepage;beforewecanstartusingZabbix,weneedtocompletetheinstallation.
Onthewelcomepage,clickonNexttobetakentothefirststep.ThiswillverifythateverythingweneedtorunaZabbixServerisinstalled.Aswehavelauncheditinacontainer,youshouldseeOKnexttoalloftheprerequisites.ClickonNexttomoveontothenextstep.
Now,weneedtoconfigurethedatabaseconnectionforthewebinterface.Here,youshouldhavethesamedetailsasyoudidwhenyoulaunchedthecontainer,asillustratedinthefollowingscreenshot:
Onceyouhaveenteredthedetails,clickonTestconnectionandyoushouldreceiveanOKmessage;youwillnotbeabletoproceeduntilthistestcompletessuccessfully.OnceyouhaveenteredthedetailsandhaveanOKmessage,clickonNext.
Nextup,arethedetailsontheZabbixServerthatthewebinterfaceneedstoconnectto;clickonNexthere.Nextup,youwillreceiveasummaryoftheinstallation.Toproceed,clickonNextandyouwillbegetconfirmationthatthe/usr/local/src/zabbix/frontends/php/conf/zabbix.conf.phpfilehasbeencreated.ClickonFinishtobetakentotheloginpage.
UsingvagrantWhilewritingthischapter,IthoughtalotaboutprovidinganothersetofinstallationinstructionsfortheZabbixServerservice.WhilethebookisallaboutMonitoringDockercontainers,havingaserviceasresourceintensiveasZabbixrunninginsideacontainerfeelsalittlecounterintuitive.Becauseofthis,thereisavagrantmachinethatusesPuppettobootstrapaworkinginstallationofZabbixServer:
[russ@mac~]$cd~/Documents/Projects/monitoring-docker/vagrant-zabbix/
[russ@mac~]$vagrantup
Bringingmachine'default'upwith'virtualbox'provider…
==>default:Importingbasebox'russmckendrick/centos71'...
==>default:MatchingMACaddressforNATnetworking…
==>default:Checkingifbox'russmckendrick/centos71'isuptodate…
.....
==>default:Debug:Receivedreporttoprocessfromzabbix.media-glass.es
==>default:Debug:Evictingcacheentryforenvironment'production'
==>default:Debug:Cachingenvironment'production'(ttl=0sec)
==>default:Debug:Processingreportfromzabbix.media-glass.eswith
processorPuppet::Reports::Store
Asyoumayhavenoticed,thereisalotofoutputstreamedtotheterminal,sowhatjusthappened?Firstofall,aCentOS7vagrantinstancewaslaunchedandthenaPuppetagentwasinstalled.Onceinstalled,theinstallationwashandedofftoPuppet.UsingtheZabbixPuppetmodulebyWernerDijkerman,ZabbixServerwasinstalled;formoredetailsonthemodule,seeitsPuppetForgepageathttps://forge.puppetlabs.com/wdijkerman/zabbix.
UnlikethecontainerizedversionofZabbixServer,thereisnoadditionalconfigurationrequired,soyoushouldbeabletoaccesstheZabbixloginpageathttp://zabbix.media-glass.es/(anIPaddressof192.168.33.11ishardcodedintotheconfiguration).
PreparingourhostmachineFortheremainderofthischapter,IwillassumethatyouareusingtheZabbixServerthatisrunningonitsownvagrantinstance.ThishelpstoensurethatyourenvironmentisconsistentwiththeconfigurationoftheZabbixagentwewillbelookingat.
TopassthestatisticsfromourcontainerstotheZabbixagent,whichwilltheninturnexposethemtotheZabbixServer,wewillbeinstallingusingtheZabbix-Docker-MonitoringZabbixagentmodulethathasbeendevelopedbyJanGaraj.Formoreinformationontheproject,seethefollowingURLs:
TheProjectpage:https://github.com/monitoringartist/Zabbix-Docker-Monitoring/TheZabbixsharepage:https://share.zabbix.com/virtualization/docker-containers-monitoring
Togettheagentandmoduleinstalled,configured,andrunning,weneedtoexecutethefollowingsteps:
1. InstalltheZabbixpackagerepository.2. InstalltheZabbixagent.3. Installtheprerequisitesforthemodule.4. AddtheZabbixagentusertotheDockergroup.5. Downloadtheauto-discoverybashscript.6. Downloadtheprecompiledzabbix_module_dockerbinary.7. ConfiguretheZabbixagentwiththedetailsofourZabbixServerandalsotheDocker
module.8. Setthecorrectpermissionsonallthefileswehavedownloadedandcreated.9. StarttheZabbixagent.
WhilethestepsremainthesameforbothCentOSandUbuntu,theactionstakentodotheinitialpackageinstallationdifferslightly.Ratherthangoingthroughtheprocessofshowingthecommandstoinstallandconfiguretheagent,thereisascriptforeachofthehostoperatingsystemsinthe/monitoring_docker/chapter04/folder.Toviewthescripts,runthefollowingcommandfromyourterminal:
cat/monitoring_docker/chapter04/install-agent-centos.sh
cat/monitoring_docker/chapter04/install-agent-ubuntu.sh
Now,youhavetakenalookatthescriptsitstimetorunthem,todothistypeoneofthefollowingcommands.IfyouarerunningCentOS,runthiscommand:
bash/monitoring_docker/chapter04/install-agent-centos.sh
ForUbuntu,runthefollowingcommand:
bash/monitoring_docker/chapter04/install-agent-ubuntu.sh
Toverifythateverythingranasexpected,checktheZabbixagentlogfilebyrunningthefollowingcommand:
cat/var/log/zabbix/zabbix_agentd.log
Youshouldseethattheendofthefileconfirmsthattheagenthasstartedandthatthezabbix_module_docker.somodulehasbeenloaded:
BeforewemoveontotheZabbixwebinterface,let’slaunchafewcontainersusingthedocker-composefilefromChapter2,UsingtheBuilt-inTools:
[vagrant@docker~]$cd/monitoring_docker/chapter02/02-multiple/
[vagrant@docker02-multiple]$docker-composeup-d
[vagrant@docker02-multiple]$docker-composescaleweb=3
[vagrant@docker02-multiple]$docker-composeps
WeshouldnowhavethreewebservercontainersrunningandarunningZabbixagentonthehost.
TheZabbixwebinterfaceOnceyouhaveZabbixinstalledyoucanopentheZabbixwebinterfacebygoingtohttp://zabbix.media-glass.es/inyourbrowser,thislinkwillonlyworkwhenyouhavetheZabbixvagrantboxupandrunning,ifyoudon’thaveitrunningthepagewilltimeout.Youshouldbepresentedwithaloginscreen.Enterthedefaultusernameandpasswordhere,whichisAdminandzabbix(notethattheusernamehasacapitalA),tologin.
Onceloggedin,youwillneedtoaddthehosttemplates.ThesearepreconfiguredenvironmentsettingsandwilladdsomecontextaroundthestatisticsthattheZabbixagentissendingtotheserver,alongwiththeauto-discoveryofcontainers.
Toaddthetemplates,gototheConfigurationtabinthetopmenuandselectTemplate;thiswillbringupalistofallthetemplatesthatarecurrentlyinstalled.ClickontheImportbuttonintheheaderanduploadacopyofthetwotemplatefilesyoucanfindinthe~/Documents/Projects/monitoring-docker/chapter04/templatefolderonyourmainmachine;thereisnoneedtochangetheruleswhenuploadingthetemplates.
Oncebothtemplateshavebeensuccessfullyimported,itistimetoaddourDockerhost.Again,gototheConfigurationtab,butthistimeselectHosts.Here,youneedtoclickonCreatehost.Then,enterthefollowinginformationintheHosttab:
Herearethedetailsoftheprecedinginformation:
Hostname:ThisisthehostnameofourDockerhostVisiblename:Here,thenameserverwillappearasinZabbixGroups:WhichgroupwithinZabbixtheserveryouwouldliketheDockerhosttobepartofAgentInterfaces:ThisistheIPaddressortheDNSnameofourDockerhostEnabled:Thisshouldbeticked
BeforeclickingonAdd,youshouldclickontheTemplatestabandlinkthefollowingtwotemplatestothehost:
TemplateAppDockerTemplateOSLinux
Hereisthescreenshotofthehost:
Onceyouhaveaddedthetwotemplates,clickonAddtoconfigureandenablethehost.Toverifythatthehosthasbeenaddedcorrectly,youshouldgototheMonitoringtabandthenLatestdata.Fromhere,clickonShowfilterandenterthehostmachineintheHostsbox.Youshouldthenstarttoseeitemsappearing:
Don’tworryifyoudon’tseetheDockersectionimmediately,bydefault,Zabbixwillattempttoauto-discovernewcontainerseveryfiveminutes.
DockermetricsForeachcontainer,Zabbixdiscoversthefollowingmetricsthatwillberecorded:
Container(yourContainersname)isrunningCPUsystemtimeCPUusertimeUsedcachememoryUsedRSSmemoryUsedswap
Apartfrom“Usedswap”,thesearethesamemetricsrecordedbycAdvisor.
CreatecustomgraphsYoucanaccessatime-basedgraphforanyofthemetricscollectedbyZabbix;youcanalsocreateyourowncustomgraphs.Inthefollowinggraph,IhavecreatedagraphthatplotsalltheCPUSystemstatsfromthethreewebcontainerswelaunchedearlierinthechapter:
Asyoucansee,IperformedafewtestsusingApacheBenchtomakethegraphalittlemoreinteresting.
Formoreinformationonhowtocreatecustomgraphs,seethegraphssectionofthedocumentationsiteathttps://www.zabbix.com/documentation/2.4/manual/config/visualisation/graphs.
ComparecontainerstoyourhostmachineAsweaddedtheLinuxOStemplateandtheDockertemplatetothehostandwearealsorecordingquitealotofinformationaboutthesystem,herewecantelltheeffectthetestingwithApacheBenchhadontheoverallprocessorload:
Wecandrilldownfurthertogetinformationontheoverallutilization:
TriggersAnotherfeatureofZabbixistriggers:youcandefineactionstohappenwhenametricmeetsacertainsetofcriteria.Inthefollowingexample,ZabbixhasbeenconfiguredwithatriggercalledContainerDown;thischangesthestatusofthemonitoreditemtoProblemwithaseverityofDisaster:
Thischangeinstatusthentriggersane-mailtoinformthat,forsomereasonthecontainerisnolongerupandrunning:
Thiscouldhavealsotriggeredothertasks,suchasrunningacustomscript,sendinganinstantmessageviaJabber,oreventriggeringathird-partyservicesuchasPagerDuty(https://www.pagerduty.com)orSlack(https://slack.com).
FormoreinformationonTriggers,Events,andNotifications,seethefollowingsectionsofthedocumentation:
https://www.zabbix.com/documentation/2.4/manual/config/triggershttps://www.zabbix.com/documentation/2.4/manual/config/eventshttps://www.zabbix.com/documentation/2.4/manual/config/notifications
SummarySo,howdoesthistraditionalapproachtomonitoringfitintoacontainer’slifecycle?
GoingbacktothePetsversusCattleanalogy,atfirstglance,ZabbixseemstobegearedmoretowardsPets:itsfeaturesetisbestsuitedtomonitoringservicesthatarestaticoveralongperiodoftime.Thismeansthatthesameapproachtomonitoringapetcanalsobeappliedtolong-runningprocessesrunningwithinyourcontainers.
Zabbixisalsotheperfectoptionformonitoringmixedenvironments.Maybeyouhaveseveraldatabaseserversthatarenotrunningascontainers,butyouhaveseveralhostsrunningDocker,andhaveequipmentsuchasswitchesandSANsthatyouneedtomonitor.Zabbixcanprovideyouwithasinglepaneofglassshowingyoumetricsforallyourenvironments,alongwithbeingabletoalertyoutoproblems.
Sofar,wehavelookedatusingAPIsandmetricsprovidedbyDockerandLXC,butwhataboutothermetricscanweuse?Inthenextchapter,wewilllookatatoolthathooksstraightintothehostmachine’skerneltogatherinformationonyourcontainers.
Chapter5.QueryingwithSysdigTheprevioustoolswehavelookedathaveallreliedonmakingAPIcallstoDockerorreadingmetricsfromLXC.Sysdigworksdifferentlybyhookingitselfintothehostsmachine’skernelwhilethisapproachdoesgoagainstDocker’sphilosophyofeachservicebeingruninitsownisolatedcontainer,theinformationyoucangetbyrunningSysdigonlyforafewminutesfaroutweighsanyargumentsaboutnotusingit.
Inthischapter,wewilllookatthefollowingtopics:
HowtoinstallSysdigandCsysdigonthehostmachineBasicusageandhowtoqueryyourcontainersinrealtimeHowtocapturelogssotheycanbequeriedlater
WhatisSysdig?BeforewestarttogetintoSysdig,let’sfirstunderstandwhatitis.WhenIfirstheardaboutthetool,Ithoughttomyselfthatitsoundedtoogoodtobetrue;thewebsitedescribesthetoolasfollows:
“Sysdigisopensource,system-levelexploration:capturesystemstateandactivityfromarunningLinuxinstance,thensave,filterandanalyze.SysdigisscriptableinLuaandincludesacommandlineinterfaceandapowerfulinteractiveUI,csysdig,thatrunsinyourterminal.Thinkofsysdigasstrace+tcpdump+htop+iftop+lsof+awesomesauce.Withstateoftheartcontainervisibilityontop.”
Thisisquiteaclaimasallthetoolsthatitisclaimingtobeaspowerfulwereallinasetofgotocommandstorunwhenlookingintoproblems,soIwasalittleskepticalatfirst.
Asanyonewhohashadtotryandtrackdownahaywireprocessoftryandtrackdownanissuethatisn’tbeingveryverboseinitserrorlogsonaLinuxserverwillknowthatusingtoolssuchasstrace,lsof,andtcpdumpcangetcomplicatedveryquicklyanditnormallyinvolvescapturingawholelotofdataandthenusingacombinationofseveraltoolstoslowly,andmanually,tracetheproblembyreducingtheamountofdatayoucaptured.
ImaginemydelightwhenSysdig’sclaimsturnedouttobetrue.ItmademewishIhadthetoolbackwhenIwasafrontlineengineer;itwouldhavemademylifealoteasier.
Sysdigcomesintwodifferentflavors,firstistheOpenSourceversionavailableathttp://www.sysdig.org/;thiscomeswithanncursesinterfacesothatyoucaneasilyaccessandquerydatafromaterminal-basedGUI.
NoteWikipediadescribesncurses(newcurses)asaprogramminglibrarythatprovidesanAPIthatallowstheprogrammertowritetext-baseduserinterfacesinaterminal-independentmanner.Itisatoolkitfordeveloping“GUI-like”applicationsoftwarethatrunsunderaterminalemulator.Italsooptimizesscreenchangesinordertoreducethelatencyexperiencedwhenusingremoteshells.
ThereisalsoacommercialservicethatallowsyoutostreamyourSysdigtotheirexternallyhostedservice;thisversionhasaweb-basedinterfaceforviewingandqueryingyourdata.
Inthischapter,wewillbeconcentratingontheopensourceversion.
InstallingSysdigConsideringhowpowerfulSysdigis,ithasoneofthemoststraightforwardinstallationandconfigurationprocessesIhavecomeacross.ToinstallSysdigoneitheraCentOSorUbuntuserver,typethefollowingcommand:
curl-shttps://s3.amazonaws.com/download.draios.com/stable/install-sysdig
|sudobash
Afterrunningtheprecedingcommand,youwillgetthefollowingoutput:
That’sit,youarereadytogo.Thereisnothingmoretoconfigureordo.Thereisamanualinstallationprocessandalsoawayofinstallingthetoolusingcontainerstobuildthenecessarykernelmodules;formoredetails,seetheinstallationguideasfollows:
http://www.sysdig.org/wiki/how-to-install-sysdig-for-linux/
UsingSysdigBeforewelookathowtouseSysdig,let’slaunchafewcontainersusingdocker-composebyrunningthefollowingcommand:
cd/monitoring_docker/chapter05/wordpress/
docker-composeup–d
ThiswilllaunchaWordPressinstallationrunningadatabaseandtwowebservercontainersthatareloadbalancedusinganHAProxycontainer.YouwillbeabletoviewtheWordPressinstallationathttp://docker.media-glass.es/oncethecontainershavelaunched.Youwillneedtoentersomedetailstocreatetheadminuserbeforethesiteisvisible;followtheon-screenpromptstocompletethesesteps.
ThebasicsAtitscore,Sysdigisatoolforproducingastreamofdata;youcanviewthestreambytypingsudosysdig(toquit,pressCtrl+c).
Thereisalotinformationtheresolet’sstarttofilterthestreamdownandrunthefollowingcommand:
sudosysdigevt.type=chdir
Thiswilldisplayonlyeventsinwhichauserchangesdirectory;toseeitinaction,openasecondterminalandyouwillseethatwhenyoulogin,youseesomeactivityinthefirstterminal.Asyoucansee,itlooksalotlikeatraditionallogfile;wecanformatoutputtogiveinformationsuchastheusername,byrunningthefollowingcommand:
sudosysdig-p"user:%user.namedir:%evt.arg.path"evt.type=chdir
Then,inyoursecondterminal,changethedirectoryafewtimes:
Asyoucansee,thisisaloteasiertoreadthantheoriginalunformattedoutput.PressCtrl+ctostopfiltering.
CapturingdataIntheprevioussection,welookedatfilteringdatainrealtime;itisalsopossibletostreamSysdigdatatoafilesothatyoucanquerythedataatalatertime.Exitfromyoursecondterminalandrunthefollowingcommandonyourfirstone:
sudosysdig-w~/monitoring-docker.scap
Whilethecommandisrunningonthefirstterminal,logintothehostonthesecondoneandchangethedirectoryafewtimes.Also,whilewearerecording,clickaroundtheWordPresssitewestartedatthebeginningofthissection,theURLishttp://docker.media-glass.es/.Onceyouhavedonethat,stoptherecordingbypressingCrtl+c;youshouldhavenowdroppedbacktoaprompt.YoucancheckthesizeofthefilecreatedbySysdigbyrunningthefollowing:
ls-lha~/monitoring-docker.scap
Now,wecanusethedatathatwehavecapturedtoapplythesamefilteraswedidwhenlookingatthereal-timestream:
sudosysdig-r~/monitoring-docker.scap-p"user:%user.name
dir:%evt.arg.path"evt.type=chdir
Byrunningtheprecedingcommand,youwillgetthefollowingoutput:
Noticehowwegetsimilarresultstowhenwewereviewingthedatainrealtime.
ContainersOneofthethingsthatwasrecordedin~/monitoring-docker.scapwasdetailsonthesystemstate;thisincludesinformationonthecontainerswelaunchedatthestartofthechapter.Let’susethisfiletogetsomestatsonthecontainers.Tolistthecontainersthatwereactiveduringthetime,wecapturedthedatafilerun:
sudosysdig-r~/monitoring-docker.scap-clscontainers
ToseewhichofthecontainersutilizedtheCPUmostofthetime,wewereclickingaroundtheWordPresssiterun:
sudosysdig-r~/monitoring-docker.scap-ctopcontainers_cpu
Tohavealookatthetopprocessesineachofthecontainersthathave“wordpress”intheirnames(whichisalloftheminourcase),runthefollowingcommand:
sudosysdig-r~/monitoring-docker.scap-ctopprocs_cpucontainer.name
containswordpress
Finally,whichofourcontainerstransferredthemostamountofdata?:
sudosysdig-r~/monitoring-docker.scap-ctopcontainers_net
Byrunningtheprecedingcommand,youwillgetthefollowingoutput:
Asyoucansee,wehaveextractedquiteabitofinformationonourcontainersfromthedatawecaptured.Also,usingthefile,youcanremovethe-r~/monitoring-docker.scappartofthecommandtoviewthecontainermetricsinrealtime.
It’salsoworthpointingoutthattherearebinariesforSysdigthatworkonbothOSXandWindows;whilethesedonotcaptureanydata,theycanbeusedtoreaddatathatyouhaverecordedonyourLinuxhost.
FurtherreadingFromthefewbasicexercisescoveredinthissection,youshouldstarttogetanideaofjusthowpowerfulSysdigcanbe.TherearemoreexamplesontheSysdigwebsiteathttp://www.sysdig.org/wiki/sysdig-examples/.Also,Irecommendyoutoreadtheblogpostathttps://sysdig.com/fishing-for-hackers/;itwasmyfirstexposuretoSysdiganditreallydemonstratesitsusefulness.
UsingCsysdigAseasyasitistoviewdatacapturedbySysdigusingthecommandlineandmanuallyfilteringtheresults,itcangetmorecomplicatedasyoustarttostringmoreandmorecommandstogether.TohelpmakethedatacapturedbySysdigasaccessibleaspossible,SysdigshipswithaGUIcalledCsysdig.
LaunchingtheCsysdigisdonewithasinglecommand:
sudocsysdig
Oncetheprocesshaslaunched,itshouldinstantlylookfamiliartoanyonewhohasusedtoporcAdvisor(minusthegraphs);itsdefaultviewwillshowyoureal-timeinformationontheprocessesthatarerunning:
Tochangethisview,knownastheProcessesview,pressF2toopentheViewsmenu;fromhere,youcanusetheupanddownarrowsonyourkeyboardtoselectaview.Asyoumayhavealreadyguessed,wewouldliketoseetheContainersview:
However,beforewedrilldownintoourcontainers,let’squitCsysdigbypressingqandloadupthefilewecreatedintheprevioussection.Todothis,typethefollowingcommand:
sudocsysdig-r~/monitoring-docker.scap
OnceCsysdigloads,youwillnoticethatSourcehaschangedfromLiveSystemtothefilepathofourdatafile.Fromhere,pressF2andusetheuparrowtoselectcontainersandthenhitEnter.Fromhere,youcanusetheupanddownarrowstoselectoneofthetwowebservers,thesewouldbeeitherwordpress_wordpress1_1orwordpress_wordpress2_1asshowninthefollowingscreen:
NoteTheremainingpartofthischapterassumesthatyouhaveCsysdigopenin-frontofyou,itwilltalkyouthroughhowtonavigatearoundthetool.Pleasefeelfreetoexploreyourselfaswell.
Onceyouhaveselectedaserver,hitEnterandyouwillbepresentedwithalistofprocessesthatthecontainerwasrunning.Again,youcanusethearrowkeystoselectaprocesstodrilldownfurtherinto.
IsuggestedlookingatoneoftheApacheprocessesthathasavaluelistedintheFilecolumn.Thistime,ratherthanpressingEntertoselecttheprocess,let’s“Echo”whattheprocesswasuptoatthetimewecapturedthedata;withtheprocessselected,pressF5.
Youcanusetheupanddownarrowstoscrollthroughtheoutput:
Tobetterformatthedata,pressF2andselectPrintableASCII.Asyoucanseefromtheprecedingscreenshot,thisApacheprocessperformedthefollowingtasks:
AcceptedanincomingconnectionAccessedthe.htaccessfileReadthemod_rewriterulesGotinformationfromthehostsfileMadeaconnectiontotheMySQLcontainerSenttheMySQLpassword
Byscrollingthroughtheremainderofthedatainthe“Echo”resultsfortheprocess,youshouldbeabletoeasilyfollowtheinteractionswiththedatabaseallthewaythroughtothepagebeingsenttothebrowser.
Toleavethe“Echo”screen,pressBackspace;thiswillalwaystakeyoualevelback.
Ifyouwantamoredetailedbreakdownonwhattheprocesswasdoing,thenpressF6toentertheDigview;thiswilllistthefilesthattheprocesswasaccessingatthetime,alongwiththenetworkinteractionandhowitisaccessingtheRAM.
Toviewafulllistofcommandsandformorehelp,youcanpressF1atanytime.Also,togetabreakdownonanycolumnsthatareonscreen,pressF7.
SummaryAsImentionedatthestartofthischapter,SysdigisprobablyoneofthemostpowerfultoolsIhavecomeacrossinrecentyears.
Partofitspoweristhewaythatitexposesalotofinformationandmetricsinawaythatneverfeelsoverwhelming.It’sclearthatthedevelopershavespentalotoftimeensuringthatboththeUIandthewaythatcommandsarestructuredfeelnaturalandinstantlyunderstandable,evenbythenewestmemberofanoperationsteam.
Theonlydownsideisthat,unlessyouwanttoviewtheinformationinrealtimeorlookintoaproblemindevelopmentstoringtheamountofdatathatisbeinggeneratedbySysdig,itcanbequitecostlyintermsofdiscspacebeingused.
ThisissomethingthatSysdighasrecognized,andtohelpwiththis,thecompanyoffersacloud-basedcommercialservicecalledSysdigCloudforyoutostreamyourSysdigdatainto.Inthenextchapter,wewilllookatthisserviceandalsosomeofitscompetitors.
Chapter6.ExploringThirdPartyOptionsSofar,wehavebeenlookingatthetoolsandservicesyouhostyourself.Alongwiththeseself-hostedtools,alargeamountofcloud-basedsoftwarehasdevelopedaroundDockerasaserviceecosystem.Inthischapter,wewilllookatthefollowingtopics:
WhyuseaSaaSserviceoverself-hostedorreal-timemetrics?Whatservicesareavailableandwhatdotheyoffer?InstallationofagentsforSysdigCloud,Datadog,andNewReliconthehostmachinesConfigurationoftheagentstoshipmetrics
AwordaboutexternallyhostedservicesSofar,toworkthroughtheexamplesinthisbook,wehaveusedlocallyhostedvirtualserversthatarelaunchedusingvagrant.Duringthischapter,wearegoingtouseservicesthatneedtobeabletocommunicatewithyourhostmachine,soratherthantryingtodothisusingyourlocalmachine,itsabouttimeyoutookyourhostmachineintothecloud.
Aswearegoingtostartandstoptheremotehostswhilewelookattheservices,itpaystouseapubliccloud,asweonlygetchargedforwhatweuse.
Thereareseveralpubliccloudservicesthatyoucanusetoevaluatethetoolscoveredinthischapter,whichoneyouchoosetouseisuptoyou,youcoulduse:
DigitalOcean:https://www.digitalocean.com/AmazonWebServices:https://aws.amazon.com/MicrosoftAzure:https://azure.microsoft.com/VMwarevCloudAir:http://vcloud.vmware.com/
Oruseyourownpreferredprovider,theonlypre-requisiteisthatyourserverispublicallyaccessible.
ThischapterassumesthatyouarecapableoflaunchingeitheraCentOS7orUbuntu14.04cloudinstanceandyouunderstandthatyouwilllikelyincurchargeswhilethecloudinstanceisupandrunning.
DeployingDockerinthecloudOnceyouhavelaunchedyourcloudinstance,youcanbootstrapDockerinthesamewaythatyouinstalledusingvagrant.Inthechapter6folderoftheGitrepository,therearetwoseparatescriptstodownloadandinstalltheDockerengineandcomposeitonyourcloudinstance.
ToinstallDocker,ensurethatyourcloudinstanceisupdatedbyrunning:
sudoyumupdate
FortheCentOSinstanceofyourUbuntu,runthefollowingcommand:
sudoapt-getupdate
Onceupdated,runthefollowingcommandtoinstallthesoftware.Duetothedifferencesinthewaydifferentcloudenvironmentsareconfigured,itisbesttoswitchovertotherootusertoruntheremainderofthecommands,todothis,run:
sudosu-
Nowyouwillbeabletoruntheinstallscriptusingthefollowingcommand:
curl-fsShttps://raw.githubusercontent.com/russmckendrick/monitoring-
docker/master/chapter06/install_docker/install_docker.sh|bash
Tocheckthateverythingworksasexpected,runthefollowingcommand:
dockerrunhello-world
Youshouldseesomethingsimilartotheterminaloutput,asshowninthefollowingscreenshot:
WhyuseaSaaSservice?Youmayhavenoticedwhileworkingwiththeexamplesinthepreviouschaptersthatthetoolswehaveusedcanpotentiallyusemanyresourcesifweneededtostartcollectingmoremetrics,especiallyiftheapplicationswewanttomonitorareinproduction.
TohelpshiftthisloadfrombothstorageandCPU,anumberofcloud-basedSaaSoptionshavestartedofferingsupporttorecordmetricsforyourcontainers.Manyoftheseserviceswerealreadyofferingservicestomonitorservers,soaddingsupportforcontainersseemedanaturalprogressionforthem.
Thesetypicallyrequireyoutoinstallanagentonyourhostmachine,onceinstalled,theagentwillsitinthebackgroundandreporttotheservices,normallycloud-basedandAPIservices.
AfewoftheservicesallowyoutodeploytheagentsasDockercontainers.Theyoffercontainerizedagentssothattheservicecanrunonstrippeddownoperatingsystems,suchas:
CoreOS:https://coreos.com/RancherOS:http://rancher.com/rancher-os/Atomic:http://www.projectatomic.io/UbuntuSnappyCore:https://developer.ubuntu.com/en/snappy/
Theseoperatingsystemsdifferfromtraditionalones,asyoucannotinstallservicesonthemdirectly;theironlypurposeistorunaservice,suchasDocker,sothatyoucanlaunchtheservicesorapplicationsyouneedtoberunascontainers.
Aswearerunningfulloperatingsystemsasourhostsystems,wedonotneedthisoptionandwillbedeployingtheagentsdirectlytothehosts.
TheSaaSoptionsthatwearegoingtolookatinthischapterareasfollows:
SysdigCloud:https://sysdig.com/product/Datadog:https://www.datadoghq.com/NewRelic:http://newrelic.com
Theyallofferfreetrialsandtwoofthemofferfreecut-downversionsofthemainservice.Onthefaceofit,theymightallappeartooffersimilarservices;however,whenyoustarttousethem,youwillimmediatelynoticethattheyareinfactallverydifferentfromeachother.
SysdigCloudInthepreviouschapter,wehadalookattheopensourceversionofSysdig.WesawthatthereisagreatncursesinterfacecalledcSysdiganditallowsustonavigatethroughallthedatathatSysdigiscollectingaboutourhost.
ThesheeramountofmetricsanddatacollectedbySysdigmeansthatyouhavetotrytostayontopofiteitherbyshippingyourfilesofftheserver,maybetoAmazonSimpleStorageService(S3),ortosomelocalsharedstorage.Inaddition,youcanquerythedatainthecommandlineonthehostitselforonyourlocalmachineusinganinstallationofthecommand-linetools.
ThisiswhereSysdigCloudcomesintoplay;itoffersaweb-basedinterfacetothemetricsthatSysdigcapturesalongwiththeoptionstoshiptheSysdigcapturesoffyourhostmachineeithertoSysdig’sownstorageortoyourS3bucket.
Sysdigcloudoffersthefollowingfunctionality:
ContainerVision™Real-TimeDashboardHistoricalReplayDynamicTopologyAlerting
Aswellas,theoptiontotriggeracaptureonanyofyourhostsandatanytime.
SysdigdescribesContainerVisionas:
“SysdigCloud’spatent-pendingcoretechnology,ContainerVision,istheonlymonitoringtechnologyonthemarketdesignedspecificallytorespecttheuniquecharacteristicsofcontainers.ContainerVisionoffersyoudeepandcomprehensivevisibilityintoallaspectsofyourcontainerizedenvironment-applications,infrastructures,servers,andnetworks-allwithouttheneedtopolluteyourcontainerswithanyextrainstrumentation.Inotherwords,ContainerVisiongivesyou100%visibilityintotheactivityinsideyourcontainers,fromtheoutside.”
BeforewedelveintoSysdigCloudanyfurther,Ishouldpointoutthatthisisacommercialserverandatthetimeofwriting,itcosts$25perhostpermonth.Thereisalsoa14-dayfullyfeaturedtrialavailable.Ifyouwishtoworkthroughtheagentinstallationandfollowtheexampleinthischapter,youwillneedanactiveaccountthatrunseitheronthe14-daytrialorapaidsubscription.
Signupfora14-dayfreetrial:https://sysdig.com/Detailsonpricing:https://sysdig.com/pricing/Introductiontothecompany:https://sysdig.com/company/
InstallingtheagentTheagentinstallationissimilartoinstallingtheopensourceversion;youneedtoensurethatyourcloudhostisrunninganup-to-datekernelandthatyouarealsobootedintothekernel.
Somecloudproviderskeepatightcontrolonthekernelsyoucanbootinto(forexample,DigitalOcean),andtheydonotallowyoutomanageyourkernelonthehostitself.Instead,youneedtochoosethecorrectversionthroughtheircontrolpanel.
Onceyouhavethecorrectkernelinstalled,youshouldbeabletorunthefollowingcommandtoinstalltheagent.Ensurethatyoureplacetheaccesskeyattheendofthecommandwithyourownaccesskey,whichcanbefoundonyourUserProfilepageorontheagentinstallationpages;youcanfindtheseat:
UserProfile:https://app.sysdigcloud.com/#/settings/userAgentInstallation:https://app.sysdigcloud.com/#/settings/agentInstallation
Thecommandtorunis:
curl-shttps://s3.amazonaws.com/download.draios.com/stable/install-agent|
sudobash-s—--access_keywn5AYlhjRhgn3shcjW14y3yOT09WsF7d
Theshelloutputshouldlooklikethefollowingscreen:
Oncetheagenthasbeeninstalled,itwillimmediatelystarttoreportthedatabacktoSysdigCloud.IfyouclickonExplore,youwillseeyourhostmachineandtherunning
containers:
Asyoucanseehere,IhavemyhostmachineandfourcontainersrunningaWordPressinstallationsimilartotheoneweusedinthepreviouschapter.Fromhere,wecanstarttodrilldownintoourmetrics.
TolaunchtheWordPressinstallationonyourcloud-basedmachine,runthefollowingcommandsastherootuser:
sudosu-
mkdir~/wordpress
curl-Lhttps://raw.githubusercontent.com/russmckendrick/monitoring-
docker/master/chapter05/wordpress/docker-compose.yml>~/wordpress/docker-
compose.yml
cd~/wordpress
docker-composeup-d
ExploringyourcontainersTheSysdigCloudwebinterfacewillfeelinstantlyfamiliar,asitsharesasimilardesignandoverallfeelingwithcSysdig:
Onceyoustarttodrilldown,youcanseethatabottompaneopensupandthisiswhereyoucanviewthestatistics.OneofthethingsIlikedaboutSysdigCloudisthatitopensupawealthofmetricsandthereshouldbeverylittlethatyouneedtoconfigurefromhere.
Forexample,ifyouwanttoknowwhatprocesseshavebeenconsumingthemostCPUtimeinthelast2hours,clickon2HinthesecondarymenuandthenfromtheViewstabinthebottom-leftclickonSystem:TopProcesses;thiswillgiveyouatableoftheprocesses,orderedbytheonesthathaveusedthemosttime.
Toapplythisviewtoacontainer,clickonacontainerinthetop-sectionandthebottom-sectionwillbeinstantlyupdatedtoreflectthetopCPUutilizationforjustthatcontainer;asmostcontainerswillonlyrunoneortwoprocesses,thismaynotbethatinteresting.So,let’shaveadeeplookattheprocessesthemselves.Let’ssay,weclickedonourdatabase
containerandwewantedinformationonwhatishappeningwithinMySQL.
SysdigCloudcomeswithapplicationoverlays,thesewhenselectedgiveyoumoregranularinformationontheprocesseswithinthecontainer.SelectingtheApp:MySQL/PostgreSQLviewgivesyouaninsightintowhatyourMySQLprocessesarecurrentlydoing:
Here,youcanseethatviewinthebottomsectionhasinstantlyupdatedtogiveawealthofinformationonwhathasbeenhappeninginthelast5minuteswithinMySQL.
SysdigCloudsupportsanumberofapplicationviews,including:
ApacheHAProxy
NGINXRabbitMQRedisTomcat
Eachonegivesyouimmediateaccesstometrics,whicheventhemostexperiencedSysAdminswillfindvaluable.
Youmayhavenoticedthatatthetopofthesecondpaneltherearealsoafewicons,theseallowyouto:
AddAlert:Createsanalertbasedontheviewyouhaveopen;itletsyoutweakthethresholdandalsochoosehowyouarenotified.SysdigCapture:Pressingthisbringsupadialog,whichletsyourecordaSysdigsession.Oncerecorded,thesessionistransferredtoSysdigCloudoryourownS3bucket.Oncethesessionisavailable,youdownloaditorexploreitwithinthewebinterface.SSHConnect:GetsaremoteshellontheserverfromtheSysdigCloudwebinterface;itisusefulifyoudonothaveimmediateaccesstoyourlaptopordesktopmachineandyouwanttodosometroubleshooting.Pintodashboard:Addsthecurrentviewtoacustomdashboard.
Outtheseoptionsicons,the“AddAlert”and“SysdigCapture”optionsareprobablytheonesthatyouwillendupusingthemost.OnefinalviewthatIfoundinteresting,isthetopologyone.Itgivesyouabird’seyeviewofyourhostandcontainers,thisisusefultooseetheinteractionbetweencontainersandhosts:
Here,youcanseemerequestapagefromtheWordPresssite(it’sintheboxontheleft),thisrequesthitsmyhostmachine(theboxontheright).Onceit’sonthehostmachine,itisroutedtotheHAProxycontainer,whichthenpassesthepagerequesttotheWordpress2container.Fromhere,theWordpress2containerinteractswiththedatabasethatisrunningontheMySQLcontainer.
SummaryandfurtherreadingAlthoughSysdigCloudisquiteanewservice,itfeelsinstantlyfamiliarandfullyfeaturedasitisbuiltontopofanalreadyestablishedandrespectedopensourcetechnology.Ifyoulike,thelevelofdetailyougetfromtheopensourceversionofSysdig,thenSysdigCloudisanaturalprogressionforyoutostartstoringyourmetricsoffsiteandalsotoconfigurealerts.SomegoodstartingpointsforlearningmoreaboutSysdigCloudare:
VideoIntroduction:https://www.youtube.com/watch?v=p8UVbpw8n24SysdigCloudBestPractices:http://support.sysdigcloud.com/hc/en-us/articles/204872795-Best-PracticesDashboards:http://support.sysdigcloud.com/hc/en-us/articles/204863385-DashboardsSysdigblog:https://sysdig.com/blog/
TipIfyouhavelaunchedacloudinstanceandarenolongerusingit,nowwouldbeagoodtimetopowertheinstancedownorterminateitaltogether.Thiswillensurethatyoudonotgetbilledforservicesthatyouarenotusing.
DatadogDatadogisafullmonitoringplatform;itsupportsvariousservers,platforms,andapplications.Wikipediadescribestheserviceas:
“DatadogisaSaaS-basedmonitoringandanalyticsplatformforITinfrastructure,operationsanddevelopmentteams.Itbringstogetherdatafromservers,databases,applications,toolsandservicestopresentaunifiedviewoftheapplicationsthatrunatscaleinthecloud.”
Itusesanagentthatisinstalledonyourhostmachine;thisagentsendsmetricsbacktotheDatadogserviceperiodically.Italsosupportmultiplecloudplatforms,suchasAmazonWebServices,MicrosoftAzure,andOpenStacktonameafew.
Theaimistobringallofyourservers,applications,andhostprovidermetricsintoasinglepaneofglass;fromhere,youcancreatecustomdashboardsandalertssothatyoucanbenotifiedofanyproblematanylevelwithinyourinfrastructure.
Youcansignupforafreetrialofthefullserviceathttps://app.datadoghq.com/signup.Youwillneedatleastatrialaccounttoconfigurethealtering,andifyourtrialhasalreadyexpiredtheliteaccountwilldo.FormoredetailonDatadog’spricingstructure,pleaseseehttps://www.datadoghq.com/pricing/.
InstallingtheagentTheagentcanbeinstalledeitherdirectlyonthehostmachineorasacontainer.Toinstalldirectlyonthehostmachine,runthefollowingcommandandmakesurethatyouuseyourownuniqueDD_API_KEY:
DD_API_KEY=wn5AYlhjRhgn3shcjW14y3yOT09WsF7dbash-c"$(curl-L
https://raw.githubusercontent.com/DataDog/dd-
agent/master/packaging/datadog-agent/source/install_agent.sh)"
Toruntheagentasacontainer,usethefollowingcommandandagainmakesurethatyouuseyourownDD_API_KEY:
sudodockerrun-d--namedd-agent-h`hostname`-v
/var/run/docker.sock:/var/run/docker.sock-v
/proc/mounts:/host/proc/mounts:ro-v/sys/fs/cgroup/:/host/sys/fs/cgroup:ro
-eAPI_KEY=wn5AYlhjRhgn3shcjW14y3yOT09WsF7ddatadog/docker-dd-agent
Oncetheagenthasbeeninstalled,itwillcallbacktoDatadogandthehostwillappearinyouraccount.
IftheagenthasbeeninstalleddirectlyonthehostmachinethenwewillneedtoenabletheDockerintegration,ifyouinstalledtheagentusingthecontainerthenthiswillhavebeendoneforyouautomatically.
Todothis,youfirstneedtoallowtheDatadogagentaccesstoyourDockerinstallationbyaddingthedd-agentusertotheDockergroupbyrunningthefollowingcommand:
usermod-a-Gdockerdd-agent
Thenextstepistocreatethedocker.yamlconfigurationfile,luckilytheDatadogagentshipswithanexampleconfigurationfilethatwecanuse;copythisinplaceandthenrestarttheagent:
cp-pr/etc/dd-agent/conf.d/docker.yaml.example/etc/dd-
agent/conf.d/docker.yaml
sudo/etc/init.d/datadog-agentrestart
Nowtheagentonourhostmachinehasbeenconfiguredandthefinalstepistoenabletheintegrationthroughthewebsite.Todothis,gotohttps://app.datadoghq.com/andclickonIntegrations,scrolldownandthenclickoninstallonDocker:
Onceyouclickinstall,youwillbepresentedwithanoverviewoftheintegration,clickontheConfigurationtab,thisgivesinstructionsonhowtoconfiguretheagent;aswehavealreadydonethisstep,youcanclickonInstallIntegration.
YoucanfindmoreinformationoninstallingtheagentandtheintegrationsatthefollowingURLs:
https://app.datadoghq.com/account/settings#agenthttps://app.datadoghq.com/account/settings#integrations
ExploringthewebinterfaceNow,youhaveinstalledtheagentandenabledtheDockerintegration,youcanstarttohavealookaroundthewebinterface.Tofindyourhost,clickon“Infrastructure”intheleft-handsidemenu.
Youshouldbetakentoascreenthatcontainsamapofyourinfrastructure.Likeme,youprobablyonlyhaveasinglehostmachinelisted,clickonitandsomebasicstatsshouldappearatthebottomofthescreen:
Ifyoudon’talreadyhavethecontainerslaunched,nowwouldbeagoodtimetodoso,letslaunchtheWordPressinstallationagainusing:
sudosu-
mkdir~/wordpress
curl-Lhttps://raw.githubusercontent.com/russmckendrick/monitoring-
docker/master/chapter05/wordpress/docker-compose.yml>~/wordpress/docker-
compose.yml
cd~/wordpress
docker-composeup-d
Now,gobacktothewebinterface,fromthereyoucanclickonanyoftheserviceslistedonthehexagon.Thiswillbringupsomebasicmetricsfortheserviceyouhaveselected.Ifyouclickondocker,youwillseealinkforaDockerDashboardamongthevariousgraphsandsoon;clickingthiswilltakeyoutoamoredetailedviewofyourcontainers:
Asyoucansee,thisgivesusournowfamiliarbreakdownoftheCPUandmemorymetrics,alongwithinthetoprightofthedashboardabreakdownofthecontaineractivityonthehostmachine;thislogsevents,suchasstoppingandstartingcontainers.
Datadogcurrentlyrecordsthefollowingmetrics:
docker.containers.running
docker.containers.stopped
docker.cpu.system
docker.cpu.user
docker.images.available
docker.images.intermediate
docker.mem.cache
docker.mem.rss
docker.mem.swap
FromtheMetricsexploreroptionintheleft-handsidemenu,youcanstarttographthesemetricsandonceyouhavethegraphs,youcanthenstarttoaddthemtoyourowncustomdashboardsorevenannotatethem.Whenyouannotateagraph,asnapshotiscreatedandthegraphshowsupintheeventsqueuealongwiththeotherevents,thathavebeenrecorded,suchascontainerstoppingandstarting:
Also,withinthewebinterfaceyoucanconfiguremonitors;theseallowyoutodefinetriggers,whichalertyouifyourconditionsarenotmet.Alertscanbesentase-mailsorviathirdpartyservices,suchasSlack,Campfire,orPagerDuty.
SummaryandfurtherreadingWhileDatadog’sDockerintegrationonlygivesyouthebasicmetricsonyourcontainers,itdoeshaveawealthoffeaturesandintegrationwithotherapplicationsandthirdparties.IfyouneedtomonitoranumberofdifferentservicesalongsideyourDockercontainers,thenthisservicecouldbeforyou:
Homepage:https://www.datadoghq.comOverview:https://www.datadoghq.com/product/MonitoringDockerwithDatadog:https://www.datadoghq.com/blog/monitor-docker-datadog/Twitter:https://twitter.com/datadoghq
TipPleaseRemember
Ifyouhavelaunchedacloudinstanceandarenolongerusingitthennowwouldbeagoodtimetopowertheinstancedownorterminateitaltogether.Thiswillensurethatyoudonotgetbilledforanyservicesyouarenotusing.
NewRelicNewReliccouldbeconsideredthegranddaddyofSaaSmonitoringtools,chancesarethatifyouareadeveloperyouwillhaveheardofNewRelic.IthasbeenaroundforawhileanditisthestandardtowhichotherSaaStoolscomparethemselves.
NewRelichasgrownintoseveralproductsovertheyear,currently,theyoffer:
NewRelicAPM:Themainapplicationperformance-monitoringtool.ThisiswhatmostpeoplewillknowNewRelicfor;thistollgivesyouthecodelevelvisibilityofyourapplication.NewRelicMobile:Asetoflibrariestoembedintoyournativemobileapps,givingAPMlevelsofdetailforyouriOSandandroidapplication.NewRelicInsights:Ahigh-levelviewofallofthemetricscollectedbyotherNewRelicservices.NewRelicServers:Monitorsyourhostservers,recordingmetricsaroundCPU,RAM,andstorageutilization.NewRelicBrowser:Givesyouaninsightintowhathappenswithyourweb-basedapplicationsoncetheyleaveyourserversandenteryourenduser’sbrowserNewRelicSynthetics:Monitorsyourapplicationsresponsivenessfromvariouslocationsaroundtheworld.
RatherthanlookingatalloftheseofferingsthatgiveusaninsightintowhatishappeningwithourDocker-basedcode,asthat’sprobablyawholebookonitsown,wearegoingtotakealookattheserverproduct.
TheservermonitoringserviceofferedbyNewRelicisavailablefreeofcharge,youjustneedanactiveNewRelicaccount,youcansignupforanaccountathttps://newrelic.com/signup/detailsonNewRelicspricingcanbefoundattheirhomepageathttp://newrelic.com/.
InstallingtheagentLiketheotherSaaSofferingswehavelookedatinthischapter,NewRelicServershasahost-basedclient,whichneedstobeabletoaccesstheDockerbinary.ToinstallthisonaCentOSmachine,runthefollowing:
yuminstallhttp://download.newrelic.com/pub/newrelic/el5/i386/newrelic-
repo-5-3.noarch.rpm
yuminstallnewrelic-sysmond
ForUbuntu,runthefollowingcommand:
echo'debhttp://apt.newrelic.com/debian/newrelicnon-free'|sudotee
/etc/apt/sources.list.d/newrelic.list
wget-O-https://download.newrelic.com/548C16BF.gpg|sudoapt-keyadd-
apt-getupdate
apt-getinstallnewrelic-sysmond
Nowthatyouhavetheagentinstalled,youneedtoconfiguretheagentwithyourlicensekey.Youcandothiswiththefollowingcommandandmakesurethatyouaddyourlicense,whichcanbefoundinyoursettingspage:
nrsysmond-config--setlicense_key=wn5AYlhjRhgn3shcjW14y3yOT09WsF7d
Nowthattheagentisconfigured,weneedtoaddthenewrelicusertothedockergroupsothattheagenthasaccesstoourcontainerinformation:
usermod-a-Gdockernewrelic
Finally,weneedtostarttheNewRelicServeragentandrestartDocker:
/etc/init.d/newrelic-sysmondrestart
/etc/init.d/dockerrestart
TipRestartingDockerwillstoptherunningcontainersthatyouhave;makesurethatyoumakeanoteoftheseusingdockerpsandthenstartthemmanuallyandbackuponcetheDockerservicerestarts.
YoushouldseeyourserverappearonyourNewReliccontrolpanelafterafewminutes.
ExploringthewebinterfaceOnceyouhavetheNewRelicserveragentinstalled,configured,andrunningonyourhostmachine,youwillseesomethingsimilartothefollowingscreenshotwhenclickingonServersinthetopmenu:
Selectingtheserverwillallowyoutostartexploringthevariousmetricsthattheagentisrecording:
Fromhere,youhavetheoptiontodrilldownfurther:
Overview:GivesaquickoverviewofyourhostmachineProcesses:ListsalloftheprocessesthatarerunningbothonthehostmachineandwithinyourcontainersNetwork:LetsyouseethenetworkactivityforyourhostmachineDisks:GivesyoudetailsonhowmuchspaceyouareusingDocker:ShowsyoutheCPUandmemoryutilizationforyourcontainers
Asyoumayhaveguessed,wearegoingtobelookingattheDockeritemnext,clickonitandyouwillseealistofyouractiveimages:
YoumayhavenoticedadifferencebetweenNewRelicandtheotherservices,asyoucanseeNewRelicdoesnotshowyoutherunningcontainers,insteaditshowsyoutheutilizationbyDockerimage.
Intheprecedingscreenshot,IhavefourcontainersactiveandrunningtheWordPressinstallationwehaveusedelsewhereinthebook.IfIwantedabreakdownpercontainer,thenIwouldbeoutofluck,asdemonstratedbythefollowingscreen:
It’saprettydullscreen,butitgivesyouanideaaboutwhatyouwillseeifyouarerunningmultiplecontainersthathavebeenlaunchedusingthesameimage.Sohowisthisuseful?Well,coupledwiththeotherservicesofferedbyNewRelic,itcangiveyouanindicationofwhatyourcontainerswereuptowhenaproblemoccurredwithinyourapplication.IfyourememberthePetsversusCattleversusChickensanalogyfromChapter1,IntroductiontoDockerMonitoring,wedon’tnecessarilycarewhichcontainerdidwhat;wejustwanttoseetheimpactithadduringtheissuewearelookinginto.
SummaryandfurtherreadingDuetotheamountofproductsitoffers,NewReliccanbealittledauntingatfirst,butifyouworkwithadevelopmentteamthatactivelyusesNewRelicwithintheirday-to-dayworkflow,thenhavingalloftheinformationaboutyourinfrastructurealongsidethisdatacanbebothvaluableandnecessary,especiallyduringanissue:
NewRelicServermonitoring:http://newrelic.com/server-monitoringNewRelicandDocker:http://newrelic.com/docker/Twitter:https://twitter.com/NewRelic
TipIfyouhavelaunchedacloudinstanceandarenolongerusingitthen,nowisagoodtimetopowertheinstancedownorterminateitaltogether,thiswillensureyoudonotgetbilledforanyservicesyouarenotusing.
SummaryWhichSaaSserviceyouchoosedependsonyourcircumstances,thereareanumberofquestionsyoushouldaskyourselfbeforeyoustartevaluatingtheSaaSofferings:
Howmanycontainerswouldyouliketomonitor?Howmanyhostmachinesdoyouhave?Isthereanon-containerizedinfrastructureyouneedtomonitor?Whatmetricsdoyouneedfromthemonitoringservice?Howlongshouldthedataberetainedfor?Couldotherdepartments,suchasdevelopmentandutilizetheservice?
WecoveredjustthreeoftheavailableSaaSoptionsinthischapter,thereareotheroptionsavailable,suchas:
Ruxit:https://ruxit.com/docker-monitoring/Scout:https://scoutapp.com/plugin_urls/19761-docker-monitorLogentries:https://logentries.com/insights/server-monitoring/Sematext:http://sematext.com/spm/integrations/docker-monitoring.html
Monitoringserversandservicesareonlyasgoodasthemetricsyoucollect,ifpossibleandifyourbudgetallows,youshouldtakefulladvantageoftheservicesofferedbyyourchosenproviders,asmoredatabeingrecordedbyasingleproviderwillonlybenefityouwhenitcomestoanalyzingproblemswithnotonlyyourcontainerizedapplications,butalsowithyourinfrastructure,codeandevenyourcloudprovider.
Forexample,ifyouaremonitoringyourhostmachineusingthesameserviceasyouusetomonitoryourcontainers,thenbyusingthecustomgraphingfunctions,youshouldbeabletocreateoverlaygraphsofCPUloadspikesofbothyourhostmachineandyourcontainer.Thisisalotmoreusefulthantryingtocomparetwodifferentgraphsfromdifferentsystemssidebyside.
Inthenextchapter,wewilllookatanoften-overlookedpartofmonitoring:shippingyourlogfilesawayfromyourcontainers/hoststoasinglelocationsothattheycanbemonitoredandreviewed.
Chapter7.CollectingApplicationLogsfromwithintheContainerOneofthemostoverlookedpartsofmonitoringarelogfilesgeneratedbytheapplicationorservicessuchasNGINX,MySQL,Apache,andsoon.SofarwehavelookedatvariouswaysofrecordingtheCPUandRAMutilizationoftheprocesseswithinyourcontainersareatapointintime,nowitstimetodothesameforthelogfiles.
IfyouarerunningyourcontainersasCattleorChickens,thenthewayyoudealwiththeissuestodestroyandrelaunchyourcontainereithermanuallyorautomaticallyisimportant.Whilethisshouldfixtheimmediateproblem,itdoesnothelpwithtrackingdowntherootcauseoftheissueandifyoudon’tknowthatthenhowcanyouattempttoresolveitsothatitdoesnotreoccur.
Inthischapter,wewilllookathowwecangetthecontentofthelogfilesfortheapplicationsrunningwithinourcontainerstothecentrallocationsothattheyareavailable,evenifyouhavetodestroyandreplaceacontainer.Wearegoingtocoverthefollowingtopicsinthischapter:
Howtoviewcontainerlogs?Deployingan“ELK”stackusingaDockercontainersstacktoshipthelogstoReviewingyourlogsWhatthirdpartyoptionsareavailable?
ViewingcontainerlogsLikethedockertopcommand,thereisaverybasicwayofviewinglogs.Whenyouusethedockerlogscommand,youareactuallyviewingtheSTDOUTandSTDERRoftheprocessesthatarerunningwithinthecontainer.
NoteFormoreinformationonStandardStreams,pleaseseehttps://en.wikipedia.org/wiki/Standard_streams.
Asyoucanseefromthefollowingscreenshot,thesimplestthingyouhavetodoisrundockerlogsfollowedbyyourcontainername:
Toseethisonyourownhost,let’slaunchtheWordPressinstallationfromchapter05usingthefollowingcommands:
cd/monitoring_docker/chapter05/wordpress/
docker-composeup–d
dockerlogswordpress_wordpress1_1
Youcanextendthedockerlogscommandbyaddingthefollowingflagsbeforeyourcontainername:
-for--followwillstreamthelogsinrealtime-tor--timestampswillshowatimestampatthestartofeachline--tail="5"willshowthelastxnumberoflines--since="5m00s"willshowonlytheentriesforthelast5minutes
UsingtheWordPressinstallationthatwehavejustlaunched,tryrunningthefollowingcommands:
dockerlogs--tail="2"wordpress_wordpress1_1
Thiswillshowthelasttwolinesofthelogs,youcanaddtimestampsusing:
dockerlogs--tail="2"–timestampswordpress_wordpress1_1
Asyoucanseeinthefollowingterminaloutput,youcanalsostringcommandstogethertoformaverybasicquerylanguage:
Thedownsideofusingdockerlogsisexactlythesameasusingdockertop,inthatitisonlyavailablelocallyandthelogsareonlypresentforthetimethecontainerisaround,youcanviewthelogsofastoppedcontainer,butoncethecontainerisremoved,soarethelogs.
ELKStackSimilartosomeofthetechnologiesthatwehavecoveredinthisbook,anELKstackreallydeservesabookbyitself;infact,therearebooksforeachoftheelementsthatmakeanELKstack,theseelementsare:
Elasticsearchisapowerfulsearchserver,whichhasbeendevelopedwithmodernworkloadsinmindLogstashsitsbetweenyourdatasourceandElasticsearchservices;ittransformsyourdatainrealtimetoaformat,whichElasticsearchcanunderstand.KibanaisinfrontofyourElasticsearchservicesandallowsyoutoqueryyourdatainafeature-richweb-baseddashboard.
TherearealotofmovingpartswithanELKstack,sotosimplifythings,wewilluseaprebuiltstackforthepurposeoftesting;however,youprobablydon’twanttousethisstackinproduction.
StartingthestackLet’slaunchafreshvagranthostonwhichtoruntheELKstack:
[russ@mac~]$cd~/Documents/Projects/monitoring-docker/vagrant-centos/
[russ@mac~]$vagrantup
Bringingmachine'default'upwith'virtualbox'provider…
==>default:Importingbasebox'russmckendrick/centos71'...
==>default:MatchingMACaddressforNATnetworking…
==>default:Checkingifbox'russmckendrick/centos71'isuptodate…
.....
==>default:=>Installingdocker-engine…
==>default:=>Configuringvagrantuser…
==>default:=>Startingdocker-engine…
==>default:=>Installingdocker-compose…
==>default:=>FinishedinstallationofDocker
[russ@mac~]$vagrantssh
Now,wehaveacleanhostthatisupandrunning,wecanstartthestackbyrunningthefollowingcommands:
[vagrant@docker~]$cd/monitoring_docker/chapter07/elk/
[vagrant@dockerelk]$docker-composeup-d
Asyoumayhavenoticed,itdidmorethatjustpulldownsomeimages;whathappenedwas:
AnElasticsearchcontainerwaslaunchedusingtheofficialimagefromhttps://hub.docker.com/_/elasticsearch/.ALogstashcontainerwaslaunchedusingtheofficialimagefromhttps://hub.docker.com/_/logstash/,itwasalsolaunchedwithourownconfiguration,whichmeansthatourinstallationlistensforlogssentfromLogspout(moreaboutthatinaminute).AcustomKibanaimagewasbuiltusingtheofficialimagefromhttps://hub.docker.com/_/kibana/.AllitdidwasaddasmallscripttoensurethatKibanadoesn’tstartuntilourElasticsearchcontainerisfullyupandrunning.Itwasthenlaunchedwithacustomconfigurationfile.AcustomLogspoutcontainerwasbuiltusingtheofficialimagefromhttps://hub.docker.com/r/gliderlabs/logspout/andthenweaddedacustommodulesothatLogspoutcouldtalktoLogstash.
Oncedocker-composehasfinishedbuildingandlaunchingthestackyoushouldbeabletoseethefollowingwhenrunningdocker-composeps:
WenowhaveourELKstackupandrunning,asyoumayhavenoticed,thereisanadditionalcontainerrunningandgivingusanELK-Lstack,sowhatisLogspout?
LogspoutIfweweretolaunchElasticsearch,Logstash,andKibanacontainers,weshouldhaveafunctioningELKstackbutwewillhavealotofconfigurationtodotogetourcontainerlogsintoElasticsearch.
SinceDocker1.6,youhavebeenabletoconfigureloggingdrivers,thismeantthatitispossibletolaunchacontainerandhaveitsenditsSTDOUTandSTDERRtoaSyslogServer,whichwillbeLogstashinourcase;however,thismeansthatyouwillhavetoaddsomethingsimilartothefollowingoptionseachtimewelaunchacontainer:
--log-driver=syslog--log-optsyslog-address=tcp://elk_logstash_1:5000
ThisiswhereLogspoutcomesin,ithasbeendesignedtocollectalloftheSTDOUTandSTDERRmessagesonahostmachinebyinterceptingthemessagesthatarebeingcollectedbytheDockerprocessandthenitroutesthemtoourLogstashinstanceinaformatthatisunderstoodbyElasticsearch.
Justasthelog-driver,itsupportsSyslogoutofthebox;however,thereisathirdpartymodulethattransformstheoutputtoJSON,whichLogstashunderstands.Asapartofourbuildwedownloaded,compiledandconfiguredthemodule.
YoucanfindoutmoreaboutLogspoutandloggingdriversatthefollowing:
OfficialLogspoutimage:https://hub.docker.com/r/gliderlabs/logspout/LogspoutProjectpage:https://github.com/gliderlabs/logspoutLogspoutLogstashmodule:https://github.com/looplab/logspout-logstashDocker1.6releasenotes:https://blog.docker.com/2015/04/docker-release-1-6/DockerLoggingDrivers:https://docs.docker.com/reference/logging/overview/
ReviewingthelogsSonow,wehaveourELKrunningandamechanisminplacetostreamalloftheSTDOUTandSTDERRmessagesgeneratedbyourcontainersintoLogstash,whichinturnroutesthedataintoElasticsearch.NowitstimetoviewthelogsinKibana.ToaccessKibanagotohttp://192.168.33.10:8080/inyourbrowser;whenyouaccessthepage,youwillbeaskedtoConfigureanindexpattern,thedefaultindexpatternwillbefineforourneedssojustclicktheCreatebutton.
Onceyoudo,youwillseealistoftheindexpatterns,thesearetakendirectlyfromtheLogspoutoutput,andyoushouldnoticethefollowingitemsintheindex:
docker.name:Thenameofcontainerdocker.id:ThefullcontainerIDdocker.image:Thenameoftheimageusedtolaunchtheimage
Fromhere,ifyouweretoclickonDiscoverinthetopmenuyouwouldseesomethingsimilartothefollowingpage:
Inthescreenshot,youwillseethatIhaverecentlylaunchedtheWordPressstackandwehavebeenusingitthroughoutthebook,usingthefollowingcommands:
[vagrant@dockerelk]$cd/monitoring_docker/chapter05/wordpress/
[vagrant@dockerwordpress]$docker-composeup–d
Togiveyouanideaofwhatisbeinglogged,hereistherawJSONtakenfromElasticseachforrunningtheWordPressinstallationscript:
{
"_index":"logstash-2015.10.11",
"_type":"logs",
"_id":"AVBW8ewRnBVdqUV1XVOj",
"_score":null,
"_source":{
"message":"172.17.0.11--[11/Oct/2015:12:48:26+0000]\"POST/wp-
admin/install.php?step=1HTTP/1.1\"2002472\"http://192.168.33.10/wp-
admin/install.php\"\"Mozilla/5.0(Macintosh;IntelMacOSX10_11)
AppleWebKit/601.1.56(KHTML,likeGecko)Version/9.0Safari/601.1.56\"",
"docker.name":"/wordpress_wordpress1_1",
"docker.id":
"0ba42876867f738b9da0b9e3adbb1f0f8044b7385ce9b3a8a3b9ec60d9f5436c",
"docker.image":"wordpress",
"docker.hostname":"0ba42876867f",
"@version":"1",
"@timestamp":"2015-10-11T12:48:26.641Z",
"host":"172.17.0.4"
},
"fields":{
"@timestamp":[
1444567706641
]
},
"sort":[
1444567706641
]
}
Fromhere,youcanstarttousethefreetextsearchboxandbuildupsomequitecomplexqueriestodrilldownintoyourcontainer’sSTDOUTandSTDERRlogs.
Whataboutproduction?Asmentionedatthetopofthissection,youprobablydon’twanttorunyourproductionELKstackusingthedocker-composefile,whichaccompaniesthischapter.Firstofall,youwillwantyourElasticsearchdatatobestoredonapersistentvolumeandyoumorethanlikelywantyourLogstashservicetobehighlyavailable.
TherearenumerousguidesonhowtoconfigureahighlyavailableELKstack,aswellas,thehostedservicesfromElastic,whichisthecreatorofElasticsearch,andalsoAmazonWebServices,whichoffersanElasticsearchservice:
ELKtutorial:https://www.youtube.com/watch?v=ge8uHdmtb1MFoundfromElastic:https://www.elastic.co/foundAmazonElasticsearchService:https://aws.amazon.com/elasticsearch-service/
LookingatthirdpartyoptionsThereareafewoptionswhenitcomestohostingcentralloggingforyourcontainersexternaltoyourownserverinstances.Someoftheseare:
LogEntries:https://logentries.com/Loggly:https://www.loggly.com/
Bothoftheseservicesofferafreetier.LogEntriesalsooffersa“LogentriesDockerFree”accountthatyoucanfindoutmoreaboutathttps://logentries.com/docker/
NoteAsrecommendedintheExploringThirdPartyOptionschapter,itisbesttouseacloudservicewhenevaluatingthirdpartyservices.Theremainderofthischapterassumesthatyouarerunningacloudhost.
Let’slookatconfiguringtheLogEntriesonanexternalserver,firstofallyouneedtohavesignedupforanaccountathttps://logentries.com/.Onceyouhavesignedup,youshouldbetakentoapageinwhichyourlogswilleventuallybedisplayed.
Tostart,clickontheAddnewlogbuttoninthetop-rightcornerofthepageandthenclicktheDockerlogointhePlatformssection.
YouhavetonameyoursetoflogsintheSelectsetsection,sogiveanametoyourlogset.YounowhavethechoiceofbuildingyourowncontainerlocallyusingtheDockerfilefromhttps://github.com/logentries/docker-logentries:
gitclonehttps://github.com/logentries/docker-logentries.git
cddocker-logentries
dockerbuild-tdocker-logentries.
Afterrunningtheprecedingcommand,youwillgetthefollowingoutput:
Beforeyoustartyourcontainer,youwillneedtogenerateanaccesstokenforyourlogsetbyclickingonGenerateLogToken.Onceyouhavethis,youcanlaunchyourlocallybuiltcontainersusingthefollowingcommand(replacethetokenwiththeoneyouhavejustgenerated):
dockerrun-d-v/var/run/docker.sock:/var/run/docker.sockdocker-
logentries-twn5AYlh-jRhgn3shc-jW14y3yO-T09WsF7d-j
YoucandownloadtheimagestraightfromtheDockerhubbyrunning:
dockerrun-d-v/var/run/docker.sock:/var/run/docker.sock
logentries/docker-logentries-twn5AYlh-jRhgn3shc-jW14y3yO-T09WsF7d–j
It’sworthpointingoutthattheautomaticallygeneratedinstructionsgivenbyLogEntrieslaunchesthecontainerintheforeground,ratherthandetachingfromthecontaineronceithasbeenlaunchedliketheprecedinginstructions.
Onceyouhavethedocker-logentriescontainerupandrunning,youshouldstarttoseelogsfromyourcontainerstreamedinreal-timetoyourdashboard:
Fromhere,youwillbeabletoqueryyourlogs,createdashboards,andcreatealertsdependingontheaccountoptionyougofor.
SummaryInthischapter,wehavecoveredhowtoquerytheSTDOUTandSTDERRoutputfromyourcontainersusingthetoolbuiltintoDocker,howtoshipthemessagestoanexternalsource,ourELKstack,andhowtostorethemessagesevenafterthecontainerhasbeenterminated.Finally,wehavelookedatafewofthethird-partyserviceswhoofferservicestowhichyoucanstreamyourlogs.
Sowhygotoallofthiseffort?Monitoringisn’tjustaboutkeepingandqueryingCPU,RAM,HDD,andNetworkutilizationmetrics;thereisnopointinknowingiftherewasaCPUspikeanhouragoifyoudon’thaveaccesstothelogfilestoseeifanyerrorswerebeinggeneratedatthattime.
Theserviceswehavecoveredinthischapterofferthequickestandmostefficientinsightsintowhatcanquicklybecomeacomplexdataset.
Inthenextchapter,wewilllookatalloftheservicesandconceptswehavecoveredinthebookandapplythemtosomerealworldscenarios.
Chapter8.WhatAretheNextSteps?Inthisfinalchapter,wewilllookatthenextstepsyoucantaketomonitoryourcontainers,bytalkingaboutthebenefitsofaddingalertstoyourmonitoring.Also,wewillcoversomedifferentscenariosandalsowhichtypeofmonitoringisappropriateforeachofthem:
Commonproblems(performance,availability,andsoon)andwhichtypeofmonitoringisbestforyoursituation.Whatarethebenefitsofalertingonthemetricsyouarecollectingandwhataretheoptions?
SomescenariosTolookatwhichtypeofmonitoringyoumightwanttoimplementforyourcontainer-basedapplications,weshouldworkthroughafewdifferentexampleconfigurationsthatyourcontainer-basedapplicationscouldbedeployinginto.First,let’sremindourselvesaboutPets,Cattle,Chickens,andSnowflakes.
Pets,Cattle,Chickens,andSnowflakesBackintheChapter1,IntroductiontoDockerMonitoring,wespokeaboutPets,Cattle,Chickens,andSnowflakes;inthatchapter,wedescribedwhateachtermmeantwhenitwasappliedtomodernclouddeployments.Here,wewillgointoalittlemoredetailabouthowthetermscanbeappliedtoyourcontainers.
PetsForyourcontainerstobeconsideredaPet,youwillbemorethanlikelytoberunningeitherasingleorasmallnumberoffixedcontainersonadesignatedhost.
Eachoneofthesecontainerscouldbeconsideredasinglepointoffailure;ifanyoneofthemgoesdown,itwillmorethanlikelyresultinerrorsforyourapplication.Worststill,ifthehostmachinegoesdownforanyreason,yourentireapplicationwillbeoffline.
ThisisatypicaldeploymentmethodformostofourfirststepswithDocker,andinnowayshoulditbeconsideredbad,frownedupon,ornotrecommend;aslongasyouareawareofthelimitations,youwillbefine.
Thispatterncanalsobeusedtodescribemostdevelopmentenvironments,asyouareconstantlyreviewingitshealthandtuningasneeded.
YouwillmorethanlikelybehostingthemachineonyourlocalcomputeroronahostingservicesuchasDigitalOcean(https://www.digitalocean.com/).
CattleForthebulkofproductionorbusinesscriticaldeployments,youshouldaimtolaunchyourcontainersinaconfigurationthatallowsthemtoautomaticallyrecoverthemselvesafterafailure,or,whenmorecapacityisneeded,additionalcontainersarelaunchedandthenterminatedwhenthescalingeventisover.
Youwillmorethanlikelybeusingapubliccloud-basedserviceasfollows:
AmazonEC2ContainerService:https://aws.amazon.com/ecs/GoogleContainerEngine:https://cloud.google.com/container-engine/JoyentTriton:https://www.joyent.com/blog/understanding-triton-containers/
Alternatively,youwillbehostingonyourownserversusingaDocker-friendlyandcluster-awareoperatingsystemasfollows:
CoreOS:https://coreos.com/RancherOS:http://rancher.com/rancher-os/
Youwon’tcaresomuchastowhereacontainerislaunchedwithinyourclusterofhosts,aslongasyoucanroutetraffictoit.Toaddmorecapacitytothecluster,youwillbebringingupadditionalhostswhenneededandremovingthemfromtheclusterwhennotneededinordertosaveoncosts.
Chickens
Itsmorethanlikelyyouwillbeusingcontainerstolaunch,processdata,andthenterminate.Thiscanhappenanytimefromonceadaytoseveraltimesaminute.Youwillbeusingadistributedschedulerasfollows:
KubernetesbyGoogle:http://kubernetes.io/ApacheMesos:http://mesos.apache.org/
Becauseofthis,youwillhavealargenumberofcontainerslaunchingandterminatingwithinyourcluster;youdefinitelywon’tcareaboutwhereacontainerislaunchedorevenhowtrafficisroutedtoit,aslongasyourdataisprocessedcorrectlyandpassedbacktoyourapplication.
LiketheclusterdescribedintheCattlesection’sdescription,hostswillbeaddedandremovedautomatically,probablyinresponsetoscheduledpeakssuchasendofmonthreportingorseasonalsalesandsoon.
SnowflakesIhopeoneofthethingsyoutookawayfromthefirstchapteristhatifyouhaveanyserversorservicesthatyouconsiderbeingSnowflakes,thenyoushoulddosomethingtoretirethemassoonaspossible.
Luckily,duetothewaythecontainerizingofyourapplicationsworks,youshouldneverbeabletocreateasnowflakeusingDocker,asyourcontainerizedenvironmentshouldalwaysbereproducible,eitherbecauseyouhavetheDockerfile(everyonemakesbackupsright?)oryouhaveaworkingcopyofthecontainerimagebecauseyouhaveexportedthecontainerasawholeusingthebuilt-intools.
NoteSometimesitmaynotbepossibletocreateacontainerusingaDockerfile.Instead,youcanbackupormigrateyourcontainersbyusingtheexportcommand.Formoreinformationonexportingyourcontainers,seethefollowingURL:
https://docs.docker.com/reference/commandline/export/
Ifyoufindyourselfinthisposition,letmebethefirsttocongratulateyouonmitigatingafuturedisasterbypromotingyourSnowflakeintoaPetorevenCattleaheadofanyproblems.
TipStillrunningaSnowflake?
IfyoufindyourselfstillrunningaSnowflakeserverorservice,Icannotstressenoughthatyoulookatdocumenting,migrating,orupdatingtheSnowflakeassoonaspossible.Thereisnopointinmonitoringaservicethatmaybeimpossibleforyoutorecover.Rememberthattherearecontainersforoldtechnologies,suchasPHP4,ifyoureallyneedtorunthem.
ScenariooneYouarerunningapersonalWordPresswebsiteusingtheofficialcontainersfromtheDockerHub;thecontainershavebeenlaunchedusingaDockerComposefileliketheonewehaveusedseveraltimesthroughoutthisbook.
YouhavetheDockerComposefilestoredinaGitHubrepositoryandyoucantakesnapshotsofthehostmachineasabackup.Asit’syourownblog,youarefinerunningitonasinglecloud-basedhost.
Asuitablemonitoringwillbeasfollows:
DockerstatsDockertopDockerlogscAdvisorSysdig
Asyouarerunningasinglehostmachinethatyouaretreatingasabackup,thereisnorealneedforyoutoshipyourlogfilestoacentrallocationasoddsareyourhostmachines;likethecontainers,itshostingwillbeonlineformonthsorpossiblyevenyears.
Itisunlikelythatyouwillneedtodigtoodeeplyintoyourcontainers’historicalperformancestats,asmostofthetuningandtroubleshootingwillbedoneinrealtimeasproblemsoccur.
Withthemonitoringtoolssuggested,youwillbeabletogetagoodinsightintowhatishappeningwithinyourcontainersinrealtime,andtogetmorethanenoughinformationonprocessesthatareconsumingtoomuchRAMandCPU,alongwithanyerrormessagesfromwithinthecontainers.
YoumaywanttoenableaservicesuchasPingdom(https://www.pingdom.com/)orUptimeRobot(http://uptimerobot.com/).TheseservicespollyourwebsiteeveryfewminutestoensurethattheURLyouconfigurethemto,checkwhetheritsloadingwithinacertaintimeoratall.Iftheydetectanyslowdownorfailureswiththepageloading,theycanbeconfiguredtosendaninitialalerttonotifyyouthatthereisapotentialissue,suchasboththeservicesmentionedhaveafreetier.
ScenariotwoYouarerunningacustome-commerceapplicationthatneedstobehighlyavailableandalsoscaleduringyourpeaktimes.Youareusingapubliccloudserviceandthetoolsetthatcomeswithittolaunchcontainersandroutetraffictothem.
Asuitablemonitoringwillbeasfollows:
cAdvisor+PrometheusZabbixSysdigCloudNewRelicServerMonitoringDatadogELK+LogspoutLogEntriesLoggly
Withthisscenario,thereisabusinessneedtonotonlybenotifiedaboutcontainerandhostfailures,butalsotoholdyourmonitoringdataandlogsawayfromyourhostserverssothatyoucanproperlyreviewhistoricalinformation.YoumayalsoneedtokeeplogsforPCIcomplianceorinternalauditingforafixedperiodoftime.
Dependingonyourbudget,youcanachievethisbyhostingyourownmonitoring(ZabbixandPrometheus)andcentrallogging(ELK)stackssomewherewithinyourinfrastructure.
Youcanalsochoosetorunafewdifferentthird-partytoolssuchascombiningtoolsthatmonitorperformance,forexample,SysdigCloudorDatadog,withacentralloggingservice,suchasLogEntriesorLoggly.
Ifappropriate,youcanalsorunacombinationofself-hostedandthird-partytools.
Whiletheself-hostedoptionmayappeartobethemostbudget-friendlyoption,therearesomeconsiderationstotakeintoaccount,asfollows:
Yourmonitoringneedstobehostedawayfromyourapplication.Thereisnopointinhavingyourmonitoringinstalledonthesamehostasyourapplication;whatwillalertyouifthehostfails?Yourmonitoringneedstobehighlyavailable;doyouhavetheinfrastructuretodothis?Ifyourapplicationneedstobehighlyavailable,thensodoesyourmonitoring.Youneedtohaveenoughcapacity.Doyouhavethecapacitytobeabletostorelogfilesandmetricsgoingbackamonth,6months,orayear?
Ifyouaregoingtohavetoinvestinanyoftheprecedingoptions,thenitwillbeworthweighingupthecostsofinvestinginboththeinfrastructureandthemanagementofyourownmonitoringsolutionagainstusingathird-partythatwilloffertheprecedingoptionsasaservice.
Ifyouareusingacontainer-onlyoperatingsystemsuchasCoreOSorRancherOS,thenyouwillneedtochooseaservicewhoseagentorcollectorcanbeexecutedfromwithinacontainer,asyouwillnotbeabletoinstalltheagentbinariesdirectlyontheOS.
Youwillalsoneedtoensurethatyourhostmachineisconfiguredtostarttheagents/collectorsonboot.Thiswillensurethatassoonasthehostmachinejoinsacluster(whichistypicallywhencontainerswillstarttopopuponthehost),itisalreadysendingmetricstoyourchosenmonitoringservices.
ScenariothreeYourapplicationlaunchesacontainereachtimeyourAPIiscalledfromyourfrontendapplication;thecontainertakestheuserinputfromadatabase,processesit,andthenpassestheresultsbacktoyourfrontendapplication.Oncethedatahasbeensuccessfullyprocessed,thecontaineristerminated.Youareusingadistributedschedulingsystemtolaunchthecontainers.
Asuitablemonitoringwillbeasfollows:
ZabbixSysdigCloudDatadogELK+LogspoutLogEntriesLoggly
Inthisscenario,youmorethanlikelydonotwanttomonitorthingssuchasCPUandRAMutilization.Thesecontainersafterallshouldonlybearoundforafewminutes,andalsoyourschedulerwilllaunchthecontaineronthehostmachinewherethereisenoughcapacityforthetasktoexecute.
Instead,youwillprobablywanttokeeparecordtoverifythatthecontainerlaunchedandterminatedasexpected.YouwillalsowanttomakesurethatyoulogtheSTDOUTandSTDERRfromthecontainerwhileitisactive,asoncethecontainerhasbeenterminated,itwillbeimpossibleforyoutogetthesemessagesback.
Withthetoolslistedintheprecedingpoints,youshouldbeabletobuildsomequiteusefulqueriestogetadetailedinsightintohowyourshortrunprocessesareperforming.
Forexample,youwillbeabletogettheaveragelifetimeofacontainer,asyouknowthetimethecontainerwaslaunchedandwhenitwasterminated;knowingthiswillthenallowyoutosetatriggertoalertyouifanycontainersarearoundforanylongerthanyouwouldexpectthemtobe.
AlittlemoreaboutalertingAlotofthetoolswehavelookedatinthisbookofferatleastsomesortofbasicalertingfunctionality;themillion-dollarquestionisshouldyouenableit?
Alotofthisisdependentonthetypeofapplicationyouarerunningandhowthecontainershavebeendeployed.Aswehavealreadymentionedafewtimesinthischapter,youshouldneverreallyhaveaSnowflakecontainer;thisleavesuswithPets,Cattle,andChickens.
ChickensAsalreadydiscussedintheprevioussection,youprobablydon’tneedtoworryaboutgettingalertsforRAM,CPU,andharddriveperformanceonaclusterthatisconfiguredtorunChickens.
Yourcontainersshouldnotbeuplongenoughtoexperienceanyrealproblems;however,shouldtherebeanyunexpectedspikes,yourschedulerwillprobablyhaveenoughintelligencetodistributeyourcontainerstohoststhathavethemostavailableresourcesatthattime.
Youwillneedtoknowifanyofyourcontainershavebeenrunninglongerthanyouexpectthemtobeup;forexample,aprocessinacontainerthatnormallytakesnomorethan60secondsisstillrunningafter5minutes.
Thisnotonlymeansthatthereisapotentialproblem,italsomeansthatyoufindyourselfrunninghoststhatonlycontainstalecontainers.
CattleandPetsWhenitcomestosettingupalertsonCattleorPets,youhaveafewoptions.
YouwillmorethanlikelywanttoreceivealertsbasedonCPUandRAMutilizationforboththehostmachineandthecontainers,asthiscouldindicateapotentialproblemthatcouldcauseslowdownwithintheapplicationandalsolossofbusiness.
Asmentionedpreviously,youwillprobablyalsowanttobealertedifyourapplicationstartstoservethecontentthatisunexpected.Forexample,ahostandacontainerwillquitehappilysitthereservinganapplicationerror.
YoucanuseaservicesuchasPingdom,Zabbix,orNewRelictoloadapageandcheckforthecontentinthefooter;ifthiscontentismissing,thenanalertcanbesent.
Dependingonhowfluidyourinfrastructureis,inaCattleconfiguration,youwillprobablywanttobealertedwhencontainersspinupanddown,asthiswillindicateperiodsofhightraffic/transactions.
SendingalertsSendingalertsdiffersforeachtool,forexample,analertcouldbeassimpleassendinganemailtoinformyouthatthereisanissuetothesoundingofanaudiblealertinaNetworkOperationsCenter(NOC)whentheCPUloadofacontainergoesabovefive,ortheloadonthehostgoesabove10.
Forthoseofyouwhorequireanon-callteamtobealerted,mostofthesoftwarewehavecoveredhassomelevelofintegrationalertaggregationservicessuchasPagerDuty(https://www.pagerduty.com).
TheseaggregationserviceseitherinterceptyouralertemailsorallowservicestomakeAPIcallstothem.Whentriggered,theycanbeconfiguredtoplacephonecalls,sendSMSmessages,andevenescalatetosecondaryon-calltechnicianifanalerthasnotbeenflaggeddownwithinadefinabletime.
Ican’tthinkofanycaseswhereyoushouldn’tlookatenablingalerting,afterall,it’salwaysbesttoknowaboutanythingthatcouldeffectyourapplicationbeforeyourendusersdo.
Howmuchalertingyouenableisreallydowntowhatyouareusingyourcontainersfor;however,Iwouldrecommendthatyoureviewallyouralertsregularlyandalsoactivelytuneyourconfiguration.
Thelastthingyouwantisaconfigurationthatproducestoomanyfalsepositivesoronethatistootwitchy,asyoudonotwanttheteamwhoreceivesyouralertstobecomedesensitizedtothealertsthatyouaregenerating.
Forexample,ifacriticalCPUalertistriggeredevery30minutesbecauseofascheduledjob,thenyouwillprobablyneedtoreviewthesensitivityofthealert,otherwiseitiseasyfortheengineertosimplydismissacriticalalertwithoutthinkingaboutit,as“thisalertcomeseveryhalfanhourandwillbeokinafewminutes”,whenyourentireapplicationcouldbeunresponsive.
KeepingupWhileDockerhasbeenbuiltontopofwell-establishedtechnologiessuchasLinuxContainers(LXC),thesehavetraditionallybeendifficulttoconfigureandmanage,especiallyfornon-systemadministrators.
Dockerremovesalmostallthebarrierstoentry,allowingeveryonewithasmallamountofcommand-lineexperiencetolaunchandmanagetheirowncontainer-basedapplications.
Thishasforcedalotofthesupportingtoolstoalsolowertheirbarriertoentry.Softwarethatoncerequiredcarefulplanningtodeploy,suchassomeofthemonitoringtoolswecoveredinthisbook,cannowbedeployedandconfiguredinminutesratherthanhours.
Dockerisalsoaveryfast-movingtechnology;whileithasbeenconsideredproduction-readyforawhile,newfeaturesarebeingaddedandexistingfeaturesareimprovedwithregularupdates.
Sofar,in2015,therehavebeen11releasesofDockerEngine;ofthese,onlysixhavebeenminorupdatesthatfixbugs,andtheresthaveallbeenmajorupdates.Detailsofeachreleasecanbefoundintheproject’sChangelog,whichcanbefoundathttps://github.com/docker/docker/blob/master/CHANGELOG.md.
BecauseofthepaceofdevelopmentofDocker,itisimportthatyoualsoupdateanymonitoringtoolsyoudeploy.Thisisnotonlytokeepupwithnewfeatures,butalsotoensurethatyoudon’tlooseanyfunctionalityduetochangesinthewayinwhichDockerworks.
Thisattitudeofupdatingmonitoringclients/toolscanbeabitofachangeforsomeadministratorswhomaybeinthepastwouldhaveconfiguredamonitoringagentonaserverandthennotthoughtaboutitagain.
SummaryAsdiscussedinthischapter,Dockerisafastmovingtechnology.Whilethisbookhasbeeninproduction,therehavebeenthreemajorversionsreleasedfrom1.7to1.9;witheachreleaseDockerhasbecomemorestableandmorepowerful.
Inthischapter,wehavelookedatdifferentwaystoimplementthetechnologiesthathavebeendiscussedinthepreviouschaptersofthisbook.Bynow,youshouldhaveanideaofwhichapproachisappropriatetomonitoryourcontainersandhostmachines,forbothyourapplicationandforthewaytheapplicationhasbeendeployedusingDocker.
Nomatterwhichapproachyouchosetotake,itisimportantthatyoustayup-to-datewithDocker’sdevelopmentandalsothenewmonitoringtechnologiesastheyemerge,thefollowinglinksaregoodstartingpointstokeepyourselfinformed:
DockerEngineeringBlog:http://blog.docker.com/category/engineering/DockeronTwitter:https://twitter.com/dockerDockeronReddit:https://www.reddit.com/r/dockerDockeronStackOverflow:http://stackoverflow.com/questions/tagged/docker
OneofthereasonswhytheDockerprojecthasbeenembracedbydevelopers,systemadministratorsandevenenterprisecompaniesisbecauseitisabletomoveataquickpace,whileaddingmorefeaturesandveryimpressivelymaintainingitseaseofuseandflexibility.
Overthenext12months,thetechnologyissettobeevenmorewidespread;theimportanceofensuringthatyouarecapturingusefulperformancemetricsandlogsfromyourcontainerswillbecomemorecriticalandIhopethatthisbookhashelpedyoustartyourjourneyintomonitoringDocker.
IndexA
alertsenabling/Alittlemoreaboutalertingsettingup,onChickens/Chickenssettingup,onCattle/CattleandPetssettingup,onPets/CattleandPetssending/Sendingalerts
AmazonEC2ContainerServiceURL/Cattle
AmazonWebServicesURL/Awordaboutexternallyhostedservices
ApacheBenchURL/RunningDockerstats
ApacheMesosURL/Chickens
AtomicURL/WhyuseaSaaSservice?
CcAdvisor
about/WhatiscAdvisor?executing,containerused/RunningcAdvisorusingacontainercompiling,fromsource/CompilingcAdvisorfromsourcereferencelink/Alternatives?
Cattleabout/Cattlecontainers,deployingonto/Cattlealerts,settingup/CattleandPets
Changelog,DockerURL/Keepingup
Chickensabout/Chickenscontainers,deployingonto/Chickensalerts,settingup/Chickens
cloudDocker,deployingin/DeployingDockerinthecloud
CloudComputingreferencelink/Chickens
Cloudscalingreferencelink/Pets,Cattle,Chickens,andSnowflakes
containerused,forexecutingcAdvisor/RunningcAdvisorusingacontainer
ContainerDownabout/Triggers
containerlogsviewing/Viewingcontainerlogs
containersresourceutilization,tracking/Whatjusthappened?comparing,tohostmachine/Comparecontainerstoyourhostmachinedeploying,ontoPets/Petsdeploying,ontoCattle/Cattledeploying,ontoChickens/Chickensdeploying,ontoSnowflakes/Snowflakes
containers,monitoringscenarios/Scenarioone,Scenariotwo,Scenariothree
containerstatsviewing/Viewingcontainerstatssubcontainers/Subcontainersdriverstatus/Driverstatusimages/Images
CoreOS
URL/WhyuseaSaaSservice?,CattleCsysdig
about/UsingCsysdigusing/UsingCsysdig
customgraphscreating/Createcustomgraphsreferencelink/Createcustomgraphs
DDashboard
launching/DashboardDatadog
URL/WhyuseaSaaSservice?,Datadogabout/Datadogagent,installing/Installingtheagentagentinstallation,URL/Installingtheagentwebinterface,exploring/Exploringthewebinterfacereferences/Summaryandfurtherreading
datavolumecontainercreating/Thenextsteps
DigitalOceanURL/Pets
DigitalOceanURL/Awordaboutexternallyhostedservices
Dockerabout/DockerURL,foruserguide/Runningavirtualserver,Thenextstepsdeploying,incloud/DeployingDockerinthecloudadvancements/Keepingup
docker-composefileused,forexecutingELKstack/Whataboutproduction?
DockerComposeabout/Docker
DockerEngineabout/Docker
Dockerexecabout/Dockerexec
dockerexportcommandURL/Snowflakes
DockerHubabout/DockerURL/Docker,WhatiscAdvisor?,Usingcontainers
DockerimagesURL/Driverstatus
Dockerstatsabout/Dockerstatsexecuting/RunningDockerstats
Dockertopabout/Dockertop
driverstatusobtaining/Driverstatus
EElasticsearch
URL,forlaunching/StartingthestackELKstack
about/ELKStackstarting/StartingthestackLogspout/Logspoutlogs,reviewing/Reviewingthelogsexecuting,docker-composefileused/Whataboutproduction?references/Whataboutproduction?
environmentcloning/Cloningtheenvironment
externallyhostedservicesabout/AwordaboutexternallyhostedservicesDocker,deployingincloud/DeployingDockerinthecloud
LLiquidWaxEster(LXE)
about/Keepinguplmctfy
URL/WhatiscAdvisor?localenvironment
launching/LaunchingalocalenvironmentLogEntries
URL/Lookingatthirdpartyoptionsconfiguring/Lookingatthirdpartyoptions
LogglyURL/Lookingatthirdpartyoptions
logsreviewing/Reviewingthelogs
LogspoutURL,forlaunching/Startingthestackabout/Logspoutreferences/Logspout
LogstashURL,forlaunching/Startingthestack
MMariaDB10
URL/Usingcontainersmetrics
collecting/Collectingmetricsrecording/Dockermetricscustomgraphs,creating/Createcustomgraphscontainers,comparingtohostmachine/Comparecontainerstoyourhostmachinetriggers,defining/Triggers
MicrosoftAzureURL/Awordaboutexternallyhostedservices
MonitoringDockerrepositoryURL/Cloningtheenvironment
NNagios
URL/Zabbixncurses
about/WhatisSysdig?NetworkOperationsCenter(NOC)
about/SendingalertsNewRelic
URL/WhyuseaSaaSservice?,NewRelicabout/NewRelicagent,installing/Installingtheagentwebinterface,exploring/Exploringthewebinterfacereferences/Summaryandfurtherreading
P—privilegedflag
about/RunningcAdvisorusingacontainerURL/RunningcAdvisorusingacontainer
PagerDutyURL/Triggers,Sendingalerts
Petsabout/Petscontainers,deployingonto/Petsalerts,settingup/CattleandPets
phpinfoURL/RunningDockerstats
PingdomURL/Scenarioone
processesisolating/Whataboutprocesses?Dockertop/DockertopDockerexec/Dockerexec
Prometheusabout/Prometheuslaunching/LaunchingPrometheusquerying/QueryingPrometheusDashboard,launching/DashboardURL,fordocumentation/Dashboarddatavolumecontainer,creating/Thenextstepsreferences/Thenextsteps
PuppetForgeURL/Usingvagrant
SSaaSservice
using/WhyuseaSaaSservice?serverstyles
selecting/Sowhatdoesthisallmean?Slack
URL/TriggersSnowflakes
about/Snowflakesreferencelink/Snowflakescontainers,deployingonto/Snowflakes
StandardStreamsreferencelink/Viewingcontainerlogs
subcontainersabout/Subcontainers
SysAdminabout/Pets,Cattle,Chickens,andSnowflakes
Sysdigabout/WhatisSysdig?installing/InstallingSysdigURL/InstallingSysdigusing/UsingSysdigbasics/Thebasicsdata,capturing/Capturingdatacontainers,listing/Containersreferences/Furtherreading
SysDigCloudURL/WhyuseaSaaSservice?
SysdigCloudabout/SysdigCloudfunctionality/SysdigCloudreferences/SysdigCloud,Summaryandfurtherreadingagent,installing/Installingtheagentagentinstallation,URL/Installingtheagentcontainers,exploring/Exploringyourcontainers
VVagrant
URL/Launchingalocalenvironmentvagrantbox
URL/RunningavirtualserverVirtualBox
URL/Launchingalocalenvironmentvirtualserver
executing/Runningavirtualserverhalting/Haltingthevirtualserver
VMwarevCloudAirURL/Awordaboutexternallyhostedservices
WWebinterface
about/TheWebinterfaceoverview/OverviewProcesses/ProcessesCPU/CPUMemory/MemoryNetwork/NetworkFilesystem/Filesystemdrawbacks/Thisisallgreat,what’sthecatch?
WordPressURL,forinstallation/UsingSysdig