32
Copyright © 2016 Splunk Inc. Marc Chéné IT Markets Product Manager , Splunk Denis Gladkikh Principal Dev Engineer (aka outcoldman), Splunk How To Run Splunk As A Docker Image?

How To Run Splunk As A Docker Image?

Embed Size (px)

Citation preview

Page 1: How To Run Splunk As A Docker Image?

Copyright©2016Splunk Inc.

MarcChénéITMarketsProductManager,Splunk

DenisGladkikhPrincipalDevEngineer(akaoutcoldman),Splunk

HowToRunSplunk AsADockerImage?

Page 2: How To Run Splunk As A Docker Image?

Disclaimer

2

Duringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthose

containedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmadeasofthetimeanddateofitslivepresentation.Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesor

functionalitydescribedortoincludeanysuchfeatureorfunctionalityinafuturerelease.

Page 3: How To Run Splunk As A Docker Image?

Whoweare?MarcChéné

DenisGladkikh

Page 4: How To Run Splunk As A Docker Image?

Agenda Agenda

WhatisDocker?WhyrunSplunk inDocker?DemoScenarios1. SetupSplunk ClusterinDocker2. ScalingUpSplunk inDocker3. ClusterUpgrade:6.4.1to6.4.2Guidance&BestPractices

Page 5: How To Run Splunk As A Docker Image?

WhatisDocker?

5

Page 6: How To Run Splunk As A Docker Image?

First,abitaboutcontainers…

Page 7: How To Run Splunk As A Docker Image?

Docker,inoneSlide• Build - Ship- Runyourapplications

– ”Infrastructureascode”– Enablesmicroservicesarchitectures– Portable– EnablesCloudMigration

• OpenSourceandCommunityMinded– DockerEngineisOpenSource– Thousandsofappscanbe“pulled”in

Dockerhub– YourdevelopersLOVEDocker

7

Page 8: How To Run Splunk As A Docker Image?

Docker– It’snotVirtualization

8

• VMs– focusonOS• Docker– focusonapplications

• Docker– lightweightandFAST

• NOTmutuallyexclusivewithVMs

Page 9: How To Run Splunk As A Docker Image?

Docker– it’sabigdeal

9

OpenSourcedrivenecosystem

Massiveincreaseinadoption…

…Butthegrowthisjustgettingstarted

Page 10: How To Run Splunk As A Docker Image?

3PrimaryContainerUseCases

ApplicationModernization

DevOps – NewApps

CloudMigration

10

Page 11: How To Run Splunk As A Docker Image?

WhyrunSplunk inDocker?

- Goals&Benefits- Splunk ReferenceArchitecture- TargetDockerEnvironment

11

Page 12: How To Run Splunk As A Docker Image?

Goals&Benefits

12

ReduceManagementCostsTimetoValueHighAvailableReducetimetoUpgradeSimplifiedRollbackStandardConfigurationsEasiertoSupport

Page 13: How To Run Splunk As A Docker Image?

Splunk ReferenceArchitectureinDocker

13

Search

CollectionTier– HEC– x3

LoadBalancer

SplunkAdmin

SearchTier– x3

IndexingTier– x4

…

…

…

…

Data

SearchHeadClusterDeployer

IndexClusterMaster

LicenseMaster

DeploymentServer

Control

Page 14: How To Run Splunk As A Docker Image?

TargetEnvironmentArchitecture

14

Dockerv1.12DockerforAWS– https://beta.docker.com/docs/aws/#upgrading-docker-and-changing-

instance-sizes

DockerSWARM

Page 15: How To Run Splunk As A Docker Image?

Splunk SizingGuidelinesinAWS

15

Storage– EBS

ê HighAvailableê Reliableê Growupto16TB

– EBSGeneralPurpose(SSD):consistentperformance– EBSProvisionedIOPS(SSD):consistentperformanceupto4KIOPS*EBSvolumescanbedeployedinaRAIDarchitecture

ComputeRequirementsperSplunk Components– 4vCPUs– 8GBRAM

Page 16: How To Run Splunk As A Docker Image?

SplunkEnterprisedeploymentonAWSSearch Heads (8+ users)c4.4xlarge 16 vCPU, 30 GB RAM

c4.8xlarge 36 vCPU, 60 GB RAM

Indexers (50-250GB/day/indexer)c4.4xlarge 16 vCPU, 30 GB RAM

d2.4xlarge 16 vCPU, 122 GB RAM

c4.8xlarge 36 vCPU, 60 GB RAM

CloudFormation TemplatesConsistent, repeatable deployments for SplunkAbstract away details of configuring distributed SplunkExtensible and customizable to fit any need

Cloudformation Templates On GitHub

Workload = Searching + Indexing

Storage- Ephemeral or EBS- Data Retention Dependent

Compute- Best Available

Archiving- S3

Best Practices for Sizing Splunk on AWS Tech BriefSplunk Cloudformation TemplatesSplunk Admin Docs

Page 17: How To Run Splunk As A Docker Image?

Whatisdocker forAWS?

17

Announcedatdockercon16Autoscaling groupsCloudFormation Updates

Page 18: How To Run Splunk As A Docker Image?

DeliveringSplunkasaContainerImage• Splunkcontainerimages

– SplunkEnterprise6.4.1– SplunkUniversalForwarder6.4.1

• IncludesconfigurationandDockerAdd-Onforcontainermonitoringout-of-the-box

• Certifiedimage• ComingsoontotheDockerStore(http://store.docker.com)

18

docker run splunk/enterprise:6.4.1-monitordocker run splunk/universalforwarder:6.4.1-monitor

Page 19: How To Run Splunk As A Docker Image?

DemoApps

19

Splunk forAWSSplunk DockerApp– UFforlogs– cAdvisor formetrics

Page 20: How To Run Splunk As A Docker Image?

DemoTime!

- SetupSplunk ClusterinDockerinAWS

20

Page 21: How To Run Splunk As A Docker Image?

Splunk ScaleUp

Goal– ScaleuptheSearchHeadsby3à Total:5– ScaleuptheIndexersby4à Total:8– ScaleuptheCollectionlayerby2à Total:4

Page 22: How To Run Splunk As A Docker Image?

Splunk inDocker– ScaleUp

22

Search

CollectionTier– HEC– x5

LoadBalancer

SplunkAdmin

SearchTier– x5

IndexingTier– x8

…

…

…

…

Data

SearchHeadClusterDeployer

IndexClusterMaster

LicenseMaster

DeploymentServer

Control

Page 23: How To Run Splunk As A Docker Image?

DemoTime!

- ScaleUpSplunk Cluster

23

Page 24: How To Run Splunk As A Docker Image?

Splunk UpgradeOrderofUpgrade– SearchHead– LicenseManager– ClusterMaster– IndexersRecommendations– Backupyourconfigurations– BackupyourdataGoal– Upgrade6.4.1to6.4.2

Page 25: How To Run Splunk As A Docker Image?

DemoTime!

- UpgradeSplunk Cluster

25

Page 26: How To Run Splunk As A Docker Image?

GuidanceandBestPractices

26

Page 27: How To Run Splunk As A Docker Image?

GuidanceandBestPracticesUnderstandthesizingfactors– Howmuchdata(rawsizes)?Daily,Peak,Retained(archivesize),Future– Howmuchsearching?UseCases,#ofpeople,Apps– Jobs:Summarization,alerting,reporting

StandardOperationProceduresDatavolumeSearchvolume

Page 28: How To Run Splunk As A Docker Image?

WhatarethePitfallsofdocker?TBD

Page 29: How To Run Splunk As A Docker Image?

WhatNow?

29

ArchitectingSplunk forHighAvailabilityandDisasterRecovery,SessionID:74762ObservationsandRecommendationsonSplunk Performance,SessionID:74765MonitoringandTroubleshootingDockeracrossCloudandOn-PremEnvironments,SessionID:IT88095Splunking AWSforEnd-to-endVisibility,SessionID:87942– Track:Splunk Platform forOperational Intelligence

Relatedbreakoutsessionsandactivities…

Page 30: How To Run Splunk As A Docker Image?

# 1. Come visit us at our boothdocker run splunk/visitourboothvisitourbooth_1 | Booth IT Markets

# 2. Try out our docker images in Docker Storedocker run splunk/enterprise:6.4.1-monitordocker run splunk/universalforwarder:6.4.1-monitor

# 3. Demos will all be available on GitHub under Splunk!git clone https://github.com/splunk/docker-gettingstarted-conf2016-sf88089.git

# 4. Visit our site to learn more about containerscurl http://www.splunk.com/containers

CalltoAction…

Page 31: How To Run Splunk As A Docker Image?

Resources

31

Splunk EducationArchitectingandDeployingSplunk 6.4– Virtual

Splunk DocsUpgradeGuide,http://docs.splunk.com/Documentation/Splunk/6.4.2/installation/Upgradeto6.4onUNIXCapacityPlanningManual,http://docs.splunk.com/Documentation/Splunk/6.4.1/Capacity/ReferencehardwareDEPLOYINGSPLUNK®ENTERPRISEONAMAZONWEBSERVICES,http://www.splunk.com/pdfs/technical-briefs/deploying-splunk-enterprise-on-amazon-web-services-technical-brief.pdf

DockerDockerforAWS,https://beta.docker.com/docs/DockerStore,http://store.docker.com

Page 32: How To Run Splunk As A Docker Image?

THANKYOU


Run Docker On Windows Using Vagrant

Run Docker On Windows Using Vagrant

Self Improvement
Docker stack to run your deck

Docker stack to run your deck

Software
Container as a Service on GPU Cloud: Our Decision among ... · Wrap docker run and docker create commands e.g. $ nvidia-docker run --rm nvidia/cuda nvidia-smi Add docker cli options

Container as a Service on GPU Cloud: Our Decision among ... · Wrap docker run and docker create commands e.g. $ nvidia-docker run --rm nvidia/cuda nvidia-smi Add docker cli options

Documents
docker run docker service is the new · Getting Started with Docker Clustering Mike Goelzer / mgoelzer@docker.com / @mgoelzer Docker Inc. docker service is the new docker run docker

docker run docker service is the new · Getting Started with Docker Clustering Mike Goelzer / [email protected] / @mgoelzer Docker Inc. docker service is the new docker run docker

Documents
Docker Threat Modeling and Top 10 - OWASP · – docker run -e SECRET=myprrecious image – docker run –env-file ./secretsfile.txt image Kubernetes + YAML secrets: be careful Mounts

Docker Threat Modeling and Top 10 - OWASP · – docker run -e SECRET=myprrecious image – docker run –env-file ./secretsfile.txt image Kubernetes + YAML secrets: be careful Mounts

Documents
Splunk Validated Architectures · Splunk Engineers Responsible for managing the Splunk lifecycle. Managed Service Providers Entities that deploy and run Splunk as a service for customers

Splunk Validated Architectures · Splunk Engineers Responsible for managing the Splunk lifecycle. Managed Service Providers Entities that deploy and run Splunk as a service for customers

Documents
Docker Basics - docs.adfinis-sygroup.ch · Check your docker version docker version Check the available docker options docker Your first hello-world container docker run hello-world

Docker Basics - docs.adfinis-sygroup.ch · Check your docker version docker version Check the available docker options docker Your first hello-world container docker run hello-world

Documents
Host Health Monitoring with Docker Run

Host Health Monitoring with Docker Run

Software
2007-02-03...docker run --rm hello-world 11 dockerd docker Docker Hub hello-world:latest a7e0c3bf2de8 (hello-world)

2007-02-03...docker run --rm hello-world 11 dockerd docker Docker Hub hello-world:latest a7e0c3bf2de8 (hello-world)

Documents
RancherOS - The perfect place to run Docker

RancherOS - The perfect place to run Docker

Technology
runC: The little engine that could (run Docker containers) by Docker Captain Phil Estes

runC: The little engine that could (run Docker containers) by Docker Captain Phil Estes

Technology
Docker Security 201 - Global AppSec · 2018-07-28 · – docker run -e SECRET=myprrecious image – docker run –env-file ./secretsfile.txt image Kubernetes + YAML secrets: be careful

Docker Security 201 - Global AppSec · 2018-07-28 · – docker run -e SECRET=myprrecious image – docker run –env-file ./secretsfile.txt image Kubernetes + YAML secrets: be careful

Documents
Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Integrang*Splunk*Dataand* FuncBonality*Using*the*Splunk ... · The*Splunk*REST*API 12! Exposes*an*APImethod*for*every*feature*in*the*product – Run*searches* – Inputdata – Manage*Splunk*configuraons*

Documents
Docker - Package and Run Applications with Linux Containers

Docker - Package and Run Applications with Linux Containers

Technology
Docker Containers - sinog.si · Dockerization of applications GITHUB (GitLab) DOCKER BUILD DOCKERHUB (Docker Registry) DOCKER RUN FROM ubuntu RUN apt-get update RUN apt-get -y install

Docker Containers - sinog.si · Dockerization of applications GITHUB (GitLab) DOCKER BUILD DOCKERHUB (Docker Registry) DOCKER RUN FROM ubuntu RUN apt-get update RUN apt-get -y install

Documents
Docker Documentation - Read the Docs · 3.1 Docker ... 5.3 Docker Run Reference ... Docker is able to run only on the 64 bit architecture

Docker Documentation - Read the Docs · 3.1 Docker ... 5.3 Docker Run Reference ... Docker is able to run only on the 64 bit architecture

Documents
Docker Containers - InterSystems · 2018. 6. 26. · (docker build –docker run) Imágenes Docker Múltiples capas de solo-lectura Construidas manualmente o vía Dockerfile Construye

Docker Containers - InterSystems · 2018. 6. 26. · (docker build –docker run) Imágenes Docker Múltiples capas de solo-lectura Construidas manualmente o vía Dockerfile Construye

Documents
How to Run Solr on Docker and Why

How to Run Solr on Docker and Why

Technology
Easier Development and Deployment @bhosmer - @ron williams fo… · Easier Development and Deployment! ... $ docker build! $ docker run! $ docker start! $ docker stop! $ docker logs!

Easier Development and Deployment @bhosmer - @ron williams fo… · Easier Development and Deployment! ... $ docker build! $ docker run! $ docker start! $ docker stop! $ docker logs!

Documents
CRI Runtimes: Who @estesp is running my pod? · 2019-12-21 · Docker Captain, containerd maintainer @estesp. @estesp Docker Containers $ docker run redis $ docker ps $ docker stop

CRI Runtimes: Who @estesp is running my pod? · 2019-12-21 · Docker Captain, containerd maintainer @estesp. @estesp Docker Containers $ docker run redis $ docker ps $ docker stop

Documents
Docker - Beginner Biologist 1 · run: Docker will run the following named item: a Docker image. hello-world: name of a Docker image with informations to run a version of the “Hello

Docker - Beginner Biologist 1 · run: Docker will run the following named item: a Docker image. hello-world: name of a Docker image with informations to run a version of the “Hello

Documents
Neutron CI Run on Docker

Neutron CI Run on Docker

Technology
Run whatever on Hercules with Docker and Singularityalex88ridolfi.altervista.org/pulsar_tutorials_material/Sep26_How_to... · Run whatever on Hercules with Docker and Singularity

Run whatever on Hercules with Docker and Singularityalex88ridolfi.altervista.org/pulsar_tutorials_material/Sep26_How_to... · Run whatever on Hercules with Docker and Singularity

Documents
Introduction to Docker container syntax and environment · Intro to docker commands Getting help: docker --help: listsallDocker commands docker run --help: lists all options for the

Introduction to Docker container syntax and environment · Intro to docker commands Getting help: docker --help: listsallDocker commands docker run --help: lists all options for the

Documents
Docker - Nico Maas"Hallo Docker Meetup 2016!" ausführen II. Demo Docker Grundlagen 14.10.2016 • docker run -d nmaas87/docker-openwrt ping 127.0.0.1 -c 50 • ping Befehl im deattached

Docker - Nico Maas"Hallo Docker Meetup 2016!" ausführen II. Demo Docker Grundlagen 14.10.2016 • docker run -d nmaas87/docker-openwrt ping 127.0.0.1 -c 50 • ping Befehl im deattached

Documents
Docker Trickkiste - pretalx · 2019-03-22 · Docker Basics Praktische Kommandos für jeden Tag. docker run Den Container nach dem Beenden löschen docker run --rm alpine.dockerignore

Docker Trickkiste - pretalx · 2019-03-22 · Docker Basics Praktische Kommandos für jeden Tag. docker run Den Container nach dem Beenden löschen docker run --rm alpine.dockerignore

Documents
Monitoring Docker Containers with Splunk - Splunk .conf · Splunk and Docker – At A Glance Visibility in your Container Environments Delivering Splunk as Containers Monitoring for

Monitoring Docker Containers with Splunk - Splunk .conf · Splunk and Docker – At A Glance Visibility in your Container Environments Delivering Splunk as Containers Monitoring for

Documents
HPC meets Docker - Using Docker Containers to run HPC worloads

HPC meets Docker - Using Docker Containers to run HPC worloads

Software
Running Splunk Enterprise within Docker · Docker is a container system Binaries and libraries are packaged in each container What is Docker? Easily create Splunk clusters

Running Splunk Enterprise within Docker · Docker is a container system Binaries and libraries are packaged in each container What is Docker? Easily create Splunk clusters

Documents
CCCEU15 run cloudstack in docker

CCCEU15 run cloudstack in docker

Software