Splunk Enterprise for IT Troubleshooting Hands-On

Copyright©2016SplunkInc.

SplunkEnterpriseforITTroubleshooting

[email protected]

2

SafeHarborStatementDuringthecourseofthispresentation,wemaymakeforward-lookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectations and estimates basedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthosecontainedinourforward-lookingstatements,pleasereviewourfilings withtheSEC. Theforward-lookingstatementsmadeinthispresentationarebeingmadeasofthetimeanddateofitslivepresentation. If reviewedafter itslivepresentation, thispresentationmaynotcontaincurrentoraccurateinformation. Wedonotassumeanyobligationtoupdateanyforward-lookingstatementswe may make. Inaddition,anyinformationaboutour roadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice. It isforinformationalpurposesonlyandshallnot beincorporatedintoanycontractorothercommitment. Splunkundertakesnoobligationeithertodevelopthefeaturesorfunctionalitydescribed ortoincludeanysuchfeatureorfunctionalityinafuturerelease.

3

Agenda1. SplunkEnterpriseOverview2. TroubleshootingwithSplunk(LiveDemo/

Walkthrough)• Installing&SettingupanApp• LogginginandSearching• ExtractingFields• TroubleshootingInfrastructure• TroubleshootingApplications• CreatinganAlert• CreatingReportsandDashboards

3. WrapUp4. Q&A

4

EscalatingITComplexity…

SaaS/PaaS

IaaS

VIRTUALIZATION

STORAGE

PACKAGEDAPPLICATIONS

CUSTOMAPPLICATIONS

HR

Email

Finance

AppSvrDB

WebSvr

INFRASTRUCTUREAPPLICATIONS

VPN

IPPhoneIdentify

SERVERS NETWORKING

4

5

…PlaguingITOperations

SaaS/PaaS

IaaS

VIRTUALIZATION

STORAGE

PACKAGEDAPPLICATIONS

CUSTOMAPPLICATIONS

HR

Email

Finance

AppSvrDB

WebSvr

INFRASTRUCTUREAPPLICATIONS

VPN

IPPhoneIdentify

SERVERS NETWORKING

Complex,silo-basedtechnologies

Disconnectedandoutdatedpointsolutions

Reactivebrute-forceproblemresolution

Over80%oftimeonmaintaining,notinnovating

5

6

EnterpriseScalability

Industry-LeadingPlatformforMachineDataAnyMachineData OperationalIntelligence

OnlineServices Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

PackagedApplications

CustomApplicationsMessaging

TelecomsOnline

ShoppingCart

WebClickstreams

Databases

EnergyMeters

CallDetailRecords

SmartphonesandDevices

RFID

PrivateCloud

PublicCloud

SearchandInvestigation

ProactiveMonitoring

OperationalVisibility

Real-TimeBusinessInsights

6

Datacenter

Containers

7

EnterpriseScalability

AnyMachineData OperationalIntelligence

OnlineServices Web

Services

ServersSecurity GPS

Location

StorageDesktops

Networks

PackagedApplications

CustomApplicationsMessaging

TelecomsOnline

ShoppingCart

WebClickstreams

Databases

EnergyMeters

CallDetailRecords

SmartphonesandDevices

RFID

Datacenter

PrivateCloud

PublicCloud

Containers

SearchandInvestigation

ProactiveMonitoring

OperationalVisibility

Real-TimeBusinessInsights

Industry-LeadingPlatformforMachineData

Anyamount,anylocation,anysource

Schema-on-the-fly

Universalindexing

NobackendRDBMS

Noneedtofilterdata

7

8

TheFocus

ApplicationDelivery

ITOperations

Security,ComplianceandFraud

BusinessAnalytics

InternetofThingsandIndustrialData

DeveloperPlatform(RESTAPI,SDKs)

8

9

TurningMachineDataIntoOperationalIntelligence

Reactive

Proactive

9

ProactiveMonitoringandAlerting

Real-TimeBusinessInsightOperational

Visibility

Searchand

Investigate

10

IndexandAnalyzeDataAcrossYourTechnologyStackSplunkAdd-Ons,TemplatesandAppsAccelerateValueFromMachineData

Norigidschemas– addindatafromanyothersource.

APISDKs UI

Server,Storage,Network

Virtualization,Containers

OperatingSystems+Databases

CustomApplications

BusinessApplications CloudServices

WebIntelligence

MobileApplications

Stream

OperationsandServiceDesks

AppPerformanceMonitoring

DBConnect

10

11

InstallSplunkandAppØ InstallSplunk(installersonUSBkeys)Ø StartSplunk

Ø splunk start--accept-license--answer-yes--no-prompt

Ø LoginØ http://localhost:8000Ø Defaultcredentials:admin/changeme

Ø InstallappØ Clickthewidgetnextto“Apps”Ø InstallappfromfileØ ChoosetheappfromtheUSBkey

Ø RestartSplunk

12

SplunkDemoEnvironment- CloudØ GotooneofthefollowingURLs:

Ø https://od-sl-longbeach-itops-01.splunkoxygen.comØ https://od-sl-longbeach-itops-02.splunkoxygen.comØ https://od-sl-longbeach-itops-03.splunkoxygen.com

Ø LogintoSplunkusingthefollowingcredentials:– User:user[1-10]@buttercupgms.com– Password:Changeme[1-10]

13

LogintoSplunk

ClickSearch&ReportingtogetstartedusingSplunk!

14

SearchingwithSplunk

Startbytyping*inthesearchbar!

15

SearchResults

Exploretheresults!

ClickonhostClickonsourcetype

Lookattheotherfields

Next,we’llextractnewfields!

16

ExtractingFields

16

Startwiththissearch:sourcetype=customlog

ThenscrolldownandclickonExtractNewFields atthebottomofthefieldlist.

17

ExtractingFields

Clickonanyeventinthelist

Fieldsthatarealreadybeingextractedarehighlighted.

ClickNexttocontinue

18

ExtractingFields

Choose RegularExpression

Don’tworry– wewon’tbewritinganyregexes!

ClickNext

19

ExtractingFields

Clickanddragtohighlightthelastfield(itwillbeOKorNSF)

Typestatus_descriptionintheFieldNamebox

ClickAddExtraction

20

ExtractingFields

20

Checkoutthestatus_descriptioncolumn!

ClickNext

21

ExtractingFields

21

NoticethePermissions

(Youdon’tneedtochangeanythingnow,butyoumayneedtolookthisupondocs.splunk.comlaterifyouhavetrouble!)

ClickFinish

YournewfieldisnowEXTRACTED!

22

TroubleshootingInfrastructureWehavereportsofproblemsonthewebsite,solet’ssearchsourcetype=access*

Lookattheavailablefields–feelfreetoexplore!

Clickonthestatus fieldtoseethetopvalues

ClickonTopvaluesbytime

23

ExtractingFields

23

ClickonExplorethefieldsIjustcreatedinSearch

Whenyoursearchresultsshowup,expandaneventbyclickingon the>

Checkoutyournewfield!

24

TroubleshootingInfrastructure

24

ChangethegraphfromaLine toaColumn

ChangeFormat toStacked

Nowwecanseethedistributionofstatuscodesoverthelasthour!

Clickon503 inthelegendonthefarright

25


25

Nowwecanseealltheeventswitha503status!

Add |statscountbyhosttoyoursearchtoseehowmany503statuscodeseachserverhas

sourcetype=access*status=503|statscountbyhost

Nowwecanseethatwebserver-01hasmoreerrors!

Click webserver-01thenclickNewSearch

26


26

Noticethatyoursearchisnow*host=“webserver-01”

Clickonsourcetype toseewhatkindsofdatawehave

Let’sstartbycheckingfordiskspaceproblems– clickondf

27


27

Noticethatyoursearchisnow*host=“webserver-01”sourcetype=df

Scrolldownandclick on thefieldPercentUsedSpace

ClickonMaximumvalueovertime

Thiswillhelpusseeifwehaveadiskfullproblem!

28


28

Nodiskspaceissueshere– themaximumdoesn’tgomuchover70%!

Let’schangeoursearchtolookforadifferentsourcetype– modifythesearchtolookforCPUdata

Yoursearchshouldlooklikethis:*host=“webserver-01”sourcetype=cpu

29


29

ScrolldownandclickonthefieldPercentUserTime,thenclickonMaximumvalueovertime

Itlookslikewefoundtheproblem! TheCPUismaxedout.

30

TroubleshootingApplications

30

Nowsearch for error

Manyresultsarecomingfromatestmachine– onanyeventwherehost=test-01,click test-01,thenclickExcludefromsearch

Yoursearchwillnowbeerrorhost!=“test-01”

TheeventsthatremainindicateaproblemwithoneoftheMySQLservers–whichone?

31

TroubleshootingApplications

31

Itlookslikemysql-02istheserverhavingissues

Theerrorsshowaproblemwritinglogfiles,solet’scheckfordiskspaceissues

Search for host=mysql-02sourcetype=df

Click on PercentUsedSpace andchoose Maximumvalueovertime

32

CreatinganAlert

32

Wefoundtheproblem– afulldisk!

Butwouldn’tanalertbebetter?

Timechart isgreatfordataovertime,butlet’schangethesearchtousestats,whichwillgiveusasinglenumberontheStatisticstab

Tomakeiteasiertoread,we’llrenamethefieldmax(PercentUsedSpace)tomaxused byaddingasmaxused totheendyoursearch

Nowyoursearchshouldbehost=mysql-02sourcetype=df |statsmax(PercentUsedSpace)asmaxused

33

CreatinganAlert

33

Click SaveAsandchoose Alert tobringupthesettings

Add a Title

Set a scheduleor choose Real-time

Set TriggerConditions– use themaxused field thatwecreated

Throttlealertstoreducenoise

Set anActionforthealert

That’sit!

34

CreatingReports

34

Startwiththesamesearchasforthealert:host=mysql-02sourcetype=df |statsmax(PercentUsedSpace)asmaxUsed

OntheVisualization tab,choose RadialGaugeasthecharttype

Youcanchangethecolorthresholdsbychoosing ColorRangesunderFormat

Click SaveAsandchoose Report

35

CreatingaDashboard

35

Fromyourreport,click AddtoDashboard tocreateanewdashboard

Click Editsowecanaddmoreinformationtothisdashboard

Click AddPanel,thenchoose ASamplePanelfromAddPrebuiltPanel

Thispanelshowsthediskspaceforanotherserver– mysql-03

Rearrangepanelsbydragginganddropping

36

UsingDashboards

36

Click on Dashboards andthenchoose WebsiteHealth

Thisdashboardletsusseethattherearesomespikesin503errorsandthatwebserver-01hasthemosterrors– easierthanallthesearcheswestartedwith!

Click onthepiechartinthepanellabeledErrorsbyServer

37

UsingDashboards

37

ThisdashboardshowsussomeOSstatistics,andwecanseetheCPUissuewithwebserver-01intheMaximumCPUbyServeroverTimepanel

Dashboardsletustroubleshootcommonproblemsfaster!

38

DramaticResults,RapidROI

38

99.7%Uptime

Acceleratedfrommonthlyreleasesto900deploysperday

Incidentreductionby90%

95%reductioninMTTR30%accelerationinSDLC

39

Nowwhat?

39

Full-featuredplatformforreal-timeOperationalIntelligenceDownloadSplunk Enterpriseforfree!

SplunkEnterpriseasacloudserviceTryoutSplunk Cloud withafreetrial!

FeelfreetokeepworkingwiththedatafromyourUSBkey!

LearnmorewithSplunk Education!

Copyright©2015SplunkInc.

• 5,000+ITandBusinessProfessionals• 175+Sessions• 80+CustomerSpeakers

PLUSSplunk University• Threedays:Sept23-25,2017• GetSplunk CertifiedforFREE!• GetCPEcreditsforCISSP,CAP,SSCP

SEPT25-28,2017WalterE.WashingtonConventionCenterWashington,D.C.CONF.SPLUNK.COM

The8th AnnualSplunkWorldwideUsers’Conference

41

ThankYou