PAN109098 - Going Splunking: Using Splunk for server log

PAN109098- GoingSplunking:UsingSplunkforserverloganalyticsinBlackboardLearn7/24/17(Monday)11:00AM- 284-285

PRESENTEDBY:CHRISBRAYBLACKBOARDADMINISTRATORATUNIVERSITYOFARKANSAS|IANGOHSOFTWAREENGINEERATJOHNSHOPKINSUNIVERSITY

Statementsregardingourproductdevelopmentinitiatives, includingnewproductsandfutureproductupgrades,updatesorenhancementsrepresentourcurrent intentions,butmaybemodified,delayedorabandonedwithout priornotice andthere isnoassurancethatsuchoffering,upgrades,updatesorfunctionalitywillbecomeavailableunlessanduntiltheyhavebeenmadegenerallyavailabletoourcustomers.

TheProblem

TheProblem

Goodmorning,

We’reseeinganerrormessagewhentryingtoreviewthefullresultsofaquizinthe“PrinciplesofEconomics– SU17.2”(ORG.SA.PRINCIPLESOFECONOMICS-SU17.2)communitysite. TheerrormessageisshownbelowandoccurswhenstudentsattempttoviewthefullresultsviaMyGradesandwhenmeandtheinstructor trytoviewtheattemptviatheFullGradeCenter. ThequizinquestionisModule5Graded.

Pleaseadvise. Thanks!

blackboard/apis/assessment/OrderingAnswerAttemptForreference,theErrorIDis4e1f842a-6f34-4657-8c76-d3d50bbb73a5.Tuesday,June27,20178:38:44AMEDT

Admins- Howoftendoyougetthisemail?

TheProblem

• Whatstepsdoyounormallytaketoanalyzetheissue?

•

AskTheAudience

TheProblem

1. Manylogs

2. LogFormats– lackconsistentformats,timeformats

3. MultipleServers– needtocentralizelogs

4. Expertise/Access– whocanaccessthelogs,whounderstandswhichlogdoeswhat?

Splunk

WhatisSplunk?https://www.splunk.com/

• “GoogleforLogfiles”-- HelgeKlein/https://helgeklein.com/blog/2014/09/splunk-work/

• “Schemaonthefly”

• Splunk EnterpriseCoreFeatures:

– CollectandIndexData

– Search,AnalyzeandVisualize

– Monitor,AlertandReport

• ProvidesPremiumSolutions:Security, ITServiceIntelligence,UserBehaviorAnalytics

Components ofSplunk

• Forwarders - Splunk softwarecaningestallkindsofdatatypesandsources.File-baseddatacanbesentvia forwarders thatresidedirectlyonthedatasources

• Indexer - AnindexeristheSplunk instancethatindexesdata.Theindexertransformstherawdataintoeventsandstorestheeventsintoanindex.

• SearchHead- Inadistributedsearchenvironment,thesearchheadistheSplunkinstancethatdirectssearchrequeststoasetofsearchpeersandmergestheresultsbacktotheuse

Managed&SelfHostedBlackboardEnvironments

ChrisBray/U.Arkansas–ManagedHosting• logsarezippedandsentoverdaily

• Splunk forwardingagenttransferslogsintolocalinstanceofSplunk

I.Goh/JohnsHopkins University- SelfHosted• useForwarderstosendlogstolocal instanceofSplunk

• prodwebservers(andourdev,testserversaswell)sendreal-timedata

WhataboutSAAS?

OfferingELK(Elasticsearch,Logstash,andKibana)?• PossiblylinktosomeoneelsedoingaDevCon onELK?

Alternatives toSplunk

• MS SystemCenterOperationsManager(SCOM) (Windows)

• Nagios (OpenSource)

• ELK(Elasticsearch,Logstash,andKibana)

BlackboardLogs– whatdoyousend,howlongisitkept/indexed

C.Bray /U.Arkansas- Manage Hosted• Blackboard

• tomcat/access-logs

• bb-email.log

• bb-authentication.log

I.Goh /JohnsHopkins University- SelfHosted• MicrosoftIIS(sourcetype: iis)

• Blackboard

• bb-services-log.txt (sourcetype:bb_services - mightbejust acustomlabelforus)

• tomcat/bb-access-log- txt(sourcetype:access_combined_wcookie)

• tomcat/stdout-stderr- log(sourcetype: log4j)

• Productionindexkeepssixmonthsofdata

Examples

SearchExamples

HeretherebeExampleshttp://bbadmin.uark.edu

ReportExamples

• StartupTimesofwebapp servers

• Search:host=hostname*source=“tomcat\\stdout-stderr-*.log""Blackboardapplicationserverreadytoacceptrequests”

• Report:

ReportExamples

• Geolocation:visualizewheretheBbStudentappishittingusfrom(beforewegetthemedu.comrequests)

• Search:host=hostname*"/webapps/Bb-mobile-bb_bb60/customAuthSuccess"|iplocation c_ip |geostats countBYc_ip

• Report:

AlertExamples

• Useareal-timealerttomonitoreventsoreventpatternsastheyhappen.Youcancreatereal-timealertswithper-resulttriggeringorrollingtimewindowtriggering

• WouldnotbeusefulintheManageHostedsituationiflogswillbedelayed(duetotransfer)

• JHU:weuseSCOMforalerts

• Example:theStartupTimesearchcouldbeusedtotriggeranemail/pager(orevenaPhilipsHuelight!)

UsesofSplunk outsideofIT

• UNLV– usingSplunk toanalyzelearningdataw/machinelearning

• https://www.splunk.com/en_us/resources/video.UzaWVuNjE60_AMjGA_NfnDfE2FGoIIFB.html#

• “Thediscoveryandminingofsuch(LMS)logsledhimtobuildadatadictionarythatenabledhimtoidentifytheevents,classifythem,andgaininsightsintotheactionsstudentswerelikelytotakeandwhichonespredictedtheirachievement“

Splunk Resources

GettingStartedwithSplunk

• https://www.splunk.com/en_us/download.html

• WhenyoudownloadSplunk Enterpriseforthefirsttime,yougetanEnterpriseTriallicensefor60days.ThisEnterpriseTriallicenseincludesallofthefeatures,butlimitstheamountofdatathatyoucanindexeachday.Thedailylimitis500MB.

• SearchTutorial-http://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/WelcometotheSearchTutorial

• QuickReferenceGuide- https://www.splunk.com/content/dam/splunk2/pdfs/solution-guides/splunk-quick-reference-guide.pdf

• Splunk Community- https://www.splunk.com/en_us/community.html

• SplunkLive!- http://splunklive.splunk.com/ (it’safreeevent!)

Splunk Free

• http://docs.splunk.com/Documentation/Splunk/6.3.2/Admin/MoreaboutSplunkFree

– Splunk Freeisdesignedforpersonal,adhocsearchandvisualizationofITdata.YoucanuseSplunkFreeforongoingindexingofsmallvolumes(<500MB/day)ofdata.Additionally,youcanuseitforshort-termbulk-loadingandanalysisoflargerdatasets--Splunk Freeletsyoubulk-loadmuchlargerdatasetsupto3timeswithina30dayperiod.Thiscanbeusefulforforensicreviewoflargedatasets.

Don’tforgettoratethissessionintheBbWorldapp.

Oh?TheSolution…

TheProblem

Goodmorning,

We’reseeinganerrormessagewhentryingtoreviewthefullresultsofaquizinthe“PrinciplesofEconomics– SU17.2”(ORG.SA.PRINCIPLESOFECONOMICS-SU17.2)communitysite. TheerrormessageisshownbelowandoccurswhenstudentsattempttoviewthefullresultsviaMyGradesandwhenmeandtheinstructor trytoviewtheattemptviatheFullGradeCenter. ThequizinquestionisModule5Graded.

Pleaseadvise. Thanks!

blackboard/apis/assessment/OrderingAnswerAttemptForreference,theErrorIDis4e1f842a-6f34-4657-8c76-d3d50bbb73a5.Tuesday,June27,20178:38:44AMEDT

Admins- Howoftendoyougetthisemail?

TheProblem– SolvingitwithSplunk 1

1. Searchfor‘4e1f842a-6f34-4657-8c76-d3d50bbb73a5’


2. Searchfor‘java.lang.NoClassDefFoundError’, overthelastsevendays


3. Seeit’sonlyappearingononehost(webapp)server

4. Takeserveroutofloadbalancer,restartservices.

5. Monitorforanyotherjava.lang.NoClassDefFoundError afterrestart

Documents

PAN109098 - Going Splunking: Using Splunk for server log