Embed Size (px)
Citation preview
Copyright©2016SplunkInc.
JamesErvinPrincipalEngineer,SecurityandComplianceSoluBonsSplunk,Inc.
ExtendingSplunk’sRESTAPIforFunandProfit
Disclaimer
2
DuringthecourseofthispresentaBon,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.WecauBonyouthatsuchstatementsreflectourcurrentexpectaBonsandesBmatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthosecontainedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentaBonarebeingmadeasoftheBmeanddateofitslivepresentaBon.IfreviewedaReritslivepresentaBon,thispresentaBonmaynotcontaincurrentor
accurateinformaBon.WedonotassumeanyobligaBontoupdateanyforwardlookingstatementswemaymake.InaddiBon,anyinformaBonaboutourroadmapoutlinesourgeneralproductdirecBonandis
subjecttochangeatanyBmewithoutnoBce.ItisforinformaBonalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.SplunkundertakesnoobligaBoneithertodevelopthefeaturesorfuncBonalitydescribedortoincludeanysuchfeatureorfuncBonalityinafuturerelease.
Overview
AttheconclusionofthispresentaBon,youshouldbeabletodiscussthefollowing: WhatisaRESTAPI? HowdoesSplunkimplementRESTstyle? HowcanIextendSplunk’sRESTAPIwithinmyapplicaBon? DoIhavetouseRESTstyle? WhywouldIwanttodoanyofthis?
3
RESTStyle:DefiniBonREST(RepresentaBonalStateTransfer)isasetofarchitecturalconstraintsthatmakeawebapplicaBon“RESTful”*: client-serverinteracBonoverHTTP statelesscommunicaBon cacheablecontent etc.
RESTisawaytodoIPC(interprocesscommunicaBon)overHTTP.
*Cf.ArchitecturalStylesandtheDesignofNetwork-basedSoRwareArchitectures;Fielding,R.h`p://www.ics.uci.edu/~fielding/pubs/dissertaBon/top.htm.2000.
4
RESTStyle:PracBcalConsideraBons
5
RESTisastyle,ratherthanastandardoraprotocol.
ThereisnoformalprotocolspecificaBonforREST,inthewaythatthereisforXML-RPC,SOAP,etc.
InpracBce,thiscanbebothliberaBngandfrustraBng. AsanAPIdesigner,youhavemanydegreesoffreedomtoworkwith. AsanAPIconsumer,APIsyouinteractwithwilldiffersubtly,evenwithinthelimiteddegreesoffreedomofferedbyREST.Atthebroadestlevel,notethatthestyledoesnotspecifyadefaultformat,althoughXMLandJSONarecommonlyimplemented.
InteracBngwithSplunkREST:directSplunk’sRESTAPIcanbeinteractedwithdirectlyintwoways:Viaarequesttoaport:localhost://8089(servedbythesplunkdprocess)
curl -k -u admin:changeme https://127.0.0.1:8089/services/saved/searches?count=1
Q:HowdoIinteractwithSplunkRESTonport8089,whenmybrowserismakingrequeststoport8000?Doesn’tthisviolatesame-originpolicy?
A:Yes!SeethenextslideforthealternaBveaccessmechanism…
6
InteracBngwithSplunkREST:proxiedViarequesttoportlocalhost://8000(“splunkweb”):
curl –k 'https://my_hostname:8000/en-US/splunkd/__raw/services/saved/searches?output_mode=json&count=1' -H 'Cookie: splunkweb_csrf_token_8000=11602893886132396046; session_id_8000=b1cba29d67a369c9b2410c4885a0bca1da0ab6fd; splunkd_8000=vjRt4ZFCbiyplxbUW2qDFe9EqTH3jCFciaRa^ul8RTQUDD_XN4WY4MTnzue6frZBd^j1xS2MC8p4oUXWWuIoGDia4tNgSNntTAgfudmFLjkKI2PtiBK0xMnf6KSafjg’
ThisishowyouinteractwithSplunkRESTfromyourJavascriptcode.Notetheinclusionofthelanguage(en-US)andthesplunkd/__rawprefix–thisisimportant!
7
ProxyingRESTCalls:HistoryPriortoSplunk6.2,Splunk’sownRESTAPIendpointswerewhitelistedinternallyandwereexposedonport8000throughaPythonproxy(proxy.py),whichwasexecutedaspartofthePythonsplunkwebprocess(aCherryPywebserver).Thishadtwodisadvantages:1.PythoncodeexecuBonwasrequiredforeachRESTcall.2.Thesetofproxiedendpointscouldnotbeextendedbyapps!SoappshadtoincludeaseparatePythoncomponentknownasa“Splunkwebcontroller”inordertoproxytheirownendpoints.ThisledtoextensiveduplicaBonofcode.
8
ProxyingRESTCalls:HistoryInSplunk6.2,the“expose”keywordwasintroducedinweb.conf.Thispermitsdirectpass-throughofrequeststocustomRESTendpointstotheC++splunkdback-end.Thishastwoadvantages:1. Pythoncodeisnolongerinvolvedinthe“hotpath”fromclienttoserverfor
customRESTendpointsaslongasaccessisdoneviathe/splunkd/__rawURI.(OtherURIsaresBllproxiedbyPythonandmaybeslower!)
2. TheappdevelopercannowexposeaRESTendpointdirectlyviaconfiguraBon,withoutwriBngaddiBonalcode.
9
AnatomyofaRESTCall:Pre-Splunk6.2
10
ClientSplunkdprocess
(splunkd,port8089)splunkwebprocess(root.py,port8000)
Customcontrollerscript CustomRESThandlerscript
1.Clientrequest
(imported)
2.Proxiedrequest
3.setuprequest/response(XML)*
4.executerequest/response(XML)*
5.Proxiedresponse6.Serverresponse
*=newPythonprocess
AnatomyofaRESTCall:Post-Splunk6.2
11
ClientSplunkdprocess
(splunkd,port8000and8089)
CustomRESThandlerscript
1.Clientrequest(to/splunkd/__raw)
3.setuprequest/response(XML)*
4.executerequest/response(XML)*
6.Serverresponse
*=newPythonprocess
ProxyingRESTCalls:BasicConfiguraBonInweb.conf(Splunk6.2andup):
[expose:correlation_searches]pattern = alerts/reviewstatusesmethods = GET,POST
NotethattheURLiswhat’sactually“exposed”here.YoucanevenexposeCoreendpointsthataren’texposedbydefault.TheabovewouldcorrespondtoaURLof:
https://your_hostname:8000/en-US/splunkd/__raw/services/alerts/reviewstatuses
12
ProxyingRESTCalls:Wildcarding[expose:correlation_searches]
pattern = alerts/correlationsearches/*
methods = GET,POST
ThisexposesaURLof:https://your_hostname:8000/en-US/splunkd/__raw/services/alerts/correlationsearches/SEARCH_NAME_HERE
ButNOT:
https://your_hostname:8000/en-US/splunkd/__raw/services/alerts/correlationsearches
13
SplunkREST:DocumentaBonSplunkprovidesa(mostly)RESTfulAPI.ThisAPIisserveduponanyrunningSplunkinstance,usuallyonport8089,andiswell-documentedhere:RESTAPIReferenceManual–URIQuickReferenceh`p://docs.splunk.com/DocumentaBon/Splunk/latest/RESTREF/RESTlistRESTAPIUserManualh`p://docs.splunk.com/DocumentaBon/Splunk/latest/RESTUM/RESTusing
restmap.confh`p://docs.splunk.com/DocumentaBon/Splunk/latest/Admin/Restmapconf
14
ExtendingtheAPI:Why?QuesBon:WhywouldyouwanttoextendtheRESTAPI?
Answer(s):Severalreasons,mostofwhicharejustgeneralprinciplesofgoodsoRwaredesign.
1. EncapsulaBon
2. ComputaBon
3. FuncBonality4. AbstracBon
5. Performance
6. AppManagement
7. CloudCompaBbility
15
ExtendingtheAPI:EncapsulaBon
16
IntheEnterpriseSecurityapp,wefrequentlyencounterproductrequirementsthatrequireconstrucBonofanewconcept.Example:A“correlaBonsearch”consistsofupto3configuraBonobjects: Asavedsearches.confentry Metadataaboutthesearch’srelatedregulatorycompliancesewngsin“governance.conf”
Metadataaboutthesearch’sworkflowin“correlaBonsearches.conf”
EncapsulaBonbehindanAPIpermitsmanipulaBonoftheseenBBesasaunitor“singleconcept”.**Note:SplunkdoesNOTprovidetransacBonalsemanBcsonconfiguraBonfiles.
ExtendingtheAPI:ComputaBonCertaintypesofcomputaBonmightbeunsafetoperformsolelyinthebrowser.
Usually,thismeansargumentvalidaBon.
Example:
IfyoucreateacustomconfiguraBonfilethathasspecializedvalidaBonrequirements,acustomRESThandlertoprovideserver-sidevalidaBonmayberequired.
17
ExtendingtheAPI:FuncBonalityTheCoreSplunkRESTAPImaynotprovideacertainfeatureyouneed.
Example:
InanearlierversionofEnterpriseSecurity,inordertopropagatesomeconfiguraBonchangesacrossaSearchHeadCluster(SHC),wehadtowriteaRESThandlerthatwould“fanout”modificaBonsacrossaclustersothateditsmadeononesearchheadwouldbevisibleontheothersearchheads.
Thesecasesaregenerallyrare.Internally,wegenerallydon’tencouragedevelopmentofsignificant“plumbing”ofthissortattheapplevel,whenitshouldreallybedoneintheCoresplunkdprocess.
18
ExtendingtheAPI:AbstracBonYoumayneedtofuture-proofyourappbyprovidingalayerofabstracBon,sothatfuturemodificaBonstotheappcanbemadewithoutrequiringsignificantfront-endoruserexperiencework.
Example:
EScontainsasmallAPIthatprovidesforstorageofsmallfilesintheKVstoreasencodedstrings.BywriBnganAPIforthis,insteadofforcingthefront-endtowritetoKVstoreAPIs,weretaintheflexibilitytoswapoutthestoragelayeratanyBmewithoutrequiringsignificantUIwork.
19
ExtendingtheAPI:PerformanceOperaBonsthatwouldgeneratemanyround-tripstotheserver,oRenbenefitfrombeingwrappedinaRESTAPI.
Example:
EScontainsafeatureknownasthe“NotableEventFramework”whichoverlaysaminimalBckeBngworkflowsystemontopofindexedSplunkevents.EdiBngeventsviathisframeworkusuallyrequiresissuingmulBplecallstodeterminetheexisBngstatusandownershipofanevent,andthenvalidaBngthatthecurrentuserhaspermissiontochangethatstatus(forinstance:notallanalystsmaybeallowedto“close”incidents).
Doingthestatuscheckcompletelyinthebrowserwouldgeneratepossiblythousandsofcallstoandfromtheserver,whichwouldbeprohibiBvelyexpensive.
20
ExtendingtheAPI:AppManagement1. Usingthe“triggers”stanzainapp.conf,youhavetheabilitytoforceRESTcallstoyourhandlerto
occur(ornotoccur)uponappstatechanges(install,update,enable,disable).
2. Splunk’s“layeredconf”systemisaverysimpledatapersistencemechanism.Youcanusethiswhenyouneedtostoreabitofdataanddon’twanttoberestrictedtoindexingitandonlybeingabletogetatitviasearch.
Example:
IntheEnterpriseSecurityapp,weuBlizethistoforcethecustomertogothroughthesetupprocessagainfollowinganappupgrade,sothattheyreceivethenewestconfiguraBons.
[triggers]reload.ess_setup = access_endpoints /admin/ess_configured
21
ExtendingtheAPI:CloudCompaBbilityInSplunkCloud,youcan’tmakethesameassumpBonsaboutyourstorageorlocalenvironment.
Nordoesthecustomerhaveshellaccesstotheserver!
ThismeansthatanyoperaBonyouusedtodobyhandviadirecteditstoconfiguraBonfiles,orviaotherdirectfilesystemaccess,isbe`erdonebyexposingthefuncBoninaRESTAPI.
ThisisprobablythemostimportantreasontobeginuUlizingcustomRESThandlersinyourapp.
Cf.SteveYegge’sinfamousgoogle+rant:h`ps://plus.google.com/+RipRowan/posts/eVeouesvaVXontheimportanceofinterfacesastheypertaintoplazormdevelopment.
22
RESTAPIs
HowdoIwritethesethings?
23
RESTInterfacesYoumaybesurprisedtodiscoverthatSplunkoffers4disBnctmethodsforwriBngRESTAPIs,eachwithuniquebehavior.Theyareshownbelowontwoaxes:theinterfacethattheAPIiswri`enin,andthelife*meoftheprocessthatexecutestheRESThandlercode.
24
ProcessLifeUme
non-persistent persistent
Interface
EAI(admin_external) Allversions 6.4andup
Non-EAI(script) Allversions 6.4andup
RESTInterfaces:EAIEAI–ExtensibleAdministraUonInterfaceDesignedtofacilitatemorerapiddevelopmentofRESTinterfacesontheC++backend.UBlizingthisinterfaceprovidessomeaddiBonalservicessuchas:
• AutomaBcpaginaBon
• AutomaBcoutputformawng(XML,JSON)
• Accesscontrol• Filtering
• LimitedargumentvalidaBonviaSplunk“eval”syntax
EAIistypicallyassociatedwithmanagementofcustomSplunkconfiguraBonfiles.
25
RESTInterfaces:EAI(example)CustomEAIhandlersareindicatedbythepresenceofthe“admin_external”stanzainrestmap.conf.Thehighlightedparametersareonlyvalidwiththissewng.
[admin_external:correlationsearches]handlertype = pythonhandlerfile = correlationsearches_rest_handler.pyhandleractions = list,edit,create,remove,_reload
**OnlyPythonscriptsaresupported.
26
RESTInterfaces:MappingEAIHandlers[admin:alerts_threaBntel]
match=/alerts
members=correlaUonsearches
##CorrelaBonSearchesHandler
[admin_external:correlaUonsearches]
handlertype=python
handlerfile=correlaBonsearches_rest_handler.py
handleracBons=list,edit,create,remove,_reload
[eai:conf-correlaBonsearches]
capability.write=edit_correlaBonsearches
27
Mapsthehandler“correlaBonsearches”totheURI“services/alerts/correlaBonsearches”
Assignrole-basedaccesscontrolsonthehandler
Endpoint-specificsewngs
RESTInterfaces:scriptA“raw”interfaceforwriBngRESTinterfaces.
ServicessuchaspaginaBon,supportformulBpleoutputformats,etc.aretheresponsibilityofthedeveloper.ConformancetoRESTstyleisalsotheresponsibilityofthedeveloper.
Usingthisinterface,youhaveabsolutefreedom.
28
RESTInterfaces:script(example)A“script”handlerisindicatedbythepresenceofthe“script”sewnginrestmap.conf.Highlighteda`ributesareonlyvalidwiththistype:
[script:notable_update]match = /notable_updatescripttype = pythonscript = notable_update_rest_handler.pyhandler = notable_update_rest_handler.NotableEventUpdaterequireAuthentication = truecapability=edit_notable_eventsoutput_modes=json
29
RESTInterfaces:MappingScriptHandlers######RESTnotableupdate######
[script:notable_update]
match=/notable_update
script=notable_update_rest_handler.py
handler=notable_update_rest_handler.NotableEventUpdate
requireAuthenBcaBon=true
capability=edit_notable_events
output_modes=json
30
MapsthehandlertotheURI“services/notable_update”
Assignrole-basedaccesscontrolsonthehandler
Theclassthatservesrequests
RESTInterfaces:Segue:Whatis“Persistence?”Beforewecantalkabouthowtowritehandlers,weneedtounderstandtheotheraxisonourchart:whatis“persistence”?
RecalltheexecuBonmodelforaSplunkRESTcallonthepreviousdiagram:
1. Thesplunkdprocessreceivesrequestonport8089.
2. Thispythonscriptisinvoked:$SPLUNK_HOME/bin/pythonrunScript.py<setup|execute>
3. ThisscriptloadstheRESThandlerusingPython’sexecfile()method,handingoffSTDINandSTDOUTasneeded.
ItdoesthistwiceforeveryRESTcall:oncetosetuptheRESThandler,oncetoexecuteit.
That’stwoinvocaBonsofPythonforeveryRESTcall.
31
RESTInterfaces:Segue:Whatis“Persistence?”“Persistent”modemeansthatthesplunkdprocesswillonlyexecuteoneprocessperRESTcall.AddiBonally,thisprocesswillpersistunBlitisidleforaperiodofBme(60seconds),atwhichpointitwillbereapedbytheprimarysplunkdprocess(nodeveloperacBonrequired).Duringthenon-idleinterval,itcanservicemulBplerequests.
ThisistheexecuBonmodelforaSplunkpersistentRESTcall:
1. Thesplunkdprocessreceivesrequestonport8089.
2. Thepythonscriptisinvokeddirectly:$SPLUNK_HOME/bin/python<YOUR_SCRIPTHERE>persistent
(subsequentrequestsgetpassedtothesameprocessdirectly)
32
RESTInterfaces:HandlerBaseClassesPythonclassesaredistributedwithSplunkthatyoucaninheritfromtowriteyourownhandlers:
33
ProcessLifeUme
non-persistent persistent
Interface
EAI(admin_external) MConfigHandler MConfigHandler
Non-EAI(script) BaseRestHandler PersistentServerConnecBonApplicaBon
RESTInterfaces:HandlerBaseClassesPythonclassesaredistributedwithSplunkthatyoucaninheritfromtowriteyourownhandlers:
34
ProcessLifeUme
non-persistent persistent
Interface
EAI(admin_external) MConfigHandler MConfigHandler
Non-EAI(script) BaseRestHandler PersistentServerConnecBonApplicaBon
RESTInterfaces:AddingEAIModePersistenceQ:WhatdidwenoUceabouttheprecedingslide?
A:TheclassesprovidingEAIsupportarethesame!
That’scorrect:enablingpersistenceonacustomhandlerwri`enusingtheEAIspecificaBonissimplyaconfigura8onchange.ToaddpersistencetoanEAIhandler,simplyaddthistoyourrestmap.conf:
handlerpersistentmode = true
However…thisisnottosaythatyourhandlerisguaranteedtoworkproperly.Why?Ifyouweredoingworkinthe__init__()methodofyourhandler,andweredependingonthatworkbeingdonetoproperlyserverequests,wheninpersistentmodethisworkwillNOTberedone–because__init__()isnevercalledagain!
35
RESTInterfaces:AddingScriptModePersistenceEnablingpersistenceona“script”customRESThandlerrequires:
1.Addthistoyourrestmap.conf:
scripttype = persist
2.RewriteyourhandlertousethenewprotocolspecificaBon.Thisisthehardpart.
GoldstarquesBon:Persistentscriptsexecuteonlyonce.Whatdoesthisimplyfordiscoverability?
36
RESTInterfaces:ClassesEAI,persistentandnon-persistent:MConfigHandler
$SPLUNK_HOME/lib/python2.7/site-packages/splunk/admin.py
Script,non-persistent(twocompeUngimplementaUons):BaseRestHandler
$SPLUNK_HOME/lib/python2.7/site-packages/splunk/rest/__init__.py $SPLUNK_HOME/etc/system/bin/sc_rest.py
Script,persistent:PersistentServerConnecUonApplicaUon
$SPLUNK_HOME/lib/python2.7/site-packages/splunk/persistconn/application.py
37
RESTInterfaces:RecommendaBonsThenon-persistentinterfacesshouldbeavoided.
38
ProcessLifeUme
non-persistent persistent
Interface
EAI(admin_external) MConfigHandler MConfigHandler
Non-EAI(script) BaseRestHandler PersistentServerConnecBonApplicaBon
RESTInterfaces:RecommendaBons1. AvoidversionsofSplunkpriorto6.2sothatyoucanmakeuseofthe“expose”web.conf
direcBve.
2. Non-persistentinterfacesshouldbeavoidedunlessyourapprequirescompaBbilitywithpre-Splunk6.4versions.
ReasonsforrecommendaUon#2:
• Persistentinterfacesofferalltheflexibilityofthenon-persistentinterfaces.
• PerformanceofpersistentRESThandlersisvastlyimproved.
• “script”handlersusingnon-persistentmodecanactuallyconflictwithRESTscriptsrunninginunrelatedapps.
39
RESTInterfaces:RecommendaBonsAND…
• PersistentRESThandlerscannowbewri`enincompiledlanguagesusingthe“driver”direcBve:
[script:my_handler_written_in_go]match = /testdriver = echodriver.arg.1 = <whatever>script = echoscripttype=persistrequireAuthentication = trueoutput_modes=json
40
RESTInterfaces:SampleCodeSamplecodeforsimplisBcRESThandlersusingalltheinterfacesdetailedinthispresentaBon(includingtheill-advisedones)isavailableat:
h`ps://github.com/jrervin/splunk-rest-examples
41
RESTInterfaces:Demo
42
THANKYOU
