Splunk> CSI:Logfiles

Splunk as a Shared Service

Geoffrey Martins Global Splunk Architect - ExxonMobil

2

Agenda

About ExxonMobil and Geoffrey Martins

Why Shared Service?

The Four Major Challenges

Final Unified Network

Potential Next Steps

Takeouts

Q&A

3

Largest International Oil & Gas Company in the World

75.000 employees worldwide

Presence in 100+ countries

2014 Numbers – Gross Income: 411 Billion Dollars – Net Income: 32 Billion Dollars

Worldwide support center in Brazil – Curitiba-PR – 1200 employees – 800 in IT only!

4

Geoffrey Martins Splunk Architect in Analytics E&D

– Live in Curitiba, Brazil; – 8 years with ExxonMobil;

.Net Developer SAP BW Consultant

– Masters Degree in Computing Sciences – PhD student at UFPR

5

Why Shared Service? • Scenario by end-2013

• Splunk first brought to the company in 2012

• Several independent Splunk networks for different departments

• Compartmentalized information • Duplicated data ingestions • Divergent reports coming from different

instances • Separate support teams and separate

development teams. • No standardization between instances. • No Dev/Sandbox environment.

6

Why Shared Service? • Challenge: Single Worldwide Splunk

Network • Aim for a single Splunk network • Explore Splunk’s main advantage: Data

sharing and collaboration • Optimize data acquisition, no duplicates. • Standardize development and developers,

all working in a single direction. • Make developers aware of each other • Share code, share ideas.

• Unify user base • Unify support

7

The Four Major Challenges:

> Unify Infrastructure

> Single User Base

> Solid Support Team

> The Massive Data Unification

8

Unify Infrastructure Gather all licenses in a single licensing server

Expand presence to all continents – Concentrate and transform data closer to the origin. – Indexers in Asia and Europe – Forwarders in Asia, Europe, Africa and South America.

Add power to Search Heads – Move from totally separate search heads to two main Search Head Clusters:

General Purpose CyberSecurity-Exclusive

Create a real region-based structure – Store data closer to origin. – Smaller transfers between sites.

9

Unify User Base Identify existing power users and form new ones – Create a real community of Splunk power users – Establish rules to form power users. Attend to three official Splunk courses

Establish a ownership process for data and apps – Each index must have a data owner – Each app must have an owner and a responsible power user.

Establish periodic power user meetings – Power Users know what each other is doing – Opportunity to showcase apps, questions help. – Exchange of ideas, use cases, etc…

10

A Solid Team Supportability Team

Centralized in one single IT team

Mix of In-House Apps and Splunk-provided solutions

In-house developed app for real-time health monitoring (Uber Admin)

Splunk and 3rd party apps for network and Universal Forwarder management.

– Distributed Management Console and SOS

– TA-ForwarderQuery

– FireBrigade, Deployment Monitor, UtilizationMonitor…

Train a support team and integrate into the community

Facilitate access to support and Splunk administrators

11

12

13

14

15

A Solid Development Environment Creation of a Development Network – 1 Search Head, 2 Indexers, 2 heavy forwarders. – Exclusive to Power Users and Admins – Change management process:

All development on dev network. Once app reach production quality, Admins move it to the production network. Exclusive allocation reserved to the Dev network.

Sandbox Environment – Single all-in-one server – No-man’s land, everyone can do anything – Area open for experiments/prototypying – Useful to state if Splunk is the right solution for the data.

16

The Massive Data Unification Bring all indexers together in a single indexer layer – Document content of all indexes and make them visible – Make users aware of all data available to them

Each department can benefit from data coming from other departments. The main cause for load duplication is UNAWARENESS of data.

– Only segregate data when necessary. Keep data Free! Company has strict rules for management and protection of information. Candidates for segregation: Private and/or Proprietary data.

Leverage Distributed Capabilities of Splunk! – Position your Indexers/Search Head strategically – Know your data! – Splunk runs on commodity hardware. Put it to use!

17

The Final Unified Network 4-node General Purpose SHC 1 Segregated Search Head 3 Deployment Servers 1 Licensing Server

30 Indexers: Most in US, Some in Europe and Asia

22 Heavy Forwarders All major sites, including Africa and South America

~6000 Universal Forwarders October: All 15.000 Servers

18

Potential Next Steps

Splunk Mobile App – Bring Splunk Accessibility to ALL Company Devices

Splunk MINT – Mobile Intelligence for In-House iOS Apps

Hunk – Proof of Concept for Hadoop

19

Take-outs on a Successful Shared Service

Leverage your power users, make them known – Awareness of each other is the key – Your power users are your greatest resource

Unify your network, make your data visible – Invest in documentation, know your data! – Bring all your data together, avoid segregation unless necessary – A development environment gives freedom and protects your Splunk network.

Keep a close eye in your network – Monitoring can let you find problems before they happen! – Splunk has superb monitoring capabilities: USE THEM! – Resiliency is cheap and essential. Be prepared. – Take retention periods very seriously!

Questions?

20