SplunkLive New York 2011: DealerTrack

New York City October 27, 2011

Copyright © 2011, Splunk Inc. New York, October 27, 2011

Chris DeMeo Sr. Director Technology Architecture


About Me

3

!   One of the founders of DealerTrack

!  With the company since incepJon

!   Technology Architecture Lead -‐ Infrastructure


About DealerTrack

4

!   Leading provider of on-‐demand soOware and services for the automoJve retail industry

!   Industry’s First and Largest Credit ApplicaJon Network

!   ConnecJng approximately 17,000 dealers with more than 1,000 lenders

!   Dealer Management SoluJons, Inventory SoluJons, Sales SoluJons and Processing SoluJons


The Challenge

5

•  Web logs, proprietary logs, database logs, operaJng system logs – No way to look through them easily, it took hours

•  Over Jme, acquired several companies resulJng in more dependencies and the need to consolidate logs across geo's and datacenters

•  In POC there was doubt about the value of the tool. How many hours would it take to get up and running? How quickly before we see value?

"We needed a tool that could correlate all logs."


The First of Many Aha Moments

6

•  Oracle RAC database cluster failover -‐ root cause analysis •  E-‐contracJng transacJon -‐ correlaJon of customer impacJng bug using 3 correlated log types

•  Database connecJons – custom input allowed trending in new ways

•  With Splunk ability to quickly get to root cause (assuming you logged something!)

•  Different from anything we used before •  Ability to show a developer the anatomy of a problem – transacJon tracing


The Second of Many Aha Moments

7

•  Capacity Planning – now has the ability to easily validate transacJon mixes and volumes

•  Capacity Planning – idenJfied “bunching” of transacJons during load tests

•  Increased product usage and customer retenJon – through idenJficaJon of their legacy product usage

•  Outage Impact – more accurate outage impact with access to addiJonal logs


What Also Helped…..

8

•  AOer 3 months execuJves felt Splunk paid for itself

•  Ease of Use • Development can build their own dashboards

• Business folks saw value and asked for informaJon

• We had Splunk up and running in less than 2 months own dashboards


Vale Add to Lending Partners

9

•  Real-‐Jme visibility into credit applicaJon and decision volumes across all lenders

•  Report to lenders where they stand against the compeJJon—drives more compeJJve response


Historical Data Search

10

!   Log view uJlity for our own proprietary logs… !   Our own tool doesn’t search over more than 7 days of data !   Now we can search historical data for trending analysis


Current Environment and Set up

11

!   Deployed across 4 applicaJon environments !   Non-‐Cloud based -‐ Run our own equipment !   Each environment has 1 server (will make into HA configuraJon)

!   Everything runs on a single server !   Virtual machines in non-‐producJon environments !   Local storage


Coming Soon

12

!   Syslog inputs !   HA configuraJon with mulJple search heads and SAN storage !   Looking to install Web AnalyJcs app for business users !   IntegraJon with other monitoring tools !   User adopJon-‐-‐work on user educaJon-‐-‐how they are using it !   Want to give Security group ability to search across years of data !   Rollout to subsidiaries so each has their own indexer

"Finding data in our logs that we never even knew was there and that people are ge8ng value from."


Lessons Learned

13

!   Before we didn't have the data, now we want to save everything !   Professional Services and EducaJon – They armed us and made us more than dangerous!

!   Developed internal user doc with best pracJces and use cases !   Knowledge of your logs will help to realize value quickly


Thank you

Chris DeMeo Sr. Director Technology Architect