Log Analytics Optimization

Optimizing Log Analytics from the Edge

April 2016

© Hortonworks Inc. 2011 – 2015. All Rights Reserved

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

About HortonworksCustomer Momentum

~800 customers (as of Feb 10, 2016)Publicly traded on NASDAQ: HDP

Hortonworks Data PlatformCompletely open multi-tenant platform for any app and any dataConsistent enterprise services for security, operations, and governance

Partner for Customer SuccessLeader in open-source community, focused on innovation to meet enterprise needsUnrivaled Hadoop support subscriptions

Founded in 2011

Original 24 architects, developers, operators of Hadoop from Yahoo!

800+E M P L O Y E E S

1500+E C O S Y S T E M

PA R T N E R S


EMBRACE AN OPEN APPROACH

MASTER THE VALUE OF DATA

EVERY BUSINESS IS A DATA BUSINESS


DATA AT REST

DATA IN MOTION

ACTIONABLEINTELLIGENCE

MODERN DATA APPLICATIONS

Actionable Intelligence from Connected Data Platforms

Capturing perishable insights from data in motion

Ensuring rich, historical insights on data at rest

Necessary for modern data applications

Hortonworks DataFlow

Hortonworks Data Platform


Optimizing Log Ingest with Hortonworks DataFlow


Why Hortonworks DataFlow?

Because even the best data scientists and most powerful platforms need the right data to analyze


Store Data

Process and Analyze Data

Acquire Data

Perception of DataFlows: Easy, Definitive

Dataflow


Reality of Dataflows: Complex, Convoluted

Store Data

Process and Analyze Data

Acquire Data

Store DataStore Data

Store Data

Store Data

Acquire Data

Acquire Data

Acquire Data

Dataflow


HDF has 130+ Processors - Multiple for Log Analytics

HTTP

Syslog

Email

HTML

Image

Hash Encrypt

Extract

TailMerge

Evaluate

Duplicate Execute

Scan

GeoEnrich

Replace

ConvertSplit

Translate

HL7

FTP

UDP

XML

SFTP

Route Content

Route Context

Route Text

Control Rate

Distribute LoadAMQP


Log Analytics Systems Today

LOG ANALYTICS PLATFORMNetwork

Device Logs

• Not all data can be captured• Not all captured data is valuable• Transport all data


Cost Effectively Expand Storage Options of Log Data

LOG ANALYTICS PLATFORM

Network Device Logs

HDPHDF

3. Cost effectively expand collection and grow timescale of logs collected

2. Content-based routing based on dynamic evaluation of content, attributes, priority

1. Integrate and enrich logs across data centers and security zones


Efficiently Expand Log Ingestion from the Edge


Network Device Logs

HDF

HDF

HDF

HDPHDF

• Expand collection to new sources of machine data• Edge analytics to transform, enrich and prioritize content based routing• Capture and transport only valuable data


Expand Analytics and Reporting Options with HDP


Network Device Logs

HDF

HDF

HDF

HDPHDF

ODBC interface traditional BI tools

Easy access to log analytics data through traditional BI tools

Give data scientists better tooling – Spark, Storm etc


Expand to small scale, remote systems


Network Device Logs

HDF

HDF

HDF

HDPHDF


Optimize Log Analytics with Content Based Routing


Edge analytics for cost-effective and efficient movement of

machine data

HDF

Intelligent, content based routing, transformation

and enrichment

Send data to alternative systems based on value,

content, priority

HDP

HDF

HDF

HDF


Splunk Optimization:Using HDP as Data Refinery


Splunk Hadoop Connect

17

Reliable bi-directional integration

ImportBrowseExport

Splunk Hadoop Connect

>2000 downloads

HA Indexes and Storage

CommodityServers

Hadoop (MapReduce &

HDFS)

Report & analyze

Custom dashboards

Monitor and alert

Ad hoc search


Splunk, Hunk & Hortonworks

YARN Ready Partner

Certified on Hortonworks Data Platform

Existing Sandbox tutorial


Splunk, Part of the Modern Data Architecture

• Bi-directional data integration between Splunk & HDP

• Collect data from across the organization, deliver it to Hadoop for refining data and batch analytics

• Output of Hadoop jobs can be imported into Splunk Enterprise for rapid analysis and visualization

• Archiving from Splunk Enterprise to Hadoop


Splunk, Part of the Modern Data Architecture

• Bi-directional data integration between Splunk & HDP

• Collect data from across the organization, deliver it to Hadoop for refining data and batch analytics

• Output of Hadoop jobs can be imported into Splunk Enterprise for rapid analysis and visualization

• Archiving from Splunk Enterprise to Hadoop


Hunk + Hortonworks

21

Explore, analyze and visualize data in HDP from one integrated platform

Simply point Hunk at your HDP cluster(s) and start exploring data immediately

Search data, change perspectives and preview results as MapReduce jobs run

INTERACTIVE EXPLORATION

RICH DEVELOPER ENVIRONMENT

Build big data apps on data in HDP using standard web languages and frameworks

FULL-FEATUREDANALYTICS

FAST TO DEPLOY AND DRIVE VALUE


Augment Splunk Deployment with Hortonworks Data Platform

Heavy Indexer

Universal Forwarders

HDP Enables

Splunk Storage

• Expansion to more data than previously feasible• Archive data from Splunk into Hadoop• Query archived Splunk data in Hadoop• Focus Splunk infrastructure on what really matters


Find out how much you can optimize your log analytics infrastructure today.

Contact [email protected]

Technology

Log Analytics Optimization