Sadayuki FuruhashiTreasuare Data, Inc.Founder & Software Architect

How we use Fluentdin Treasure Datafor system metrics

A creator of Fluentd

Self-introduction

> Sadayuki Furuhashi> twitter/github: frsyuki

> Treasure Data, Inc.Founder & Software Architect

> Open source projectsMessagePack - efficient serializer (original author)

Fluentd - event collector (original author)

Metrics collection from Rails apps

fluentdRails app

fluentd

fluentdRails app

MetricSense for Ruby

aggregate logs usingout_forward plugin

local fluentd collects logsand buffer them in local disk

class ApplicationController before_filter :setup_metrics after_filter :post_metrics

def setup_metrics MetricSense.clear! MetricSense.tag "#{params[:controller]}_#{params[:action]}" if current_user.present? MetricSense.segment( :account=>current_user.account_id, :agent=>request.env[‘HTTP_AGENT’]) end end

def post_metrics MetricSense.measure! end

# in application code: # MetricSense.value :size=>10

Metrics collection from Hadoop

fluentd

HadoopCluster

HadoopClusterhadoop metrics plugin

(original plugin)

✓ number of failed tasks✓ number of used slots✓ blacklisted nodes✓ memory usage of JobTracker✓ memory usage of NameNode/DataNode✓ HDFS put/get latencyetc...

Metrics collection from DB

fluentd

script in_exec pluginruns a script every1 hour

runs SQL statements

✓ users’ activity and stats✓ job queue length (PerfectQueue stats)etc...

Metrics collection from Hadoop

fluentdRails app

fluentdRails app

fluentd

HadoopJobTracker

HadoopJobTracker

script

Metrics utilization

fluentd

Librato Metricsfor realtime analysis

Treasure Datafor historical analysis

Pager Dutyfor alerts

out_tdlogout_metricsense

Next version of Fluentd

> <filter>> <match> in <source>> <label> for error records> New process model & Live restart

<source> type tail path /var/log/httpd.log format apache tag not_filtered.apache</source>

<match not_filetered.**> type rewrite remove_prefix not_filtered <rule> key status pattern ^500$ ignore true </rule></match>

<match **> type forward host log.server</match>

Before

Mysterious tag

tag operations

<source> type tail path /var/log/httpd.log format apache tag apache</source>

<filter **> type rewrite <rule> key status pattern ^500$ ignore true </rule></filter>

<match **> type forward host log.server</match>

After (v11)

Filter plugins!

<source> type tail path /var/log/httpd.log format apache tag apache

<filter **> type rewrite <rule> key status pattern ^500$ ignore true </rule> </filter></source>

<match **> type forward host log.server</match>

After (v11)

<filter>/<match> in <source>

<source> type tail path /var/log/httpd.log tag apache</source>

<match **> type forward host log.server</match>

Before

I want to add flowcounter here...

<source> type tail path /var/log/httpd.log tag apache</source>

<match flow.traffic> type forward host traffic.server</match>

<match **> type copy <store> type flowcounter tag flow.traffic </store>

<store> type forward host log.server </store></match>

Before

Nested!

<source> type tail path /var/log/httpd.log tag apache</source>

<filter **> type copy <match> type flowcounter tag flow.traffic <match> type forward host traffic.server </match> </match></filter>

<match **> type forward host log.server</match>

After (v11)

Filtering pipeline

<source> type forward</source>

<filter **> type copy <match> type file path /mnt/local_archive </match></filter>

<label alert> <match **> ... </match></label>

<label analysis> ...</label>

# copy & label & forward<filter **> type copy <match> type forward label alert host alerting.server </match></filter>

# copy & label & forward<filter **> type copy <match> type forward label analysis host analysis.server </match></filter>

After (v11)