Writeyourown monitoring Motivation ... - RRDtool · RRDtool TobiasOetiker Motivation AboutRRDtool Programming Summary Writeyourownmonitoring softwarewithRRDtool TobiasOetiker ISG.EE-ETHZürich

Write your ownmonitoring

software withRRDtool

Tobias Oetiker

Motivation

About RRDtool

Programming

Summary

Write your own monitoringsoftware with RRDtool

Tobias Oetiker

ISG.EE - ETH Zürich

OSCON 2005Portland, Oregon, August 3, 2005



Tobias Oetiker

Motivation

About RRDtool

Programming

Summary

Netflow Data presented withFluxscope



Tobias Oetiker

MotivationThe world today

About data collection

A graph says more than1’000 words

About RRDtool

Programming

Summary

Content

MotivationThe world todayAbout data collectionA graph says more than1’000 words

About RRDtool

Programming

Summary



Tobias Oetiker




About RRDtool

Programming

Summary

Numbers everywhere

I 15 users onlineI 1.34 PB free tape spaceI 31’223 ifInOctetsI 100 degrees in the server

roomI 87 Mb/s transfer rate



Tobias Oetiker




About RRDtool

Programming

Summary

All these questions

I Are we running out ofdiskspace?

I Is there a pattern in the highserver load?

I Is there really globalwarming?

I Can we do something aboutit?

I Collect that data!



Tobias Oetiker




About RRDtool

Programming

Summary

How to collect data

I collect everything, analyzelater (aka never)

I data with known propertiesmakes life simpler

I from research to production



Tobias Oetiker




About RRDtool

Programming

Summary

Requirements for a solution

I handle counters and gaugesI be nice to the serverI setup for specific taskI data-pre processingI and forget about it . . .



Tobias Oetiker




About RRDtool

Programming

Summary

Condor use at UW



Tobias Oetiker




About RRDtool

Programming

Summary

Making data accessible

I graphicalrepresentation

I eye candy for thepeople

I analysis at a glanceI make my boss look

good to his boss



Tobias Oetiker




About RRDtool

Programming

Summary

Data post-processing

I highlight interestingproperties

I combine data fromseveral sources

I add extra Information



Tobias Oetiker

Motivation

About RRDtoolHistory

Features

Graphing features

Programming

Summary

Content

Motivation

About RRDtoolHistoryFeaturesGraphing features

Programming

Summary



Tobias Oetiker

Motivation


Features

Graphing features

Programming

Summary

1995: MRTG was only a start

I logfiles in text formatI gnuplot for graphsI 1996 - MRTG is used

for amazing tasksI more performanceI more flexibility



Tobias Oetiker

Motivation


Features

Graphing features

Programming

Summary

The rrd TOOL

I a building blockI basis for a better

MRTG (2nd system)I basis for hundreds of

other toolsI Google says: “industry

standard”



Tobias Oetiker

Motivation


Features

Graphing features

Programming

Summary

The Round Robin Database

I lossy storage, fixed file sizeI current data is the most

interestingI consolidation functions for

feature extractionI artificial data-sourcesI holt-winters aberrant behavior

detectionI constant step sizeI fixed disk space



Tobias Oetiker

Motivation


Features

Graphing features

Programming

Summary

Feeding Data

I combat jitter by takingacquisition time intoaccount.

I preserve data-volumeI on-the-fly data

validationI deal with unknown

dataI time is a one way

street



Tobias Oetiker

Motivation


Features

Graphing features

Programming

Summary

Basic Graphing

I auto scalingI auto labelingI sensible defaultsI quick resultsI anti-aliased outputI multiple formats

(png/eps/pdf/svg)



Tobias Oetiker

Motivation


Features

Graphing features

Programming

Summary

Advanced Graphing

I change colors, fonts,sizes

I data from severaldatabases

I data processing withRPN math

I alpha transparency(rrggbbaa)

I support of localebased characterencoding

I truetype fonts



Tobias Oetiker

Motivation

About RRDtool

ProgrammingLanguage bindings

Basic operations

Advanced Graphing

Summary

Content

Motivation

About RRDtool

ProgrammingLanguage bindingsBasic operationsAdvanced Graphing

Summary



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

On the command line

Just another Unix command.coderrdtool cmd arg

examplerrdtool info demo.rrd



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Via STDIN

Save startup time by feeding several commands.codeecho cmd arg | rrdtool -

exampleecho info demo.rrd | rrdtool -

I several commandsI fast operationI simple interface



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

perl module

I love coding in perl . . .codeuse RRDs;my $return = RRDs::cmd arg;

exampleuse RRDs;use Data::Dumpermy $ret = RRDs::info "demo.rrd";print Dumper $ret;



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Other Bindings

I RRDcgiI Perl Pipes - RRDpI PythonI TclI and more externally



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Creating Round Robin Databases

I what data do I want to look at?I how long do I want to keep it?I what are the interesting properties of my data?I what to keep in one rrd file?

Commandrrdtool create filename [–start start time] [–step step][DS:ds-name:DST:heartbeat:min:max][RRA:CF:xff:steps:rows]

DST: COUNTER, GAUGE, DERIVE, ABSOLUTE,New in 1.2: COMPUTECF: AVERAGE, MIN, MAX, LAST,New in 1.2: Aberrant Behavior Detection.



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Create Example

examplerrdtool create example.rrd --step=60\

DS:in:COUNTER:600:0:1000 \RRA:AVERAGE:0.5:1:100 \RRA:AVERAGE:0.5:10:100 \RRA:MAX:0.5:10:100

I resolution 60 secondsI COUNTER data sourceI updates at least every 600 secondsI accept rates between 0 and 1000I three archive with 100 entriesI store unknown unless half are known.



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Adding Data into a Round RobinDatabase

Commandrrdtool update filename [–template ds:ds:. . . ]time:value:value. . . [. . . ]

I data value and acquisition timeI timestamps have to increase with every updateI instead of time use N for nowI templates are for redundancy only



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Update Example

examplerrdtool update example.rrd \

--template in N:39344



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Data re-bin-ing

300s 300s

1

3

2280koctets

400s

Data Aquisition

700 octets/sec

300s 300s

Data Storage



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Data Consolidation

AVERAGE RRA with 3 PDP

AVERAGE RRA with 2 PDP

RRA with 1 PDP



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

The first graph

Commandrrdtool graph graph.png DEF:var=rrdfile:DS-name:CFLINE1:var#rrggbb:label

1. prepare data (DEF)2. draw graph (LINE1)



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Data processing with RPN(remember the HP days)What if you have collected octetsbut want to present bits?Calculated time series and variablesCDEF:var=RPN expression

Examples

a,8,* a ∗ 8 (bit to byte conversion)a,b,+ a + ba,b,c,IF if (a!=0) then b else ca,1800,TREND half-hour sliding window averagea,b,c,LIMIT if (a>b AND a<c) then a else ’UNKN’



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Data processing with RPN

Calculated single value variableVDEF:var=RPN expression

var is associated with a time and a data value.Examples

a,TOTAL rate multiplied with intervala,AVERAGE average value of aa,95,PERCENT find 95-percentile



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

MRTG like Graph

Examplerrdtool graph graph.png \

DEF:in=data.rrd:input:AVERAGE \DEF:out=data.rrd:output:AVERAGE \AREA:in#ff0000:incoming \LINE2:out#00ff00:outgoing



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Graph with marked maximum

Examplerrdtool graph graph.png \

DEF:a=data.rrd:a:AVERAGE \VDEF:max=a,MAXIMUM \LINE2:a#00ff00:outgoing \LINE1:max#ff0000:maximum\\g \VRULE:max#ff0000 \GPRINT:max:"at %.2lf"



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Graph with marked maximum



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Graph with 95 percentile

Examplerrdtool graph.png \

DEF:in=data.rrd:input \DEF:out=data.rrd:output \CDEF:base=in,out,MAX \VDEF:n95=base,95,PERCENT \AREA:in#0000b0:incoming \AREA:out#00ff0080:outgoingLINE1:base#000000:maximum \LINE1:n95#ff0000:"95%-tile "\GRPINT:n95:"at %.2lf %s"



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Graph with 95 percentile



Tobias Oetiker

Motivation

About RRDtool


Basic operations

Advanced Graphing

Summary

Mailserver Statistics



Tobias Oetiker

Motivation

About RRDtool

Programming

Summary

Content

Motivation

About RRDtool

Programming

Summary



Tobias Oetiker

Motivation

About RRDtool

Programming

Summary

Summary

I carefully design yourRRD structure

I use RPN math to beatyour data into form

I write your ownmonitors, RRDtooldoes the boring stuff

I look good

More information on . . .http://www.rrdtool.org

http://www.rrdtool.org

Documents

Writeyourown monitoring Motivation ... - RRDtool · RRDtool TobiasOetiker Motivation AboutRRDtool Programming Summary Writeyourownmonitoring softwarewithRRDtool TobiasOetiker ISG.EE-ETHZürich