20040718-NOC-Fowler

8/13/2019 20040718-NOC-Fowler

1/70

NOC Tools TutorialLuke Fowler

Alert Mon & RSS Tools:Chris Small

Joint Techs, Columbus, OHJuly, 2004

http://tools.globalnoc.iu.edu/

8/13/2019 20040718-NOC-Fowler

2/70

Tools Overview

Open Source Home Grown

RANCIDMRTGRRDtoolSyslog-ngNagios

Syslog Analysis ToolSNAPPRouter ProxyWeather MapNagios Plugins /Config GeneratorAlert MonVisible BackboneFirewall Filter Viz.Network InformationDatabase

8/13/2019 20040718-NOC-Fowler

3/70

RANCID

Really Awesome New CIsco Differ

Puts router/switch configs into a CVSrepository. Sends e-mail messages showing router

config diffs.

Works on more than just Cisco devices!

8/13/2019 20040718-NOC-Fowler

4/70

MRTG

Multi Router Traffic Grapher Collects time-series statistics via SNMP,

graphs these data.

Collector process invoked by cron

Can store data in .log or RRD format Released under the GNU General PublicLicense (GPL)

Very popular!

http://www.mrtg.org/

8/13/2019 20040718-NOC-Fowler

5/70

RRDtool

Round Robin Database

Written by Tobi Oetiker, author of MRTG. Stores time-series data in a fixed-sized file.

When the file gets full the oldest samples

are discarded to make room for new data.

Does not do data collection -- you supplythe collector (MRTG or SNAPP might do

the trick)

http://rrdtool.eu.org/

8/13/2019 20040718-NOC-Fowler

6/70

RRDtool (contd)

Several data types: gauge, derive, counter,absolute

set the step of your RRD file to yourcollection interval

RRAs definitions dictate data retentionoptions. You choose the granularity of dataretention. Very flexible (eg. store 30-second maximum data for 6 months, store

5 minute averages for 4 years)

8/13/2019 20040718-NOC-Fowler

7/70

RRDtool (contd) RRDtool includes a tool to create graph images

of data stored in .rrd files.

Can consolidate data sources, apply RPN (reversepolish notation) expressions to items beinggraphed

Graphs lines, (solid) areas, and stackedareas.

Outputs .gif and .png

All RRDtool operations availalbe through Perllibrary RRDs, and C library.

8/13/2019 20040718-NOC-Fowler

8/70

syslog-ng

Sink for your syslogs

Stores syslogs in a user-defined format

Filter logs on regular expressions, and additionalfunctions like host, program field, facility code, etc.

Can forward syslogs on to another syslog server.

Can be used to store syslogs in a relational database.

We use syslog-ng along with our syslog analysis scriptsat the Global NOC.

http://www.balabit.com/products/syslog_ng/

8/13/2019 20040718-NOC-Fowler

9/70

Nagios

Open source network monitoring system.Formerly Netsaint.

Web-based user interface Flexible plugin based host/service check

execution. Nagios spawns small plugin

executables for each check.

Plugins are very easy to write.

Licensed under the GNU GPL

8/13/2019 20040718-NOC-Fowler

10/70

Service Detail

8/13/2019 20040718-NOC-Fowler

11/70

Status Summary

8/13/2019 20040718-NOC-Fowler

12/70

Alert History

8/13/2019 20040718-NOC-Fowler

13/70

IU Nagios Plugins Plugins written at IU, used to monitor a

variety of network services. Releasedunder the IU Open Source License.

Mostly written in Perl, BGP plugin writtenin C

ISIS OSPF MSDP

PIM BGPInterface up/

down

Cisco/JuniperCPU Load

TL1alarmmonitoring

http://www.sourceforge.net/projects/nagiosplugins/

8/13/2019 20040718-NOC-Fowler

14/70

IU Nagios Plugins

Get plugin usageby invoking theplugin with no

arguments

BGP pluginrequires NET-

SNMP libraries,other pluginsrequire Getopt::Std, Net::SNMP.

8/13/2019 20040718-NOC-Fowler

15/70

Configuring a new

plugin in Nagios

Define a command statement for each plugin

8/13/2019 20040718-NOC-Fowler

16/70

Configuring a Service

Check

Add a service check definition for each instance of aservice you wish to monitor.

8/13/2019 20040718-NOC-Fowler

17/70

Nagios at the Global NOC

We auto-generate much of our Nagiosconfig based on router/switch configs.

Network devices are polled viaJUNOScript, and configs stored byRANCID are analyzed to construct Nagiosconfig.

NOC Operators watch an Alert Mondisplay, which shows information about allpending alerts.

8/13/2019 20040718-NOC-Fowler

18/70

AlertMon

Centralized Alert Monitor Provides one location to see current alarms Allows the GlobalNOC to easily correlate

alarms and the status of the progressresolution

Documentation and Procedure Hub NOC Big Board

http://alertmon.grnoc.iu.edu/cgi-bin/alertmon.cgi

8/13/2019 20040718-NOC-Fowler

19/70

AlertMon Features

Visually and audibly flagging new alerts

Collapses alerts when parent is down Tickets, documentation, andprocedures are linked to each alert

Alert Filters (by administrativegrouping, alert type, priority)

8/13/2019 20040718-NOC-Fowler

20/70

AlertMon

8/13/2019 20040718-NOC-Fowler

21/70

AlertMon Workflow

New Alert on AlertMon

NOC uses documentation presented ineach alert to troubleshoot the problem

If problem is real and persistant the NOCclaims the message and opens a ticket

The NOC follows the escalationprocedures linked from the alert

8/13/2019 20040718-NOC-Fowler

22/70

AlertMon Design

SOAP is used to pass message frommonitoring server (Nagios) to data

collectors

XML and RSS Files are generated

CGI Front-End reads the XML file andgenerates a HTML version

Distributed Architecture

8/13/2019 20040718-NOC-Fowler

23/70

AlertMon

SystemArchitecture

8/13/2019 20040718-NOC-Fowler

24/70

Monitored Services

ICMP

ISIS and BGP Peering

Layer 1 Alarms CPU and Disk Thresholds

Process Monitoring TCP Performance

8/13/2019 20040718-NOC-Fowler

25/70

AlertMon -- Future

Features Monitoring servers other than Nagios XSL and Widget clients

GridCat , MonALISA and Gangliaintegration

RSS History

Better multiple monitoring server support Better filters and o tions in the clients

8/13/2019 20040718-NOC-Fowler

26/70

AlertMon URLsAlertMon CGI

AlertMon RSS

AlertMon Doc

http://alertmon.grnoc.iu.edu/cgi-bin/alertmon.cgi

http://alertmon.grnoc.iu.edu/alertmon.rss

http://alertmon.grnoc.iu.edu/doc

8/13/2019 20040718-NOC-Fowler

27/70

RSS Tools

Really Simple Syndication

Various GlobalNOC Tools generate RSSfiles

Can be used to create custompresentations of data

Correlation of events to possible causes ofperformance degradation

8/13/2019 20040718-NOC-Fowler

28/70

Abilene Backbone

Interface Changes Shows the interface changes taken from

router syslogs

Calculates duration of the outage

http://ndb1-blmt.abilene.ucaid.edu/bb-int-changes.html

8/13/2019 20040718-NOC-Fowler

29/70

Interface Changes

8/13/2019 20040718-NOC-Fowler

30/70

RSS Feeds (cont)

AlertMon -- http://alertmon.grnoc.iu.edu/alertmon.rss

Abilene Notifications -- http://ratt.uits.iu.edu/ab_notif.rss

More mailing lists RSS Feeds coming soon

8/13/2019 20040718-NOC-Fowler

31/70

Weathermap

8/13/2019 20040718-NOC-Fowler

32/70

Weathermap

uses MRTG .log or RRDtool .rrd files as input You configure a background image, locations of

arrows, and nodes, etc.

A perl script that is run via Cron will update yourtemplate image, producing an updatedweathermap graphic

Available for free to educational and non-profitorganizations.

SVG version of the Weathermap software isunder development.

Available at: http://tools.globalnoc.iu.edu/

8/13/2019 20040718-NOC-Fowler

33/70

Weathermap Configuration Set up file

paths, legendand clock

parameters,and scaleparameters.

8/13/2019 20040718-NOC-Fowler

34/70

link settings tell theweathermap where todraw arrows betweennodes.

area settings indicatewhere to place links todetailed graphs fornodes/arrows.

Weathermap Configuration

8/13/2019 20040718-NOC-Fowler

35/70

label settings tellthe weathermapwhere to placenodes, and what

captions to placenext to them.

Weathermap Configuration

R P

8/13/2019 20040718-NOC-Fowler

36/70

Router Proxy

8/13/2019 20040718-NOC-Fowler

37/70

Router Proxy

A web interface to router/switch showcommands.

Connects to devices using Telnet or SSH Users enter their command on a HTML

form, submit, and results are returned in

the lower HTML frame.

Router Proxy for Abilene at http://loadrunner.uits.iu.edu/~routerproxy/

abilene/

8/13/2019 20040718-NOC-Fowler

38/70

Router Proxy: Commands On our router proxy implementations, we allow:show ip, show ipv6, show interface, show controller, show

route-map, traceroute, ping, show version, show environment,show atm, show proc, show bgp, mtrace, show msdp, showpim, show multicast, show route, show chassis, show policy,

show isis

You set up your router proxy instance to allowor deny any commands you choose.

We disallow certain expressions that could take alot of processor time on the router (e.g. the |operator, so a user cannot do something such aspiping output to a complicated match

expression)

8/13/2019 20040718-NOC-Fowler

39/70

8/13/2019 20040718-NOC-Fowler

40/70

Router Proxy Config

Global configurationparameters, such as theusername/password the

router proxy will use tologin to devices, paths tosupport files, headeroutput for the web formare set first.

The router sectionspecifies parameters foreach router proxy device.

8/13/2019 20040718-NOC-Fowler

41/70

Router Proxy Config

commandparameters specifyattributes for

router proxycommands.

rule parametersdefine regularexpressions toallow or disallowin user commands.

8/13/2019 20040718-NOC-Fowler

42/70

SNAPP

SNMP Network Analysis and PresentationPackage

Short-interval data collection for SNMP variables.

Stores data in RRDtool .rrd files Web front-end for viewing data

XML based configuration

Custom view groupings / custom graphing

Basic threshold reporting

Licensed under the GNU GPL SNAPP

8/13/2019 20040718-NOC-Fowler

43/70

SNAPP

8/13/2019 20040718-NOC-Fowler

44/70

SNAPP Data Collector

Multi-threaded, persistent data collector

Written in C

Talks to SNMP devices using the UCD-SNMPlibrary, writes data to RRD files using librrd.

Configuration information stored as XML

Changes made to collections via the front-endgenerate a HUP signal to the collector, causing itto re-load configuration data.

8/13/2019 20040718-NOC-Fowler

45/70

SNAPP Front-End

Web based CGIs, written in Perl Session support, logins can persist between

browser sessions, for up to 1 month.

Restrict administrative access by groupmembership

Collections and users may be members ofseveral groups

Flexible visualization options

8/13/2019 20040718-NOC-Fowler

46/70

SNAPP Collection Classes

Each collection isassociated with acollection class

Specifies SNMP OIDs tocollect, data retentionparameters, defaultgraphing behavior, datacollection interval, etc.

Supports graph mathvia RRD RPNexpressions

8/13/2019 20040718-NOC-Fowler

47/70

SNAPP: Adding Collections

Select a collection class Select collection-specific

parameters (SNMP host,

community string, groupaccess, threshold override,etc.)

Click Add Link -- a newRRD file will be created, theSNAPP config will beupdated, and the collectorwill reload its configuration.

8/13/2019 20040718-NOC-Fowler

48/70

SNAPP: Custom Views

Allows users to group a set ofgraphs together (e.g. allcollections on a single router)

Default graphs for custom viewmembers are shown together ona single page

View can be created based ongroups, or individual collections

Anyone can see and createcustom views, admins can edit/remove.

8/13/2019 20040718-NOC-Fowler

49/70

SNAPP: Custom Graphs

Allows custom graphsto be generated based

on start and end date/time

Allows a user to selectwhich SNMP variablesto show / hide

8/13/2019 20040718-NOC-Fowler

50/70

Syslog Scripts

Gather syslogs using the syslog-ng tool

We separate out logs into one file per

network (filter by facility code)

Each log file gets processed by the syslogscripts, generating an e-mail containing all

interesting log entries for the past 24 hours

Requires Perl with Config::INIfiles

8/13/2019 20040718-NOC-Fowler

51/70

Syslog Scripts

S l S i t C fi

8/13/2019 20040718-NOC-Fowler

52/70

Syslog Scripts Config

The syslog-ng.templatefile is used to generate asyslog-ng.conf file

active elements areenclosed in < > tags

Configure log file paths,ownership, and filtering(e.g. by facility)

S l S i t C fi

8/13/2019 20040718-NOC-Fowler

53/70


The config sectioncontains group and facilitynames for the sysloggroup

The email sectioncontains recipients ofsyslog reports

The hosts sectioncontains parameters foreach device we arereceiving syslogs from


8/13/2019 20040718-NOC-Fowler

54/70


The filters section is used todescribe which log entries toreport

Any entry that matches anitem listed in exclude will notget reported

Any entry that matches anitem listed in include will get

reported

An entry with a priority levellisted in priority will getincluded (if not in exclude list)

Vi ibl B kb

8/13/2019 20040718-NOC-Fowler

55/70

Visible Backbone

Visible Backbone collects information from Juniperrouters using JUNOScript, Junipers XML/RPCinterface

Presents a set of web pages with various views ofdata collected

Access to archived data is available through a SOAPinterface

Requires Perl, the JUNOScript API, XML::DOM,XML::Simple, and File::Copy

Consists of an XML collector, and a set of scripts toprocess each type of XML data

Vi ibl B kb

8/13/2019 20040718-NOC-Fowler

56/70

Visible Backbone

V bl B kb C fi

8/13/2019 20040718-NOC-Fowler

57/70

Visible Backbone Config

There are twoconfiguration files:.rtr and

.cmd

The .rtr file containsa list of routers to

collect data from,the .cmd file a list ofcommands toexecute.

8/13/2019 20040718-NOC-Fowler

58/70

Visible Backbone Config

Finally, you supply theXML collector with ausername andpassword to login toyour router(s) with.Search for login in

xml-collector.pl andreplace the login andpassword values

Fi ll Fil Vi li i

8/13/2019 20040718-NOC-Fowler

59/70

Firewall Filter Visualization

Collect, store, and graph time series data ofcounters from Juniper firewall filters

Data collector, invoked via Cron, usesJUNOScript to collect firewall filtercounters from a set of Juniper routers

Aggregates per-interface counters on a per-

router and per-network basis

Displays sets of graphs on a series of webpages

Firewall FilterVisualization

8/13/2019 20040718-NOC-Fowler

60/70


Firewall FilterVisualization

8/13/2019 20040718-NOC-Fowler

61/70

All data stored in RRD files, RRDtool usedto produce graphs In use on Abilene core nodes

We have a filter applied inbound onconnector interfaces, counting bits andpackets matching a variety of protocols/

ports

We are also using firewall filters to collectIPv6 statistics


Network Information

8/13/2019 20040718-NOC-Fowler

62/70

Currently under development

Repository for contact, equipment, circuit,monitoring, and topology data

Relational database with a web based front-end

Drives configuration of monitoring / measurement /management systems

Various auto-population scripts talk to networkdevices, collecting data to affect database state

Consolidates monitoring/measurement systemconfigurations in one central repository

Network Information

Database

Organizational Entities

8/13/2019 20040718-NOC-Fowler

63/70

Organizational Entities

Record details pertaining toan organizational entity as awhole

Vendors, customers,internal contacts

Each entity has one ormore contacts associatedwith it.

Entities also are associatedwith agreements (eg.membership agreement)

Contacts

8/13/2019 20040718-NOC-Fowler

64/70

Contacts

Contact Management: Contacts have an

arbitrary number ofcontact methodssuch as Office phone,email, cell phone, etc.

Contact methods aretime sensitive

Contacts may beassociated with morethan one entity

Nodes

8/13/2019 20040718-NOC-Fowler

65/70

Nodes

Node Management:

Records node data,such as IP address, linksto monitoringconfiguration, rackelevation, links todevices sectioncontaining physicalhardware data, etc.

Devices

8/13/2019 20040718-NOC-Fowler

66/70

Devices

Devices Management:

Records attributes of physicaldevices such as serial number,

slot number, hardware andsoftware versions, device type,etc.

Represented as a N-ary treeof arbitrary depth (e.g. JuniperT640 chassis is a root devicewith child FPC2, which inturn has children OC192 PICin slot 2/3/0 and OC48 PICin slot 2/3/1 )

POPs

8/13/2019 20040718-NOC-Fowler

67/70

POPs

Records POP facilityowner, address, longitude/latitude, CLLI code,

manned/unmanned status,etc.

Linked to node section,showing nodes whichreside in this POPlocation

Linked to rack section,showing locations of baysinside the POP.

8/13/2019 20040718-NOC-Fowler

68/70

IU Open Source License

Many of our tools are licensed under theIU Open Source License.

Very similar to the BSD license. You are free to use, modify, and distribute

our code, subject to a few conditions.

You must credit Indiana University whendistributing our code, or derivative code. A full copy of the license is included with

each tool.

SourceForge Projects

8/13/2019 20040718-NOC-Fowler

69/70

SourceForge Projects

Software Releases

Discussion Forums

Bug Reporting CVS Repositories

SourceForge links at http://tools.globalnoc.iu.edu/

Get Global NOC tools

8/13/2019 20040718-NOC-Fowler

70/70

http://tools.globalnoc.iu.edu/

Get Global NOC tools

at:

Documents

20040718-NOC-Fowler