Small Tools and ScriptsSmall Tools and ScriptsL3 link monitoring with Nagios

Ernst Heiriernst.heiri@switch.ch

TF-NOC, Brussels, 11. October 2011

Content

1. The goal2 Monitoring L3 links with Nagios2. Monitoring L3 links with Nagios3. Link discovery and Nagios configuration4. Generating an interactive graph for NagVisGe e a g a e ac e g ap o ag s

© 2011 SWITCH 2

Monitoring Router and L3 LinksL3 NetworkOSPF

Nagios: Polling Routers with ping and SNMP

© 2011 SWITCH 3

Nagios – Link monitoring

Nagios model:•A service object is assigned to one single host object (router)•A host is periodically checked by pingA host is periodically checked by ping•A service is checked periodically e.g. using SNMP•The piece of SW used for service checks is called ‘plugin’

But:Links are not directly supported in Nagios - a link depends on two hosts (routers)

Router 1Our work-around:Host: ‘virtual host’ 127.0.0.1 e.g. backbone-10GService: wrapper plugin check_multi

Interface 1: plugin check ifoperstatus

Router 2L2 Link

Interface 1: plugin check_ifoperstatus& Interface 2: plugin check_ifoperstatus& OSPF 1: plugin check_ospf& OSPF 2: plugin check_ospf Interface 1 Interface 2OSPF 1 OSPF 2

© 2011 SWITCH 4

Link discovery and Nagios configuration

BA1 ZH2PS2./nagios-switch-net-explorer start-host [SNMP-community]

SNMP:

1. discover all links:

PS2OSPF

SNMP:Recursive search for OSPF neighbors

Result file: backbone.linksswiBA1.switch.ch --> swiBA2.switch.ch link: swiBA1-10GE-3-1.switch.chswiBA1 switch ch > swiPS2 switch ch link: swiBA1 10GE 3 3 switch ch

BA2 EZ2swiBA1.switch.ch --> swiPS2.switch.ch link: swiBA1-10GE-3-3.switch.chswiBA1.switch.ch --> swiPS2.switch.ch link: swiBA1-G1-1.switch.chswiBA2.switch.ch --> swiBA1.switch.ch link: swiBA2-10GE-3-1.switch.ch…

2. Use file backbone.links to generate Nagios Configuration:./gen-nagios (-w)

Result files: Nagios configuration filesba1t3p1-ba2t3p1.cfgba1t3p1 ba2t3p1.cfgba1t3p1-ba2t3p1.cmdba1t3p3-ps2t3p1.cfgba1t3p3-ps2t3p1.cmd…Limitations:

f

© 2011 SWITCH 5

•multiple links between routers – need to be fixed manually•links which can’t be measured at all – are manually excluded

Nagios Configuration files - examplesresult file: ba1t3p1-ba2t3p1.cfgdefine service {

name BA1t3p1-BA2t3p1use backbone-link-checkcheck_command check_multi!-f /etc/nagios3/objects/lan/backbone/cfg/ba1t3p1-ba2t3p1.cmd

i d i ti BA1t3 1 BA2t3 1service_description BA1t3p1-BA2t3p1host_name backbone-10G

}

result file: ba1t3p1-ba2t3p1.cmdcommand[ BA1t3p1_if ] = /usr/lib/nagios/plugins/check_ifoperstatus -v 2 -D i -C hctiws -H BA1 -d TenGigabitEthernet3/1g g gcommand[ BA2t3p1_if ] = /usr/lib/nagios/plugins/check_ifoperstatus -v 2 -D i -C hctiws -H BA2 -d TenGigabitEthernet3/1command[ BA1t3p1_ospf ] = /usr/local/nagios/libexec/check_ospf -r BA1 -c hctiws -n 130.59.37.102:BA2command[ BA2t3p1_ospf ] = /usr/local/nagios/libexec/check_ospf -r BA2 -c hctiws -n 130.59.37.101:BA1

additional configs: backbone-link-checkd fi i {define service{

use backbone-generic-servicename backbone-link-checkservicegroups backbone-linksnotifications_enabled 1max_check_attempts 5

}

define host {use generic-hosthost_name backbone-10Galias bb-10G Links

© 2011 SWITCH 6

address 127.0.0.1}

Nagios classic Display

© 2011 SWITCH 7

Make an Interface for NagVis

BA1 ZH2PS2

link:backbone-10G:PS2t3p2-ZH2t3p3

link:backbone-10G:BA1t3p3-PS2t3p1

link:backbone-1GPS2 6 1 PS1 1 3

1. Draw a picture of your L2 Network e.g. with OmniGraffle

BA2 EZ2

PS2

link:backbone-1G:EZ2g6p1-PS1g1p3

PS2g6p1-PS1g1p3link:backbone-10G:EZ2t1p2-ZH2t1p2

link:backbone-10G:BA1t3p1-BA2t3p1

2. Add URLs to the links3. Export as HTML

BA2 EZ2

link:backbone-10G:BA2t1p4-EZ2t5p4

<map name="GraffleExport"><area shape=rect coords="165,11,192,23" href="ps2">

Result file: topology-L3-example.html

<area shape rect coords 165,11,192,23 href ps2 ><area shape=circle coords="179,17,13" href="ps2">

...</map><img border=0 src="topology-L3-example.png" usemap="#GraffleExport">

4. Convert it into NagVis configuration with Perl script extract_coordinates.pl./extract coordinates.pl topology-L3-example.html >topology-L3-nagvis

© 2011 SWITCH 8

./extract_coordinates.pl topology L3 example.html topology L3 nagvis

NagVis ConfigurationOutput of script: topology-L3-nagvis.pl

define global {header_menu=0}

….define service {host_name=backbone-1Gservice description=PS2g2p4 PS1g1p3

define shape { icon=topology-L3-example.jpgx=0y=0z=2}

service_description=PS2g2p4-PS1g1p3iconset=switch_linkx=169y=44z=3label_show=0}}

define host {z=1x=164y=2

}define service {host_name=backbone-10Gservice_description=BA1t3p3-PS2t3p1iconset=switch_linkx=93

host_name=ps2label_show=0iconset=switch_router}

define host {

y=8z=3label_show=0}

define service {{z=3x=164y=2host_name=ps2label_show=0iconset=switch router transparent

{host_name=backbone-10Gservice_description=PS2t3p2-ZH2t3p3iconset=switch_linkx=240y=8z=3

© 2011 SWITCH 9

iconset switch_router_transparent}…

z 3label_show=0}

Graphical Interface with NagVis

© 2011 SWITCH 10

Summary

• Monitoring links with Nagios• Discovery of L3 links in an OSPF network• Generating Nagios configuration of these links• Generating a graphical view for NagVis

Improvements:• Suppress link alarms if one of the two nodes is down• Suppress link alarms if one of the two nodes is down• More Infos about the links – e.g. bandwidth and traffic load• Eliminating manual tuning – e g double links between 2Eliminating manual tuning e.g. double links between 2

nodes• Code: More generalized, better structure

© 2011 SWITCH 11

Disclaimer and Credits

Disclaimers:•It’s designed for the architecture and nomenclature of SWITCH networkg•All code is available ‘as-it-is’ without support•The code is not nice – we know it – the idea is nice!

All credits to my colleagues and Fabian Mauchle and Ulrich Schmid

© 2011 SWITCH 12

All credits to my colleagues and Fabian Mauchle and Ulrich Schmid

Questions

?

© 2011 SWITCH 13