If you can't read please download the document
Upload
nagios
View
718
Download
1
Embed Size (px)
Citation preview
PowerPoint Presentation
NRDS:
Enhancements
Mike Weber
NRDS: Introduction
Basic Functionality
Passive checks with Active Config Updates
Multiple Configurations
Group Check Categories
Create Multiple Check Times
Store-and-Forward
Create timestamps to maintain data in disconnects.
Basic Functions
Passive Checks
Active Configuration Updates
Multiple Configurations
Multiple Group Checks
Grouping checks based on type provide more efficient management
aspects. (Linux_OS, Oracle, Apache, etc.)
Multiple Check Times
Multiple check times lessens the load on the server and provides
best practice use of resources. (5 min., 30 min.,etc.)
Multiple Download Options
NRDS can be used to download, packages, plugins,
scripts,etc.
Multiple Configurations
Why Multiple ConfigurationsReduce Load on Server
Logs Analyzed for Summaries
daily, weekly, monthly
Low Tech Skills
Limited Host Access
Multiple Configurations
Spreading the Load on a Busy Servercommand[Calls: Daily Guest]=/usr/lib/nagios/plugins/daily_guest.sh 2amcommand[Calls: Daily Summary]=/usr/lib/nagios/plugins/daily_summary.shcommand[Calls: Daily Voicemail]=/usr/lib/nagios/plugins/daily_voice.sh2.10amcommand[Daily Web Consol Use]=/usr/lib/nagios/plugins/check_web_consolcommand[Calls: Daily Room]=/usr/lib/nagios/plugins/daily_room.sh2.20amcommand[Calls: Daily Room-Room]=/usr/lib/nagios/plugins/daily_room_room.sh2.30amcommand[Calls: Daily Admin]=/usr/lib/nagios/plugins/daily_admin.sh2.35amcommand[Calls: Daily Admin-Guest]=/usr/lib/nagios/plugins/daily_admin_guest.sh2.45amcommand[Calls: Daily Admin-Room]=/usr/lib/nagios/plugins/daily_admin_room.sh2.55amcommand[Calls: Daily Admin-VoiceM]=/usr/lib/nagios/plugins/daily_admin_voicemail.sh3.05amcommand[Calls: Weekly Guest]=/usr/lib/nagios/plugins/weekly_guest.sh 2.40am on Mondaycommand[Calls: Weekly Voicemail]=/usr/lib/nagios/plugins/weekly_voice.sh2.50am on Mondaycommand[Calls: Weekly Room]=/usr/lib/nagios/plugins/weekly_room.sh3am on Monday
Multiple Configurations
calls/min/ave/maxCalls: Daily Guest2013-09-05 12:31:46from_guest:1054/0/18/446Calls: Daily Room2013-09-01 20:07:23from_room:127/0/15/194Calls: Daily Room-Room2013-09-01 20:07:23from_room_to_room:2/0/0/1Calls: Daily Voicemail2013-09-01 20:07:23from_guest_to_voicemail:24/3/15/56Calls: Monthly Guest2013-09-05 03:20:26from_guest:1054/0/18/446Calls: Monthly Room2013-09-05 03:41:04from_room:127/0/15/194Calls: Monthly Room-Room2013-09-05 03:51:08from_room_to_room:2/0/0/1Calls: Monthly Voicemail2013-09-05 03:30:38from_guest_to_voicemail:24/3/15/56Calls: Weekly Guest2013-09-01 20:07:23from_guest:2108/0/18/446Calls: Weekly Room2013-09-05 04:00:00from_guest_to_voicemail:24/3/15/56Calls: Weekly Room-Room2013-09-05 03:10:34from_room:127/0/15/194Calls: Weekly Voicemail2013-09-01 20:07:23from_guest:1054/0/18/446
Multiple Configurations
Multiple Configurations
Install Procedure
SSH into the hotel pbx and become root.sudo su
cd /tmp
wget --no-check-certificate -Oinstall.tar.gz
"https://192.168.1.1/nrdp/?cmd=nrdsgetclient&token=Gh78TFcD&configname=
Unpack the packagetar xzvf install.tar.gzInstall the packagecd
clients
./installnrds install 30Note, this install script will be used for
every hotel. It is going to install NRDS and download the package
for the rest of the installation.Any host called install in the
Unconfigured Objects should be removed, the hostname will be
changed in the next step.
Multiple Configurations
Install the Packagecd /tmp
tar xvf package64.tar
cd packageNow you will execute a script that will ask for the name
of the hotel server and the IP Address of the hotel server, be sure
these are absolutely correct as the script will change all of the
files to reflect the options entered.Enter the host when asked.Now
execute the install script../install.sh
Multiple Configurations
Purpose of package64.tarDownload All Plugins
Download All NRDS Groups
When the script executes it will detect and setup the proper
groups.
Download install.sh
Multiple Configurations
Multiple Configurations
Multiple Configurations
Group Configuration Files All Point to One Hostnameadmin_server.cfg admin_server.pl admin_server_updater.plapp_server.cfg app_server.pl app_server_updater.plcustom.cfg custom.pl custom_server_updater.ploracle_db_server60.cfg oracle_db_server60.pl oracle_db_server60_updater.ploracle_db_server.cfg oracle_db_server.pl oracle_db_server_updater.pl
Multiple Configurations
Admin_Server.cfgCONFIG_VERSION="0.2"CONFIG_OS="Linux"CONFIG_NAME="oracle_admin_server"URL="http://192.168.5.5/nrdp/"TOKEN="j5pj5adf89"PLUGIN_DIR="/usr/local/nagios/libexec"SEND_NRDP="/usr/local/nrdp/clients/send_nrdp.sh"TMPDIR="/usr/local/nrdp/clients/tmp"COMMAND_PREFIX=""LOG_FILE=""UPDATE_CONFIG="1"UPDATE_PLUGINS="1"
command[ADMIN]=/usr/local/nagios/libexec/check_url.pl
http://example.com:8000
command[Apps Listener]=/usr/local/nagios/libexec/check_procs -c 1:1
-a APPS_$NAGSID -C tnslsnr -u $USER
command[Apache Server]=/usr/local/nagios/libexec/check_http -H
example.com -p 8000 -w 5 -c 10
command[OACore Server]=/usr/local/nagios/libexec/check_procs -w 2:5
-c 1:5 -a name=oacore -C java -u $USER
command[OAFM Server]=/usr/local/nagios/libexec/check_procs -w 1:5
-c 1:5 -a name=oafm -C java -u $USER
Multiple Configurations
Admin_Server.plRequired Changes to Custom Groups
my $nrds_updater = "/usr/local/nrdp/clients/nrds/admin_server_updater.pl";
my $configfile = "/usr/local/nrdp/clients/nrds/admin_server.cfg";
Multiple Configurations
Admin_Server_updater.plRequired Changes to Custom
Groups
my $configfile = "/usr/local/nrdp/clients/nrds/admin_server.cfg";
Multiple Configurations
Install.sh#!/bin/bashCHECK_ORACLE_HEALTH_FILE=/usr/local/nagios/libexec/check_oracle_healthDEFAULT_CONFIG_FILE="/tmp/package64/nrds/custom.cfg"##### 64 Bit Nagios Monitoring Install #####arch=$(uname -m)if [ ${arch} == 'x86_64' ]; thenecho "64_bit Operating System Detected"yum install -y libaiorpm -Uvh /tmp/package64/rpms/oracle-instantclient11.2-basic-11.2.0.2.0.x86_64.rpmrpm -Uvh /tmp/package64/rpms/oracle-instantclient11.2-devel-11.2.0.2.0.x86_64.rpmrpm -Uvh /tmp/package64/rpms/oracle-instantclient11.2-sqlplus-11.2.0.2.0.x86_64.rpmecho "#####"elseecho "This is not a 64_bit Operating System, this script can only be used for 64_bit Systems"exitfi
Multiple Configurations
Install.sh##### Install Nagios Directories and Plugins
#####useradd nagios
mkdir -p /usr/local/nagios/libexec
cd plugins
cp check* /usr/local/nagios/libexec/
cp urlize /usr/local/nagios/libexec/
cp negate /usr/local/nagios/libexec/
chown -R nagios:nagios /usr/local/nagios/libexec/
chmod +x /usr/local/nagios/libexec/*
Multiple Configurations
Install.sh##### Install check_oracle_health #####cd ..cd supportexport ORACLE_HOME="/usr/lib/oracle/11.2/client64"export LD_LIBRARY_PATH="$ORACLE_HOME/lib"tar zxvf check_oracle_health-1.7.8.1.tar.gzcd check_oracle_health-1.7.8.1./configuremakemake installmkdir /var/tmp/check_oracle_healthchown -R nagios /var/tmp/check_oracle_healthecho "ORACLE_HOME=/usr/lib/oracle/11.2/client64"echo "LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib"cp /tmp/package64/support/check_oracle_health /usr/local/nagios/libexec
Multiple Configurations
Install.sh##### VNC Ports #####config_file=${DEFAULT_CONFIG_FILE}VNC_START_PORT=5900
vnc_ports=$(sudo netstat -anptu | grep 'vnc[[:space:]]*$'| tr -s ' '|cut -d ' ' -f4|cut -d ':' -f2)
i=0for port in ${vnc_ports} ; do : $((i++)) printf "command[Check VNC %02d]=/usr/local/nagios/libexec/check_tcp -H localhost -p %d -w 5 -c 8 -e\n" ${i} ${port} >> ${config_file}done
Multiple Configurations
Install.sh##### CPU Core Set Up #####arg0=2arg1=1.5arg2=1arg3=3arg4=2.5arg5=0.5cpu_nb=$(grep -c ^processor /proc/cpuinfo)arg0=$((cpu_nb * arg0))arg1=$(echo "scale=1;${cpu_nb} * ${arg1}" | bc)arg1=$(echo ${arg1} | sed -e 's/\.0$//')arg2=$((cpu_nb * arg2))arg3=$((cpu_nb * arg3))arg4=$(echo "scale=1;${cpu_nb} * ${arg4}" | bc)arg4=$(echo ${arg4} | sed -e 's/\.0$//')arg5=$(echo "scale=1;${arg5} + 2 * ${cpu_nb}" | bc)arg_list=$(printf "%s %s,%s,%s %s %s,%s,%s" \ "-w" ${arg0} ${arg1} ${arg2} \ "-c" ${arg3} ${arg4} ${arg5})echo "command[Check Load]=/usr/local/nagios/libexec/check_load ${arg_list}" >>\ ${config_file}
Multiple Configurations
Install.sh##### Set Hostname #####host=$(hostname)echo $hostecho "#####"echo "Use the hostname above as a base as you type the server name"read nameecho "#####"oldString="txb"newString="$name"dpath="/tmp/package64/nrds/*.cfg"sed -i "s/${oldString}/${newString}/g" /tmp/package64/crontabfor i in $dpath; dosed -i "s/${oldString}/${newString}/g" $idone
Multiple Configurations
Install.sh##### Determine Linux Version #####version=$(lsb_release -rs|cut -f1 -d.)echo $version
if [ $version == '6' ]; thenecho "version 6"yum install -y glibc\* perl-YAML perl-Time-HiReselif [ $version == '5' ]; thenecho "version 5"yum install -y glibc\* perl-YAMLrpm -ivh /tmp/package64/rhel5/perl-Time-HiRes-1.9724-1.el4.rfx.x86_64.rpmelseecho "This install script will not work for RHEL 4"exitfi
Multiple Configurations
Install.sh##### Manage Nagios Environement #####echo 'NAGPATH=/usr/local/nagios/libexec; export NAGPATH' >> /home/nagios/.bash_profileecho 'LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib; export LD_LIBRARY_PATH' >> /home/nagios/.bash_profileecho 'ORACLE_HOME=/usr/lib/oracle/11.2/client64; export ORACLE_HOME' >> /home/nagios/.bash_profileecho 'TNS_ADMIN=/home/nagios; export TNS_ADMIN' >> /home/nagios/.bash_profileecho 'PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/nagios/bin:$ORACLE_HOME/bin; export PATH' >> /home/nagios/.bash_profileecho 'HOME=/home/nagios; export HOME' >> /home/nagios/.bash_profile
Multiple Configurations
Install.sh##### Oracle DB Plugins Install #####cp /tmp/package64/nrds/oracle_db* /usr/local/nrdp/clients/nrds/
Multiple Configurations
Install.sh##### App Server Install #####APP_SERVER=$(ps -ef|grep tnslsnr|grep -v grep|wc -l)if [ ${APP_SERVER} == '1' ]; thenecho "Installing Application Server"cp /tmp/package64/nrds/oracle_app* /usr/local/nrdp/clients/nrds/sed -i 's/\#\*\/5/\*5/' /tmp/package64/crontabelseecho "No Application Server Installed...no plugins will be installed for App Server"fi
Multiple Configurations
Install.sh##### Admin Server Install #####ADMIN_SERVER=$(ps -ef|grep FNDLIBR|grep -v grep|wc -l)if [ ${ADMIN_SERVER} == '1' ]; thenecho "Installing Admin Server"cp /tmp/package64/nrds/oracle_admin* /usr/local/nrdp/clients/nrds/sed -i 's/\#\*\/6/\*6/' /tmp/package64/crontabelseecho "No Admin Server Installed...no plugins will be installed for Admin Server"fi
Multiple Configurations
Install.sh##### Move New Configs #####cd /tmp/package64/nrdscp nrds* /usr/local/nrdp/clients/nrds/cp custom* /usr/local/nrdp/clients/nrds/chown nagios:nagios /usr/local/nrdp/clients/nrds/*
##### Install Cron Jobs #####/tmp/package64/crontab
exit
Multiple Configurations
Crontab Script#!/bin/bashcrontab -u nagios -rcrontab -u nagios -l | { cat; echo "### Cron Jobs for nagios ######## OS Metric Checks and Oracle Server Checks #####*/5 * * * * /usr/local/nrdp/clients/nrds/nrds.pl -H txb 2>&1*/4 * * * * /usr/local/nrdp/clients/nrds/custom.pl -H txb 2>&1*/6 * * * * /usr/local/nrdp/clients/nrds/oracle_db_server.pl -H txb 2>&1#*/5 * * * * /usr/local/nrdp/clients/nrds/oracle_app_server.pl -H txb 2>&1#*/6 * * * * /usr/local/nrdp/clients/nrds/oracle_admin_server.pl -H txb 2>&101 * * * * /usr/local/nrdp/clients/nrds/oracle_db_server60.pl -H txb 2>&1"; } | crontab -u nagios -exit
Store-and-Forward
Basic Problem
Network connections broken for multiple days.
Solution: NRDS Timestamps
Using timestamps NRDS is able to rebuild data and graphing on the
Nagios server.
Store-and-Forward
Example: Broken Connection
Network connections breaks 5:17 AM.
Store-and-Forward
Example: Broken Connection
Network connections breaks 5:17 AM.
Store-and-Forward
Example: Broken Connection
Network connections restored 6:30 AM.
Store-and-Forward
Example: Broken Connection
Network connections restored 6:30 AM.
Store-and-Forward
Example: Broken Connection
Network connections breaks 2:20 PM.
Store-and-Forward
Example: Broken Connection
Network connections breaks 2:20 PM.
Store-and-Forward
Example: Broken Connection
Network connections restored 6:26 AM (16 hours).
Store-and-Forward
Example: Broken Connection
Network connections restored 6:26 AM (16 hours).
Store-and-Forward
NRDS ClientAdd a cache line in /usr/local/nrdp/clients/nrds/nrds.cfgCACHEDIR="/usr/local/nrdp/clients/cache"
Create folder /usr/local/nrdp/clients/cacheThe cache needs to
collect timestamped data in order to rebuild the information
later.mkdir /usr/local/nrdp/clients/cache
chown nagios.nagios /usr/local/nrdp/clients/cache
chmod 0775 /usr/local/nrdp/clients/cacheTimestamps Being
Createddrwxr-xr-x 2 nagios nagios 4096 Sep 20 07:39 mylock
-rw------- 1 nagios nagios 1742 Sep 20 07:24 nrdp.6pvqRR1IJ
-rw------- 1 nagios nagios 1742 Sep 20 07:30 nrdp.B6qVekdsa
-rw------- 1 nagios nagios 1742 Sep 20 07:36 nrdp.biayklzbM
-rw------- 1 nagios nagios 1742 Sep 20 07:33 nrdp.CuPu74NX6
Store-and-Forward
NRDS ClientReplace /usr/local/nrdp/clients/send_nrdp.sh original if [ "x$url" == "x" -o "x$token" == "x" ] echo "Usage: send_nrdp -u url -t token"
Reference to cache directory if [ "x$url" == "x" -o "x$token" == "x" -o "x$directory" == "x" ] echo "Usage: send_nrdp -u url -t token -D directory"
Store-and-Forward
NRDS ClientReplace /usr/local/nagios/libexec/process_perfdata.pl chown nagios.nagios /usr/local/nagios/libexec/process_perfdata.plchmod 0755 /usr/local/nagios/libexec/process_perfdata.pl
if ( $NAGIOS{PERFDATA} =~ /\s*__check_timestamp__=(\d+)/ ) { $NAGIOS{TIMET} = $1; print_log( "Found encoded timestamp ($NAGIOS{TIMET}) in $NAGIOS{HOSTNAME} / $NAGIOS{SERVICEDESC} ($NAGIOS{PERFDATA})", 1 ); # remove encoded timestamp $NAGIOS{PERFDATA} =~ s/(\s*__check_timestamp__=\d+)//; print_log( "Perfdata now is: ($NAGIOS{PERFDATA})", 2 ); 218a231,232 } print_log( "Processing Line [$_]", 2 );
Script Downloads
NRDS Modifications:
Beginlinuxservers.com/nagiosconf
User: nagiosconference
Password: 54TBwh9
Only available during conference.
Conclusion
Mike Weber
Click to edit Master title style
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
9/22/14
Click to edit Master title style
Click to edit Master subtitle style
9/22/14
Click to edit Master title style
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
9/22/14
Click to edit Master title style
Click to edit Master text styles
9/22/14
Click to edit Master title style
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
9/22/14
Click to edit Master title style
Click to edit Master text styles
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Click to edit Master text styles
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
9/22/14
Click to edit Master title style
9/22/14
9/22/14
Click to edit Master title style
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Click to edit Master text styles
9/22/14
Click to edit Master title style
Click to edit Master text styles
9/22/14
Click to edit Master title style
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
9/22/14
Click to edit Master title style
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
9/22/14
PRESENTATION TITLE
Presenter Name
9/22/14
Click to edit Master title style
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
9/22/14
PRESENTATION TITLE
Presenter Name
9/22/14
Click to edit Master title style
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
9/22/14
Click to edit Master title style
Click to edit Master text styles
9/22/14
Click to edit Master title style
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
9/22/14
Click to edit Master title style
Click to edit Master text styles
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Click to edit Master text styles
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
9/22/14
Click to edit Master title style
9/22/14
9/22/14
Click to edit Master title style
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
Click to edit Master text styles
9/22/14
Click to edit Master title style
Click to edit Master text styles
9/22/14
Click to edit Master title style
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
9/22/14
Click to edit Master title style
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
9/22/14