Deployment of PerfSONARMotivation (3/3) The 26th APAN Meeting , Queenstown, 5Aug 2008 We’d like to make it easy to discovery the available network data on other networks and see

Deployment ofperfSONAR

The 26th APAN Meeting , Queenstown, 5 Aug 2008

Takatoshi Ikeda,APAN-JP NOC/

KDDI R&E Labs/NICT

John Hicks,TransPAC2/

Indiana University

The 26th APAN Meeting , Queenstown, ５ Aug 2008

Contents

1. Introduction

2. Specifications

3. Installation procedure


IntroductionSpecifications

Installation procedure

Motivation (1/3)

The APAN Tokyo XP and TransPAC2 NOC support many demonstrations and experiments over multi-domain networks throughout the year. However, there are some inefficiencies with the current support model.


APANTokyoNW1

NW2

We’d like to make it easy to discover the available network data on other networks and see it on standardized format.

Checking the network data of other networks

It’s not easy to get the network data of all networks and see its graphs1. Search the web page for the data2. Check what data is available3. Recognize the different format of the graphs

and data.

Inefficiency of support for demos and experiments

Motivation (2/3)

Preparation for network performance measurement


We’d like to make it easy to discover the available performance measurement servers on the other networks and do measurement immediately

APANTokyo NW1

Measurement

Iperf?

Yes

IP & port?

Information

When

Date and Time

time

MuchTime !

It takes so much time to do performance measurement 1. Check the available measurement servers

and tools on other networks2. Exchange the information about the

measurement servers3. Coordinate the measurement schedule4. Exchange the measurement data

Motivation (3/3)


We’d like to make it easy to discovery the available network data on other networks and see it on standardized format.

We’d like to make it easy to discovery the available performance measurement servers on the other networks and do measurement immediately

What we can do

Efficient support for events Looking at whole network data over multi-domain network

efficiently Measuring the network performance immediately between

domains

Advanced monitoring Monitoring the multi-domain network Integrated monitoring with several network metrics

Contribution to research activity Providing network data in quasi-real-time while researcher is

doing experiment. Providing measurement data to people who research the

routing /path control



Introduction

SpecificationsInstallation procedure

Environment

Deployment point APAN Tokyo XP TransPAC2 LA

perfSONAR software v0.9 perfSONAR-PS software package


www.jp.apan.net/noc/

TokyoXPTransPAC2 LA

TransPAC2

JP-LA 10G

perfSONAR Services



Lookup

Topology

Authentication

Transformation

Resource Protector

Measurement Point

Measurement Archive

Interface Usage

Delay

Throughput

Lookup Service

PerfSONAR Services Deployment Services

resource registration and discovery for the

perfSONAR services

Measure throughput and expose its data

Measure delay and expose its data

expose Interface Usage data

1

2

3

4

PerfSONAR-PS v0.9 Packages

Lookup Service

Topology Service

Circuit Status Measurement Archive

Status Measurement Archive

perfSONAR-BUOY

PingER Services

SNMP Measurement Archive

Server Allocation

Server Allocation & Install Packages



No

Service PerfSONAR-PSPackage

HostnameAPAN TP2

1 Lookup Service(LS)

Lookup Service ps1.jp.apan.net 134.68.142.46

2 InterfaceUsage(Usage)

SNMP MA nms2.jp.apan.net nms7.transpac2.net

3 Throughput PerfSONAR-BUOY nms1.jp.apan.net nms1.transpac2.netnms6.transpac2.net

4 Delay PerfSONAR-BUOY nms4.jp.apan.net nms4.transpac2.netnms6.transpac2.net

Hardware Specification

Hardware (APAN Tokyo XP)



No Service Spec1 LS Barebone SuperMicro SuperServer 6015X-TV

CPU Intel Xeon QuadCore 2.66GHz *2Memory DDR2-667 1GB * 2Disk SATA 500GB * 2 (RAID1)NIC On-board 10/100/1000 * 2

234

UsageThroughputDelay

Barebone SuperMicro SuperServer 6013PCPU Intel Xeon 3.06GHz *2Memory DDR266 1GB * 2Disk SAS 73GB * 2 (RAID1)NIC On-board 10/100/1000 * 2

Software Specification (1/2)

OS & Kernel (APAN Tokyo XP)



No Service OS Kernel

1 LS CentOS 5.1 x86_64(Linux)

2.6.18-53.1.21.el5

2 Usage

CentOS 5.1 i386(Linux)

2.6.24.7 + web100-2.5.19

3 Throughput

2.6.18-53.1.21.el54 Delay

Software Specification (2/2)

Measurement Tools



No Service Tool Description Ver.2 Usage RRDtool Database which store traffic

data1.2.23

SNAPP(or Cacti)

Gathering usage data of routers by SNMP

2.0

3 Throughput Iperf Throughput Measurement 2.0.2

BWCTL Wrapper tool for Iperf 1.2a

4 Delay OWAMP One-way delay measurement

3.0c

Network Configuration



APANTokyoXP

10G Backbone

nms1(Throughput)

nms2(Usage)

nms4(Delay)

ps1(Lookup)

1 GbECopper

AlaxalaAX7808S

Dell PowerConnect

5324

1 GbECopper

1 GbECopper

TransPAC2LA POP

10G Backbone

HP ProCurveSwitch 3400cl

JuniperT640

JuniperT320

TransPAC2US-JP

10Gbps

nms7(usage)

134.68.142.46(LS)

nms6(Throughput

& Delay)


IntroductionSpecifications

Installation procedure

Installation Report

Document

Installation Report v0.9http://www.jp.apan.net/noc/perfSONAR/installation-report-v0.9.doc


OS & Basic setting



OS ＆ Basic Setting （Brief)

OS Basic Installation for the server Select “Server-GUI” at “set of software” section Add some packages for development SELinux is “Disabled”

Basic Setting Update the installed packages Stop unnecessary services(daemons) Change run level Configure TCP Wrapper Configure Perl CPAN



OS installation（All host)

1. Download the disk image from CentOS Web page below and make an install disk.

2. CentOS Download -http://www.centos.org/modules/tinycontent/index.php?id=15

3. Insert the install disk you created to CD or DVD drive then boot the server.

4. Set parameters depends on your environment within install wizard[Items of install wizard]

– Language – arbitrary setting – Keyboard – arbitrary setting – Disk Partition – arbitrary setting– Network – setting depend on your environment– Time zone – arbitrary setting– Root Password – arbitrary setting– Package – see next slide



Package （All host)

Select [ Server-GUI ] and check [ Customize now ] .

Select following packages as additional packages.



Group PackageApplication Development Libraries

Development Tools

Legacy Software Development

Basic setting （All host)

You do basic setting after package install and rebooting.• Firewall – default setting• SElinux – Select [disabled] (To avoid unexpected

access denied) • Time – arbitrary setting• User account – arbitrary setting• Sound card – arbitrary setting• Reboot the server



Package update (All host)

A lot of initial packages are needed to update due to security and functional reason. We manage RPM packages by using yum

Set up for yum# yum -y install yum-fastestmirror

Add new repository, DAG# rpm --import http://dag.wieers.com/rpm/packages/RPM-

GPG-KEY.dag.txt# vi /etc/yum.repos.d/dag.repo

(see installation report)

Update the packages# yum -y update



Unnecessary Linux Services

Stop Unnecessary Linux ServicesTo improve security level, save system resource and avoid the

performance problems, turn off unnecessary Linux services

# /sbin/service <SERVICE> stop & /sbin/chkconfig<SERVICE> off

Unnecessary services for perfSONAR-PS are belowavahi-daemon, bluetooth, cups, hidd, ipmi, irqbalance, lvm2-monitor, nfslock, pcscd, xfs, isdn, setroublesboot



Run level & TCP Wrapper

Run levelChange the run level to turn off X window.

# vi /etc/inittabid:3:initdefault:

TCP WrapperFor control the access.

# vi /etc/hosts.allowALL : < Allowed host># vi /etc/hosts.denyALL : ALL



CPAN

perfSONAR-PS packages are available at CPAN. We install them by using CPAN.

So, configure the CPAN

# cpanWe have to reconfigure CPAN.pm due to following uninitialized parameters:following messages are only question. Explanation messages are snipped.Are you ready for manual configuration? [yes] CPAN build and cache directory? [/root/.cpan]<snip>

Select your continent (or several nearby continents) [] choose the number to select prefer mirror site.



Lookup Service



Lookup Service (server #1)

Install software



Software Version Description

dbxml 2.3.10

an embedded XML database with XQuery-based access.This is used to store the service registrationinformation. .

perfSONAR_PSLookup Service

0.9 perfSONAR_PS Lookup Service

Lookup Service (Brief)

1. Oracle XML Database• Install dbxml (2.3.10)

2. Perl module for Lookup Service• perfSONAR_PS::Services::LS::LS

3. PerfSONAR_PS Daemon• Configure perfSONAR_PS Daemon by using

configuration utility “psConfigureDaemon”



dbxml

perfSONAR-PS stores the service registration information to Oracle Sleepycat XML Database.

# wget –c http://download.oracle.com/berkeley-db/dbxml-2.3.10.tar.gz

# gzip -dc dbxml-2.3.10.tar.gz | tar xf -# cd dbxml-2.3.10# ./buildall.sh --enable-perl --prefix=/usr/local/dbxml-

2.3.10<snip># echo "/usr/local/dbxml-2.3.10/lib" >

/etc/ld.so.conf.d/dbxml.conf# /sbin/ldconfig



perl-rrdtool

perl-rrdtool is perl module for rrdtool. This module is necessary for perfSONARconfiguration tool (psConfigureDaemon)

# yum -y install perl-rrdtool



perfSONAR_PS::Services::LS::LS

The software packages for perfSONAR_PSLookup Service are available at CPAN

# cpanTerminal does not support AddHistory.cpan shell -- CPAN exploration and modules

installation (v1.7602)ReadLine support available (try 'install

Bundle::CPAN')cpan> install perfSONAR_PS::Services::LS::LS



perfSONAR_PS::Services::LS::LS(cont.)

Configuration utility is prepared for PerfSONARDaemon. This utility tool is an interactive mode

# psConfigureDaemon<snip>What file should I write the configuration to?

[/etc/perfsonar/daemon.conf]: 1) Add/Edit endpoint2) Enable/Disable port/endpoint3) Set global values4) Save configuration5) Exit? []: xx

See Installation Report




Run the perfSONAR-PS daemon for Lookup Service

# mkdir /etc/perfsonar/xmldb# /sbin/service perfsonar-daemon start/usr/bin/perfsonar --config /etc/perfsonar/daemon.conf --piddir=/var/run --pidfile=perfsonar-daemon.pid --logger=/etc/perfsonar/logger.conf2008/07/11 00:06:15 (2421) WARN> LS.pm:148 perfSONAR_PS::Services::LS::LS::init - Setting 'reaper_interval' to '0'./etc/init.d/perfsonar-daemon start: perfsonar-daemon started




Check Psping

psping is check tool whether perfSONAR services is available or not.psping can be installed by CPAN

# cpancpan> install perfSONAR_PS::Client::Echo

# pspinghttp://<HOSTNAME>:8080/perfSONAR_PS/services/LSService http://localhost.localdomain:8080/perfSONAR_PS/services/LS is up-Time to make request: 0.26802



Interface Usage



Interface Usage (Brief)

1. RRDtool + SNAPP or Cacti• Install RRDtool• Install SNMP agent for Interface Usage

– SNAPP or Cacti

2. SNMP MA Module• perfSONAR_PS::Services::MA::SNMP • Configure perfSONAR_PS SNMP MA

– store.xml

3. PerfSONAR_PS Daemon• Configure perfSONAR_PS Daemon by using configuration

utility “psConfigureDaemon”



Interface Usage (server #2)

Install software



Software Version Description

RRDTool 1.2.23

the OpenSource high performance data loggingand graphing system.Interface usage data is stored to this tool’s formatdatabase.

NetSNMP 5.3.1-24Free SNMP tool.This tool is used to collect MIB information ofinterface usage from network equipments.

MySQL 5.0.45-7Open source database.The configuration for Cacti is stored to MySQL

Cacti 0.8.7b-2RRDTool frontend tool which has RRDTool's datastorage and graphing functionality

perfSONAR_PSSNMP MA

0.9 perfSONAR_PS SNMP Measurement Archive

RRDtool & NetSNMP

RRDtoolTo store the interface usage data, install the RRDtool

and related libraries# yum -y install rrdtool.i386 rrdtool-devel.i386

Net-SNMPInstall Net-SNMP utilities. Cacti gathers the data by

SNMP# yum -y install net-snmp-utils



MySQL

InstallInstall MySQL and related packages.# yum -y install mysql-server mysql-devel

RunRun MySQL server # /sbin/service mysqld start<snip>Starting MySQL: [ OK ]

PHP ModuleInstall PHP module for MySQL# yum -y install php-mysql



Cacti

Cacti gathers the interface usage data by SNMP and stored it to RRDtool format database. This data in RRDTooldatabase is published by perfSONAR-PS SNMP MA.

Install# yum -y install cacti cacti-docs

DB settingSet up mySQL DB for Cacti.# mysqladmin --user=root create cacti# mysql cacti < /var/www/cacti/cacti.sql# mysql --user=root mysql<snip>mysql> grant all on cacti.* to cacti@localhost identified

by ‘<PASSWORD>’ ;mysql> flush privileges;



Cacti (cont.)

Cacti setting for DBModify configuration file for Cacti # vi /var/www/cacti/include/config.php$database_username = "cacti";$database_password = "<PASSWORD>";

Poller.phpCheck whether the script (poller.php) which gather the data

works or not.# php /var/www/cacti/poller.phpsh: -q: command not foundsh: -: command not found^C*If above messages appeared and no response after

executing command, it’s working. Please exit by press Ctrl-C



Cacti (Cont.)

CronSet up cron for gathering the data periodically.# crontab -u cacti -e*/5 * * * * /usr/bin/php /var/www/cacti/poller.php >

/dev/null 2>&1

HTTPDCacti can be configured by WEB interface. Also, Cacti

publish the traffic graphs on the WEB. So, turn up HTTP server.

# vi /etc/httpd/conf.d/cacti.confallow from <ALLOW_HOST>

# /sbin/service httpd startThe 26th APAN Meeting , Queenstown, 7 Aug 2008


Cacti (Cont.)

setting for CactiBasic setting for Cacti can be set up by WEB Interface.

Access to HTTP server from client.http://<hostname>/cacti/



perfSONAR_PS::Services::MA::SNMP

InstallThe software packages for perfSONAR_PS SNMP MA

are available at CPAN distribution system. Following is installation procedure from CPAN.

# cpan<snip>cpan> install perfSONAR_PS::Services::MA::SNMPCPAN: Storable loaded ok<snip>Shall I follow them and prepend them to the queue of modules we are

processing right now? [yes] yes* Answer “ yes “ for same question above.



perfSONAR_PS::Services::MA::SNMP(cont.)

Set up daemonConfiguration utility is prepared for PerfSONAR

Daemon. This utility tool is an interactive mode

# psConfigureDaemon<snip>What file should I write the configuration to? [/etc/perfsonar/daemon.conf]: 1) Add/Edit endpoint2) Enable/Disable port/endpoint3) Set global values4) Save configuration5) Exit? []: xx

See Installation Report




Set up SNMP MASet up the SNMP MA. # vi /etc/perfsonar/store.xml

Sample configuration is available at Install Report.See Install Report.




runrun the perfSONAR-PS daemon for SNMP MA# /sbin/service perfsonar-daemon start/usr/bin/perfsonar --config /etc/perfsonar/daemon.conf --

piddir=/var/run --pidfile=perfsonar-daemon.pid --logger=/etc/perfsonar/logger.conf

/etc/init.d/perfsonar-daemon start: perfsonar-daemon started




Check Psping

psping is check tool whether perfSONAR services is available or not.psping can be installed by CPAN

# cpancpan> install perfSONAR_PS::Client::Echo

# pspinghttp://<HOSTNAME>:8080/perfSONAR_PS/services/snmpMAService http://<HOSTNAME>:8080/perfSONAR_PS/services/snmpMA is up-Time to make request: 0.26802

* if SNMP MA is available, you can see above message.If not, you can see “down” instead of “up” on above message.




Check perfSONAR-UI



Throughput & delay

John-san show you.



Reference

perfSONARhttp://www.perfsonar.net/

perfSONAR-PShttp://www.internet2.edu/performance/pS-PS/

APAN-JP PerfSONARhttp://www.jp.apan.net/noc/perfSONAR/

TransPAC2http://www.transpac.org/

PerfSONAR UIhttp://monstera.man.poznan.pl/jra1-

wiki/index.php/PerfsonarUIThe 26th APAN Meeting , Queenstown, 7 Aug 2008


http://www.perfsonar.net/�

http://www.internet2.edu/performance/pS-PS/�

http://www.jp.apan.net/noc/perfSONAR/�

http://www.transpac.org/�

http://monstera.man.poznan.pl/jra1-wiki/index.php/PerfsonarUI�

http://monstera.man.poznan.pl/jra1-wiki/index.php/PerfsonarUI�

www.themegallery.com

Documents

Deployment of PerfSONARMotivation (3/3) The 26th APAN Meeting , Queenstown, 5Aug 2008 We’d like to make it easy to discovery the available network data on other networks and see