8000 INM SR6.0 Maintenance Procedures

8000 Intelligent Network Manager SR6.0Maintenance Procedures

76.8060-70211A11.07.2014

Document Information

© 2014 Tellabs. All rights reserved.

This manual is protected by U.S. and international copyright laws, conventions and treaties. Your right to use this manual issubject to limitations and restrictions imposed by applicable licenses and copyright laws. Unauthorized reproduction, modification,distribution, display or other use of this manual may result in criminal and civil penalties. The following trademarks and servicemarks are owned by Tellabs Operations, Inc. or its affiliates in the United States and/or other countries: TELLABS®, TELLABS®

logo, TELLABS and T symbol®, T symbol®, CORIANTTM.

Any other company or product names may be trademarks of their respective companies.

The specifications and information regarding the products in this manual are subject to change without notice. All statements,information, and recommendations in this manual are believed to be accurate but are presented without warranty of any kind,

express or implied. Users must take full responsibility for their application of any products.

Adobe® Reader® are registered trademarks of Adobe Systems Incorporated in the United States and/or other countries.

Tellabs and Coriant are joining forces. You may see references to Coriant or Tellabs when doing business with us. Contactinformation is available at http://www.coriant.com.

8000 Intelligent Network Manager SR6.0 76.8060-70211AMaintenance Procedures © 2014 Tellabs.

2

http://www.coriant.com

Document Information

Terms and Abbreviations

Term Explanation

DB Database

LAN Local Area Network

76.8060-70211A 8000 Intelligent Network Manager SR6.0© 2014 Tellabs. Maintenance Procedures

3


4

Table of Contents

Table of Contents

About the Manual .............................................................................................................. 7

Objectives....................................................................................................................................................................... 7Audience......................................................................................................................................................................... 7Related Documentation .................................................................................................................................................. 7Conventions.................................................................................................................................................................... 7Document Conventions .................................................................................................................................................. 8Product Name Rebranding ............................................................................................................................................. 8Documentation Feedback............................................................................................................................................... 8

1 Introduction ................................................................................................................. 10

2 Automatic Maintenance Procedures ......................................................................... 11

2.1 Enabling Automatic Database Backups .............................................................................................................. 12

3 Fault History Maintenance.......................................................................................... 13

3.1 Configuring Automatic Deletion of Fault History............................................................................................... 13

4 Daily Maintenance....................................................................................................... 14

4.1 Sybase SQL Server Daily Maintenance Tasks .................................................................................................... 144.1.1 General................................................................................................................................................. 144.1.2 Verifying Sybase Database Backups ................................................................................................... 144.1.3 Verifying Tape Backups....................................................................................................................... 154.1.4 Monitoring Size of Data Devices and Transaction Logs..................................................................... 154.1.5 Monitoring Disk Space Usage and Log Files...................................................................................... 164.1.6 Checking Disk Space........................................................................................................................... 164.1.7 Investigating and Removing Core Files from Solaris and Linux Database Server ............................. 174.1.8 Truncating Sybase Log Files on Windows Database Server ............................................................... 174.1.9 Truncating Sybase Log Files on Solaris and Linux Database Server.................................................. 174.1.10 Verifying Database Statistics Updating .............................................................................................. 18

4.2 Checking 8000 Intelligent Network Manager Servers and Communication Links............................................. 184.2.1 Checking 8000 Intelligent Network Manager Servers ........................................................................ 184.2.2 Checking Communication Links and Paths to All Network Nodes .................................................... 19


5

Table of Contents

5 Weekly Maintenance .................................................................................................. 20

5.1 Sybase SQL Server Weekly Maintenance Tasks ................................................................................................. 205.1.1 Verifying Status of Scheduled Procedures in Scheduling Center........................................................ 205.1.2 Verifying Sybase SQL Server Database Configuration Parameters .................................................... 215.1.3 Verifying Index Reorganization in 8000 Intelligent Network Manager Database .............................. 225.1.4 Verifying 8000 Intelligent Network Manager Database Consistency ................................................. 235.1.5 Testing Restoration of Database Backup............................................................................................. 24

5.2 Monitoring Database Performance Data Size ..................................................................................................... 265.3 Checking Disk Space of 8000 Intelligent Network Manager Computers (and Moving Log Files) .................... 265.4 Checking Disk Space on ThinClient Server ........................................................................................................ 27

6 Monthly Maintenance.................................................................................................. 28

6.1 Deleting Audit Log.............................................................................................................................................. 286.2 Monitoring Size of Fault Tables .......................................................................................................................... 286.3 Accounting Management Events......................................................................................................................... 306.4 Node Configuration Backup ................................................................................................................................ 30

Suggested Weekly Schedule for Automatic Maintenance Procedures........................ 32


6

About the Manual

About the Manual

Objectives

This manual provides instructions on the procedures that have to be executed on a regular basis inorder to maintain the 8000 Intelligent Network Manager system.

Audience

This manual is designed for system administration personnel.

Related Documentation

The document numbering scheme consists of the document ID, indicated by numbers, and thedocument revision, indicated by a letter. The references in the Related Documentation table beloware generic and include only the document ID. To make sure the references point to the latestavailable document versions, please refer to the relevant product document program on the Tellabsand Coriant Portal by navigating to www.portal.tellabs.com > Product Documentation > DataNetworking > 8000 Intelligent Network Manager > Technical Documentation.

8000 Intelligent Network Manager SR6.0Software Installation Manual (76.8060-70201)

Provides instructions on installing and configuring8000 Intelligent Network Manager in differentcomputer platforms (workstation, servers).

8000 Intelligent Network Manager SR6.0Third party Hardware and Software Requirements(76.8060-70222)

Describes all the needed third party softwarecomponents and versions (operating systems andsoftware versions) in the 8000 Intelligent NetworkManager system. Describes also the relevantrequirements and recommendations of the 8000Intelligent Network Manager system.

Replication Server Installation for Tellabs® 8000Intelligent Network Manager Resilience Service(76.8000-70196)

Provides instructions on how to install SybaseOpenSwitch Server for 8000 Intelligent NetworkManager Resilience Service hot standby. Thismanual should be used as a supplement togetherwith Sybase documentation for installing SybaseOpenSwitch Server.

Conventions

The following typographical conventions are used to help distinguish elements in the text in thismanual.


7

https://www.portal.tellabs.com/global/portal/home.jsp

About the Manual

Text Element Explanation

Bold Indicates the name of an item (window/dialog/button, etc.) you need to select.When more than one item needs to be selected in a step, the items to be selectedare separated by a dash (-).

Italics Normally italics is used when referring to another document.

Courier Indicates the contents of a file, a command or a part of a program.

Document Conventions

This is a note symbol. It emphasizes or supplements information in the document.

This is a caution symbol. It indicates that damage to equipment is possible if the instructionsare not followed.

This is a warning symbol. It indicates that bodily injury is possible if the instructions are notfollowed.

Product Name Rebranding

Product names are being rebranded and as a result the product name format consists of a numericalidentifier and a descriptive part.

Additionally, the previous Tellabs 7300 product family is renamed and the individual product namesare also changed. The table below lists previous and new product names. You may see instancesof both the previous and the new product names in the customer documents during the transitionperiod to the new naming system.

Previous Product Name New Product Name

Tellabs® 7300 metro Ethernet switching series 7090 Packet Transport Platform Series

Tellabs® 7305 7090-05 CE

Tellabs® 7307 7090-07 CE

Tellabs® 7310 7310

Tellabs® 7325 7090-25 CE

Tellabs® 7345 7090-45 CE

Documentation Feedback

Please contact us to suggest improvements or to report errors in our documentation:


8

About the Manual

Email: [email protected]


9

1 Introduction

1 Introduction

This document consists of instructional step lists for the procedures that must be executed on aregular basis to properly maintain the 8000 Intelligent Network Manager system.

The document contains Sybase database maintenance tasks.

In case of the NMRS configuration, refer to the Replication Server Installation for Tellabs® 8000Intelligent Network Manager Resilience Service document.

Please note that if these system maintenance procedures are not carried out, it may lead tototal management outage. If there is not a valid database backup available, system recoveryto operational state can take several days or even weeks, and in most cases requiresa massive amount of manual work by Coriant. This work is not covered by the SupportAgreement but is charged separately at an hourly rate.

The procedures are classified into four categories, as listed below.

1. What can be done automatically.

2. What should be done every day.

3. What should be done every week.

4. What should be done every month or whenever necessary.

These procedures are to be used with 8000 Intelligent Network Manager. In addition to theseprocedures, the normal maintenance tasks for the operating system should be performed. Forexample, events in the Event Viewer such as Application Log, Security Log and System Log shouldbe monitored on a regular basis and truncated before they become too large.


10

2 Automatic Maintenance Procedures


8000 Intelligent Network Manager provides a way to automate some database maintenance andbackup procedures that previously had to be done manually. Backing up the separate PM database issupported in the Scheduling Center.

The following database housekeeping procedures are automatically scheduled to run.

• Resourcelock Housekeeper at 15 minutes past every hour

• DB Server Housekeeper at 1 minute before every hour

• Server Monitoring Housekeeper at 23:30 PM of every day

• Delete PMS Raw Data at 02:00 AM of every day

• Daily DB Check at 04:15 AM of every day

• Daily DB Maintenance at 03:15 AM of every day. The following tasks are executed as a groupwithin 3.5 hour time frame.

• Rebuild Indexes

• Reclaim Unused Space

• Update Table Statistics

• Update Index Statistics

• Weekly DB Check at 00:15 AM of every Sunday. The following tasks are executed as a groupwithin 2.5 hour time frame.

• Check Table Consistency

• Check Index Consistency

The following maintenance tasks are optional and disabled by default in the Scheduling Center.

• WebWholeNetworkTrendDataCollectionSchedule

• CPU Average Load History

• Check PMS Faults

• Calculate PMS Faults

Keep the following in mind when scheduling tasks in the Scheduling Center.

• Refer to the guidelines provided in Appendix Suggested Weekly Schedule for Automatic Main-tenance Procedures for scheduling the various maintenance procedures.

• It is important for the maintenance procedures that they do not overlap. As the execution timesvary depending on the network factors such as size, these intervals need to be assessed and ad-justed after observing a few runs. They should also be periodically checked as mentioned in theweekly maintenance task. Refer also to chapter 5.1.1 Verifying Status of Scheduled Proceduresin Scheduling Center.


11


2.1 Enabling Automatic Database Backups

Daily and weekly maintenance procedures are activated by default. However, the database backupprocedures are disabled by default and those must be enabled if database backups will be takenwith Scheduling Center.

Keep the following in mind when scheduling database tasks in the Scheduling Center.

• The database backup procedures (Dump Database, Dump Transaction and Dump Master) shouldbe scheduled by the Scheduling Center.

• The suggested scheduling times and intervals of the database backup procedures are as follows:Full database dump at 21:00 every day; Database Transaction dump at 03:00, 07:00, 11:00, 15:00and 19:00 every day; Database Master dump at 19:30 every Sunday.

• If database backup procedures are being scheduled, the backup directory is defined usingbackupdir field in the nms database table. Default values are H:\backup\ for Windows and/sybase_backup/ for Solaris/Linux. Initially the backup directory is defined during the databasecreation or update. If the database backup directory needs to be changed later, it can be changedwith ISQL. Note that the Generic Scheduler Service (GSS) process on Management Serverand the Scheduling Center on the 8000 Intelligent Network Manager workstation need to berestarted to take effect after making the change to database.

• If the PM database is used, the PMS backup directory is defined using pmbackupdir field in thenms database table.

System Sybase SQL Server

Tool(s) to use Scheduling Center

Step 1 Make sure that the GSS is enabled and running on one or multiple Management Servers.

• If it is not enabled, shut down a Management Server, start the 8000 Intelligent NetworkManager installation program with the reconfigure option and select also the SchedulingService from the list of services to be installed.

Step 2 Login with superuser privileges to the Dashboard.

Step 3 Check that the following privileges are given to superuser.

• General PrivilegesDatabase administration

Enhanced Reporting

Script or Command

• SchedulingAll privileges

Step 4 Open the Scheduling Center tool.

• Select the Tools - Scheduling Centermenu option in the Dashboard. The Scheduling Cen-ter dialog opens.

Step 5 Right-click the database backup task and select Enable from the pop-up menu.

Step 6 Repeat step 5 for the other required database backup tasks.

Step 7 Click the Exit button to close the Scheduling Center.


12

3 Fault History Maintenance

3 Fault History Maintenance

3.1 Configuring Automatic Deletion of Fault History

System 8000 Intelligent Network Manager system

Time Late evening or night time

Tool(s) to use Fault Management System

Description Automatically deletes the faults that are older than the configured period. Inthe long run, the fault history table grows so big that it is necessary to deletepart of the old faults. Otherwise the performance of 8000 Intelligent NetworkManager system suffers. If you want to save all faults, you must always savethe latest database backup before deletion of the fault history.

The deletion of the fault history creates a large amount of transaction log data. The amountof free space in the log device should be periodically checked as mentioned in the dailymaintenance tasks. For more information, see 4.1.4 Monitoring Size of Data Devices andTransaction Logs.

8000 Intelligent Network Manager can be configured to automatically delete fault history older thanthe configured period of days. Both element and service faults can be configured separately.

Procedure

Step 1 Open the Fault Management tool to configure fault history deletion.

• Select the Tools - Fault Managementmenu option. The Fault Managementmain windowopens.

• Select the Fault - Configure History Settings menu option.The Configure Fault History Deleting dialog opens.

Step 2 Configure the number of days of the element fault history to be retained.

• Enable the Automatic Element Fault History Deleting enabled option.

• Configure the required number of days after which the element fault history can be deletedin the Delete element faults from Fault History older than – days field.

Step 3 Configure the number of days of the service fault history to be retained.

• Enable the Automatic Service Fault History Deleting enabled option.

• Configure the required number of days after which service fault history can be deleted in theDelete service faults from Fault History older than – days field.

Step 4 Click OK to exit.


13

4 Daily Maintenance

4 Daily Maintenance

4.1 Sybase SQL Server Daily Maintenance Tasks

4.1.1 General

System Sybase SQL Server, Solaris, Windows 2008/2012, Linux

Time Late evening or night time

Tool(s) to use Windows Explorer, telnet

Description The 8000 Intelligent Network Manager database contains all the informationon the network, for example, 8100 Managed Access System and 8600 SmartRouters. Therefore, you have to take a backup of the 8000 IntelligentNetwork Manager database every day. If the 8000 Intelligent NetworkManager database is corrupted or destroyed for some reason, e.g. a hard diskproblem, you must restore the latest database backup.

You should take a backup of the 8000 Intelligent Network Manager database on the hard diskof the Database Server and on tape at least every day. The backup procedures take a dump ofthe transaction logs every four hours of the day, a full dump of the database once every day anda master database dump once every week and moves these dumps to tape once every day. Studythese procedures carefully in order to understand them and to be able to check if they functionproperly on your computer.

4.1.2 Verifying Sybase Database Backups

You should check every day that the database backup files are created to the backup directory. In theWindows Database Server, by default the backups are saved to H:\backup\ directory. Check thatthere is a current database dump file and four transaction log files for each day with correct timestamps. If some files are missing, check that the backup scripts are running or if the SchedulingCenter is used, check that there are no errors during the backup tasks. Moreover, check the Sybasebackup.log in case of a backup failure.

In the Solaris or Linux Database Server, by default backups are stored to the /sybase_backup/directory. Check that the database and transaction log backup files of the last 7 days exist. If somefiles are missing, check that the backup scripts are running or if the Scheduling Center is used,check that there are no errors during the backup tasks. Moreover, check the Sybase backup.login case of a backup failure.

In case of the separate PM database, check also that the PM database backups are taken and saved tothe backup directory.


14

4 Daily Maintenance

4.1.3 Verifying Tape Backups

If tape backups are used, the tape should be changed every morning. You should have at least 9tapes: one for each weekday, the 8th tape should be used every second Monday, and the 9th tapeshould be used once every month. If there is a need to have backups older than a month, the operatorcan use more tapes for this purpose as needed. This procedure eliminates the possibility of total lossof the database if one tape is faulty. The tapes should be tagged with their day or number accordingto their usage to avoid any confusion. Store the tapes in a safe place (protected from fire). Youshould never store backups in the same office/place where your Database Server is.

4.1.4 Monitoring Size of Data Devices and Transaction Logs

System Sybase SQL Server

Time Any time

Tool(s) to use ISQL, NMS Component Fault Monitoring Window

Description Checks that the transaction logs are not filling up, which would prevent theusage of 8000 Intelligent Network Manager. It is also useful to check howmuch of the database device is currently used.

Procedure

Step 1 Open the Command Prompt window.

Step 2 Start an ISQL session.isql -SDBSERVER -Usa -Ppassword

Step 3 Check how much of the transaction log (free Kbytes in nms_log1) is used with the followingcommand.

1>sp_helpdb db80002>go

Result

The transaction log should never fill up if everything is functioning fine because it is dumpedautomatically by the backup procedure that you must have set up. If the transaction log fills up, youmust dump it and find out why it filled up, because normally it should not happen.

Step 4 Check how much free space there is in the data device (free Kbytes in nms_db1).1>sp_helpdb db80002>go

Step 5 If the separate PM database is used, then repeat steps 1 to 4. In ISQL commands, replace thedatabase name db8000 with db8000_pms.

Result

The amount of free space should never drop below 10 Mbytes (10240 Kbytes). If this happens,check if it is caused by an excessive number of stored fault history events. If this is the case, delete apart of the fault history. Otherwise the network has grown so big that the size of the data devicemust be extended (contact your local Technical Support for the 8000 system). The audit log tablecan cause the database to fill up extensively. Delete or move the data from the audit log to anotherdatabase if necessary.


15

4 Daily Maintenance

It is also possible to monitor database device sizes from the Fault Monitoring Management, i.e.FMS shows an alarm message describing which device is filling up and which limit has beenexceeded. These limits and other necessary configurations are automatically created and configuredby the 8000 Intelligent Network Manager setup process. By default Fault Management will showalarms when a data and log device is 70, 80 or 90 percent full. These default alarm limits can bemodified as instructed below.

Faults are also created if the database is not backed up according to the plan. The faults are visiblein the NMS Component Fault Monitoring - Database. MEI fault is visible if the last backup wasdone over 4 hours ago (the minimum transaction log dump time), DMA fault for over 12 hours, andPMA fault for over 24 hours.

Configuring Alarm Levels

Step 1 Open the Fault Management tool.


Step 2 Select the Options - NMS component monitoring menu option in the Fault Management mainwindow.

Step 3 Double-click on the computer icon at the lower right hand corner of Network View.The NMS Component Fault Monitoring window opens.

Step 4 In the NMS Component Fault Monitoring window, Select Options – Configure Alarm Limits.

Step 5 Click the Add or the Edit button.Configure the alarms for different data and log devices and set the corresponding limits usingthe Add New Alarm Limit dialog.

4.1.5 Monitoring Disk Space Usage and Log Files

System Sybase SQL Server, Solaris, Windows 2008/2012, Linux

Time Any time

Tool(s) to use Telnet, Windows Explorer

Description Check that the Database Server disks are not filling up, which would preventthe usage of 8000 Intelligent Network Manager.

4.1.6 Checking Disk Space

The disk space usage of all file systems of the Database Server should be checked to verify if it isclose to full. On a Solaris and Linux Database Server the df –k command can be used to check diskspace usage. Necessary action should be taken to clear space on those file systems which are closeto full by clearing large and old log files. On a Windows Database Server, check that all physicallyinstalled disks has enough free space for Sybase SQL server.


16

4 Daily Maintenance

4.1.7 Investigating and Removing Core Files from Solaris and Linux DatabaseServer

Core files are created when a crash has occurred at the operating system level. Thus, these core filesshould be analyzed with the help of a Solaris administrator to understand what caused the problem.In order to list the core dump files from the Database Server, run the following command.

$ find / —name core

After moving the core files to a different location for investigation, they should be deleted on theserver to save space.

$ find / —name core —print | xargs rm

4.1.8 Truncating Sybase Log Files on Windows Database Server

In the Sybase versions prior to 15.5, the sybase log files (%SYBASE%\BACKUP.LOG and%SYBASE%\MS.LOG) should be emptied during the maintenance window when the database isdown.

In the Sybase version 15.5 or later, the Sybase Database Server error log can be changed dynamicallyand it does not require an ASE reboot.

Procedure

Step 1 Open the ISQL session to the database.

Step 2 Run the SQL command to change the main Sybase error log.1>sp_errorlog “change log”, “$SYBASE\$SYBASE_ASE\new.log”

Step 3 Delete the old error log file.

Step 4 Change again the error log file to the initial error log to go back to the initial situation.1>sp_errorlog “change log”, “$SYBASE\$SYBASE_ASE\error.log”

Step 5 Similarly, repeat the above steps 2 to 4 for sybase\SYB_BACKUP.log.

Step 6 If the old logs are not needed for any investigation, they should be deleted or moved to a differentstorage server.

4.1.9 Truncating Sybase Log Files on Solaris and Linux Database Server

The running Database Server Log file cannot be deleted as ASE loses file handle on it. Instead, itmust be purged by copying null to it as shown below.

Procedure

Step 1 Open a terminal connection to the Solaris server.

Step 2 Go to the log file location.$ cd /sybase/log

Step 3 Copy the log file to the archive.$ cp DBSERVER.log DBSERVER.log.old


17

4 Daily Maintenance

Step 4 Purge the existing log file.$ cp /dev/null DBSERVER.log

Step 5 Similarly, repeat the above steps 2 to 4 for /sybase/log/SYB_BACKUP.log.

Step 6 If the old logs are not needed for any investigation, they should be deleted or moved to a differentstorage server.

4.1.10 Verifying Database Statistics Updating


Time Daily and always after indexes have been reorganized


Description Improves the performance of the database access.

The execution status of the scheduled database statistics procedures should be checked if there wereany problems during execution. You can see the status of the operation in the Last exec statuscolumn. When the operation is completed the Last exec status is Successful. If the Last exec statusis Failed, execute the procedure manually using steps below and inspect the logs for any errors.

Moreover, the execution times should be verified so that they do not overlap. If there are anyoverlaps, intervals should be reorganized using steps in 2.1 Enabling Automatic Database Backups.

Procedure


• Select the Tools - Scheduling Centermenu option in the Dashboard. The Scheduling Cen-ter dialog opens.

Step 2 Select the Daily DB Maintenance, Update Table Statistics task, and click the Execute button.

Step 3 Wait until the Last exec status is Successful.

Step 4 If the Last exec status is Failed, inspect the logs for errors.

• Inspect the logs in the output section of the window for any errors.

• Take corresponding actions to correct the errors.

Step 5 Similarly, repeat the above steps 2 to 4 with the Update Index Statistics task.

Step 6 Click Exit to close the Scheduling Center tool or continue running other maintenance procedures.

4.2 Checking 8000 Intelligent Network Manager Servers andCommunication Links

4.2.1 Checking 8000 Intelligent Network Manager Servers


Time Any time


18

4 Daily Maintenance


Description Checks that Management and Communication Server(s) are functioning.

Procedure



Step 2 Check the fault status of 8000 Intelligent Network Manager servers.

• Select the Options - NMS Component Monitoring menu option.

• Double-click the computer icon at the lower right hand corner of Network View.The NMS Component Fault Monitoring dialog opens.

• Check that there are no faults for any 8000 Intelligent Network Manager servers.

Result

There should not be any faults in the servers. If there are, try to find out the reason and solve theproblem, or contact your local Technical Support for the 8000 system.

4.2.2 Checking Communication Links and Paths to All Network Nodes


Time Any time


Description Checks that communication links or paths to all Network Nodes are working.

Procedure



Step 2 Open the network fault report for active or unacknowledged faults.

• Select the Fault - Network Fault Report – Active or Unacknowledged Faults menu op-tion.

Step 3 Check for any alarms indicating that communication links or paths with the Network Elementsare broken.

Result

Check that there are no used Network Nodes with broken communication links or paths.

If any of them has an alarm, check first that the cables from DXX Server(s) or Communicationserver(s) to the 8000 system network are connected correctly. If this does not solve the problem,contact your local Technical Support for the 8000 system.


19

5 Weekly Maintenance


5.1 Sybase SQL Server Weekly Maintenance Tasks

5.1.1 Verifying Status of Scheduled Procedures in Scheduling Center


Time Any time


Description Verifies the status of the different scheduled procedures in the SchedulingCenter to check if they are running as planned and their intervals are notconflicting with each other.

If your network grows extensively, e.g. new network elements are added or moved all the time,the execution times of different procedures scheduled by Scheduling Center increase andmay cause conflicts due to overlapping if the intervals between them are not long enough.

The execution status of all the scheduled procedures should be checked if there were any problemsduring execution. Moreover, the execution times should be verified so that they do not overlap.If there are any overlaps, intervals should be reorganized using steps in 2.1 Enabling AutomaticDatabase Backups.

Procedure


• Select the Tools - Scheduling Center menu option in the Dashboard.

Step 2 Select a scheduled procedure from the presented list.

Step 3 Verify that the Last exec time of the procedure matches the specified scheduling.

Step 4 Verify that the Last exec status of the procedure is Successful.

Step 5 If the Last exec status is Failed, execute the procedure and inspect the logs for any errors.

• Select the procedure and click the Execute button.

• After running the procedure, inspect the log in the output section of the window for anyerrors.


Step 6 Check the Last exec duration to determine how much time the procedure has taken to execute.Based on the execution time, determine if there is any overlap between the different scheduledprocedures. If there are any overlaps, intervals should be reorganized using steps in 2.1 EnablingAutomatic Database Backups. The intervals can also be reduced if the execution times are short andthe intervals are wider than necessary.


20


Step 7 Similarly, repeat the steps 2–6 for every scheduled procedure in the list.

5.1.2 Verifying Sybase SQL Server Database Configuration Parameters


Time Night time or early in the morning

Tool(s) to use Scheduling Center, FMS and ISQL

Description Checks the usage of database configuration parameters such as ProcedureCache, number of open Partitions etc. The values of these parameters need tobe readjusted based on their usage.

In addition to periodic maintenance, note that the Database Server parameters need to berevised in cases of major changes in network size, number of users, large operations donewith 8000 Intelligent Network Manager, upgrades or change to Database Server hardware.Monitoring the database performance is quite essential in order to find out about needs tochange the Database Server parameters. Refer to Sybase Database Server documentation forfurther information and contact your Sybase administrator to change database parameters.Note that changing the database configuration values may need the Sybase SQL Serverrestart so it is recommended to carry out these tasks in the maintenance window.

Procedure

Step 1 Make sure that procedure DB Server Housekeeper has been executed successfully in theScheduling Center as instructed in chapter 5.1.1 Verifying Status of Scheduled Procedures inScheduling Center.

Step 2 Check with NMS Component monitoring if there are Database Server configurations that needmodification.

• Open the Fault Management tool in the Dashboard by selecting the Tools - Fault Man-agement menu option.

• Select the Fault - NMS Component Monitoring menu option. The NMS ComponentFault Monitoring dialog opens.

• In the Database window, double-click the Database Server, which is marked with its host-name.

• If there are faults listed, check the Problem field to see if there are itemsmarked asDatabaseserver configuration required and check the corresponding Alarming object to see whichISQL query needs to be performed.


21


Step 3 Configure the new Sybase SQL Server database configuration values.

• Login to the Management Server with a username that has administrator rights. In Linux-based Management Server login to the console.

• Run the C:\%NMSDIR%\sql\config_g.cmd file by double-clicking it in Explorer. InLinux-based Management Server run the $NMSINSTDIR/sql/. config_g.sh command.8000 Intelligent Network Manager Database Wizard starts.

• Select the Configure Server task and click Next.

• Login as a sa, give the sa password and click Next.

• In the Configure Server window, check that the Recommended configuration values arethe same as the ones suggested by Fault Management and click Next.

• Check the new values and click Configure in the Confirm the Parameters for the Config-ure Server window.

• Follow the screen instructions. If the Sybase SQL Server restart is required, stop and startthe Sybase SQL Server.

• Wait until the Configure Server status is successful message appears.

• Click the Details button and check that the file contains no error messages.

• Click Next.

• Click Finish to exit from the Configure Server tool.

Step 4 Verify that faults do not exist anymore in the FMS.

Step 5 Click Exit to close the Scheduling Center tool and FMS.

5.1.3 Verifying Index Reorganization in 8000 Intelligent Network ManagerDatabase


Time Night time or early in the morning


Description Keeps the performance of the database access optimal. If you do notreorganize indexes on a regular basis, the performance suffers in the long run.

If your network grows extensively, e.g. new network elements are added or moved all thetime, we advise you to reorganize the indexes more often, e.g. twice a week, to increase theperformance of the Database Server.

The execution status of the scheduled index reorganization procedure should be checked if therewere any problems during execution. You can see the status of the operation in the Last exec statuscolumn. When the operation is completed the Last exec status is Successful. If the Last exec statusis Failed, execute the procedure manually using steps below and inspect the logs for any errors.



22


Procedure



Step 2 Select the Rebuild Indexes task from Daily DB Maintenance Schedule and click the Executebutton.



• Inspect the logs in output section of the window for any errors.


Step 5 Similarly, repeat the steps 2 to 4 with the task Reclaim Unused Space, Update Table Statisticsand Update Index Statistics.


5.1.4 Verifying 8000 Intelligent Network Manager Database Consistency


Time Recommended to be run when the use of the network management systemis light, e.g. early in the morning.


Description Checks the consistency of the database. If the database has inconsistenciesyou may have serious problems in 8000 Intelligent Network Manager system.

The execution status of the scheduled database consistency procedures should be checked if therewere any problems during execution. You can see the status of the operation in the Last exec statuscolumn. When the operation is completed the Last exec status is Successful. If the Last exec statusis Failed, execute the procedure manually using steps below and inspect the logs for any errors.


Procedure



Step 2 Select theWeekly DB Check, Check Table Consistency task, and click Execute.



• Inspect the logs in output section of the window for any errors.


Step 5 Similarly, repeat the above steps 2 to 4 with the task Check Index Consistency.



23


5.1.5 Testing Restoration of Database Backup


Time Any time

Tool(s) to use ISQL

Description Test the restoration of 8000 Intelligent Network Manager database backuponce a week. Use a second database (e.g. sim_secdb8000) for this purpose.

This procedure is possible only if you have enough space on your Database Server to create asecond database with the same characteristics, e.g. the size of devices, as in your productiondatabase, or if you have another server on which the database dump can be loaded. However,it is highly recommended that a separate isolated Database Server is used to avoid anyaccidental interruption on the production Database Server.

Procedure

Step 1 Create a sim_secdb8000 database that has the same characteristics as your network database, e.g.the size of devices. See 8000 Intelligent Network Manager Software Installation Manual forinstructions on creating a database.

Step 2 Load the dump into sim_secdb8000.

• Open the Command Prompt window.

• Start an ISQL session.isql -SDBSERVER -Usa –Ppassword

• Load the dump into sim_secdb8000.If you are using aWindows based Database Server, enter the following command:

1> load database sim_secdb8000 from ‘h:\backup\dumpdb.bkp’2> go

If you are using a Solaris or Linux based Database Server, enter the following command.

1> load database sim_secdb8000 from ‘/sybase_backup/re-store/dumpdb.bkp’2> go

Result

There should be no errors during the load operation.


24


Step 3 In case transaction dumps are being tested too, load the transaction dumps as shown below, one afterthe other in the same order that they were created. If the transaction dumps are not being tested,skip this step and proceed from step 5 onwards.

• Load the transaction dump into sim_secdb8000.If you are using aWindows based Database Server, enter the following command.

1> load tran sim_secdb8000 from ‘h:\backup\dumptran.bkp’2> go

If you are using a Solaris based Database Server, enter the following command.

1> load tran sim_secdb8000 from ‘/sybase_backup/re-store/dumptran.bkp’2> go

• Repeat the above for each transaction dump, in the same order that they were created.

Result

There should be no errors during the load operation.

Step 4 Bring sim_secdb8000 online and check the NMS table.1>online database sim_secdb80002>go

1>use sim_secdb80002>go

1> select * from nms2> go

Result

The command must have been executed and it should display information of the installed 8000Intelligent Network Manager.

Step 5 Check if the database backup contains up-to-date information.1>select max(time) from sim_secdb8000..osrchan2>go

Result

The result should be close to the time when the database backup was taken. If not, it is likely thatthe backup is older in time than it is thought to be.

Step 6 If the separate PM database is used, repeat steps 2 to 6 for the PM database. If the db8000 anddb8000_pms databases are the same size, sim_secdb8000 can be used for testing both databases.Otherwise create an own test database for the PM database and use it for testing.

It is very important that you do not perform any hardware actions using this database (e.g.update parameters in a network element) with a workstation unless you are sure that thereis no connection to the production network from the server running the sim_secdb8000database. The production database could be damaged and cut off traffic. You can alsocreate a simulation database instead of a second database in order to avoid this problem.Completing steps 5 and 6 should be enough to check that the restoration of the database isfunctioning properly.

If you fail to restore your 8000 Intelligent Network Manager database backup, contact yourlocal Technical Support for the 8000 INM system.


25


5.2 Monitoring Database Performance Data Size


Time Any time

Tool(s) to use 8000 Performance Management

Description Monitors the PMS data size in order to not let the database fill up extensively.

Procedure

Step 1 In the Dashboard, select the Monitoring – Performance Management option. The 8000Performance Management dialog opens.

Step 2 Select the Performance – Performance Management Configuration menu option. ThePerformance Management configuration for Database dialog opens.

Step 3 Check the size of the PMS data.

• Check the Database size MB fields in the 8000 Configuration tab.

• Check the Database size MB fields in the 8100 Configuration tab.

Result

It is difficult to set a rule for the acceptable PMS data size in the database. The acceptable sizedepends on the overall database size, Database Server type and memory as well as the impact on theduration of running the database maintenance scripts. However, tables which are larger than severalmegabytes may cause problems in the SQL server and decrease its performance.

You can affect the size of the PMS data by setting the corresponding Interval history and agingdays variables in the Performance Management Configuration window.

5.3 Checking Disk Space of 8000 Intelligent Network ManagerComputers (and Moving Log Files)

Time Any time

Tool(s) to use Commands and scripts

Description Checks that there is enough disk space available for log writing.

Procedure

Step 1 On each computer where you are getting log files, e.g. nmstrace.log, you should check that the harddisk is not filling up. No more than 90 per cent of the disk space should be used. To make this stepeasier, you can use the parameter y to cut log files at midnight with the following command.

nmstrace 0123456 * trace.log y


26


Step 2 Check if there are any unnecessary log files and delete them.

• If you find any unnecessary log files, select the directory of the file and delete the files.There may be logs in the%NMSDIR%\LOG folder. In addition, Phoenix.log is locatedin \%NMSDIR%\Java\Extern\Phoenix\Logs.

• Delete other unnecessary files.

Step 3 As long as there are no errors or unsolved entries in the nms tracelog files, there is no need to keepthem to occupy disk space, so delete or move the log files to another computer via network.

5.4 Checking Disk Space on ThinClient Server

Time Any time

Tool(s) to use Commands

Description Checks that there is enough disk space available for log writing on theThinClient Server. ThinClient Server maintains session data ex. logs for allthe users accessing 8000 Intelligent Network Manager in their correspondingdirectories. The nms logs created by these individual sessions should beregularly cleaned up in order to ensure there is enough disk space on theThinClient Server.

Procedure

Step 1 The disk space usage of all file systems of the ThinClient Server should be checked to verify if it isclose to full. No more than 90 per cent of the disk space should be used. The following main nmsdirectory should be checked if there are any excessive log files and they should be deleted.

%NMSDIR%\log

Step 2 Check if there are any excessive log files in all the user sessions created in ThinClient server.

• The individual user session nms logs are in the C:\shared\users\citrix_users\<user-name>\Coriant 8000\log directories.

• Make sure the workstation of the corresponding user is not running and delete all excessivelog files in each directory.


27

6 Monthly Maintenance


6.1 Deleting Audit Log

Tool(s) to use Dashboard

Description Delete the Audit Log entries that are older than one month. In the long run,the audit log table becomes so big that it is necessary to delete part of the oldentries. Otherwise, the performance of 8000 Intelligent Network Managersystem suffers. If you want to save all entries, you must always save the latestdatabase backup before deleting the Audit Log.

This procedure creates a large amount of transaction log data. Check first that you haveenough free space in your log device before executing this operation.

Procedure

Step 1 Select the Audit Log entry truncation date.

• Select the Security - Audit Logs - Manager Audit Log menu option in the Dashboard.The Audit Log dialog opens.

• Select the Options - Refresh menu option.

• Select the Options - Delete events menu option.The Delete dialog opens.

• Define the deletion time to one month earlier than the current date.

Step 2 Delete the faults.

• Click the OK button.The specified Audit Log events are deleted from the database.

6.2 Monitoring Size of Fault Tables

Tool(s) to use ISQL

Description It is very important to monitor the size of the tables that can grow a lot inthe database. In some cases, if a table is too big, the SQL server can haveproblems creating indexes, or some entries can become corrupted.The performance of the database decreases a lot if some tables are too big.

Step 1 Start an ISQL session.isql -SDBSERVER -Usa -Ppassword


28


Step 2 Check the size of the fault tables.1>use db80002>go

1>sp_spaceused faulthist2>go

1>sp_spaceused fault2>go

Result

It is rather difficult to set a rule for the acceptable table size. The acceptable table size depends onvarious parameters such as the computer type and the amount of memory. However, the tableswhich are bigger than several megabytes may cause problems in the SQL server and decreaseits performance.

If the fault history table is too big, you should keep the entries corresponding to a shorter periodof time as mentioned in chapter 3 Fault History Maintenance.

If the fault table is too big, start clearing up the faults in your network more efficiently.


29


6.3 Accounting Management Events

Accounting management creates a large amount of transaction log data. Check first thatyou have enough free space in your log device using DB command sp_helpdevice beforeexecuting this operation.

Time Any time

Tool(s) to use Accounting Management

Description Delete the events that are older than one month. If you want to save allevents, you must always save the latest database backup before deleting theevents. Contact your local Technical Support for 8000 system for moredetailed instructions on backing up and restoring a database.

Procedure

Step 1 Open the Accounting Management tool.

• Select the Tools – Accounting Management menu option.The Accounting Management dialog opens.

Step 2 Click the Delete button.

Step 3 Type a month-old date and click OK. All events that occurred before that date are deleted.

Step 4 Click Yes.

6.4 Node Configuration Backup

Time Any time

Tool(s) to use Network Editor

Description In 8000 Intelligent Network Manager it is possible to configure periodicNode Configuration Backups.

Procedure

Step 1 In the Dashboard select the Network Building - Network Editor option.The Network Editor main window opens.

Step 2 Select the Edit – Communication Servers menu option.The NWED:Communication Server Parameters dialog opens.

Step 3 Select the relevant Communication Server from the Name drop-down list, and select the relevantadapter to configure in the Communication Adapters list box.

Step 4 Click the Parameters button.The NWED: Generic SNMP Adapter dialog opens.

Step 5 Click the Polling Policies button.The NWED:Polling Policy Parameters dialog opens.

Step 6 Tick the Configuration Backup check box and click the Parameters button next to it.The NWED:Configuration Backup dialog opens.


30


Step 7 Select either the Periodic backup or Daily backup radio button, and fill in the relevant parametersand the number of backup files per node if available.

Step 8 Click the OK button to confirm the parameters and close the dialog.

Step 9 Click the Update button in the NWED:Polling Policy Parameters dialog to activate the policy.

Step 10 Repeat steps 3 to 9 for all network element types which support taking backups.


31

Suggested Weekly Schedule for Automatic Maintenance Procedures

Suggested Weekly Schedule for AutomaticMaintenance Procedures

Suggested weekly scheduling of automatic maintenance procedures is presented below. This canonly be used as a guideline as the running times of various procedures vary based on the size ofthe network, hence they need to be adjusted as per requirement. Note that the following databasehousekeeping procedures are automatically scheduled to run every hour and therefore are notpresented in the table below: Resourcelock Housekeeper (at 15 minutes past every hour) and DBServer Housekeeper (only for Sybase at 1 minute past every hour). If a separate PMS databaseis used, generic rule for scheduling is that backup procedures are scheduled one hour after thecorresponding main database procedures.

On Windows, Solaris and LinuxSybase Database Server

Mon Tue Wed Thu Fri Sat Sun

00:15 WeeklyDB Check

03:00 Trans-actiondump #1

Transac-tion dump#1

Trans-actiondump #1

Trans-actiondump #1

Transac-tion dump#1

Transac-tion dump#1

Transac-tion dump#1

03:15 DailyDBMainte-nance

Daily DBMainte-nance







Transac-tion dump#2

Trans-actiondump #2

Trans-actiondump #2

Transac-tion dump#2

Transac-tion dump#2

Transac-tion dump#2


Transac-tion dump#3

Trans-actiondump #3

Trans-actiondump #3

Transac-tion dump#3

Transac-tion dump#3

Transac-tion dump#3


Transac-tion dump#4

Trans-actiondump #4

Trans-actiondump #4

Transac-tion dump#4

Transac-tion dump#4

Transac-tion dump#4


Transac-tion dump#5

Trans-actiondump #5

Trans-actiondump #5

Transac-tion dump#5

Transac-tion dump#5

Transac-tion dump#5

19:30 Masterdatabasedump

21:00 Fulldatabasedump

Fulldatabasedump

Fulldatabasedump

Fulldatabasedump

Fulldatabasedump

Fulldatabasedump

Fulldatabasedump


32

Suggested Weekly Schedule for Automatic Maintenance Procedures

On Windows, Solaris and LinuxSybase Database Server

23:00 Tapebackup

Tapebackup

Tapebackup

Tapebackup

Tapebackup

Tapebackup

Tapebackup

23:30 ServerMoni-toringHouse-keeper

ServerMoni-toringHouse-keeper





ServerMonitor-ing House-keeper


33

Documents

8000 INM SR6.0 Maintenance Procedures