Tellabs 8000 Intelligent Network Manager

Embed Size (px)

Citation preview

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    1/40

    Tellabs8000 Intelligent Network Manager SR2.1

    Maintenance Procedures

    76.8021-70211A14.02.2011

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    2/40

    Document Information

    2011 Tellabs. All rights reserved.

    This Tellabs manual is owned by Tellabs or its licensors and protected by U.S. and international copyright laws, conventions and

    treaties. Your right to use this manual is subject to limitations and restrictions imposed by applicable licenses and copyright laws.

    Unauthorized reproduction, modification, distribution, display or other use of this manual may result in criminal and civil penalties.

    The following trademarks and service marks are owned by Tellabs Operations, Inc. or its affiliates in the United States and/orother countries: TELLABS, TELLABS logo, TELLABS and T symbol, and T symbol .

    Any other company or product names may be trademarks of their respective companies.

    The specifications and information regarding the products in this manual are subject to change without notice. All statements,

    information, and recommendations in this manual are believed to be accurate but are presented without warranty of any kind,

    express or implied. Users must take full responsibility for their application of any products.

    Adobe Reader are registered trademarks of Adobe Systems Incorporated in the United States and/or other countries.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    2

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    3/40

    Document Information

    Terms and Abbreviations

    Term Explanation

    AMPS Automatic Maintenance Procedures Service

    CST Configuration and Service Tracker

    DB Database

    LAN Local Area Network

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    3

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    4/40

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    4

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    5/40

    Table of Contents

    Table of Contents

    About the Manual .............................................................................................................. 7

    Objectives........................................................................................................ ............................................................... 7

    Audience.......................................................................... ............................................................................................... 7

    Related Documentation ................................................................................................. ................................................. 7

    Conventions........... .............................................................................................................................. ........................... 8

    Document Conventions ............................................................................................. ..................................................... 8

    Documentation Feedback .................................................................................................. ............................................. 8

    1 Introduction ................................................................................................................. 9

    2 Automatic Maintenance Procedures .........................................................................10

    2.1 Activating Automatic Maintenance Procedures .................................................................................................. 11

    2.2 Configuring Automatic Maintenance Procedures on Database High Availability Setup.................................... 13

    3 Fault History Maintenance..........................................................................................14

    3.1 Configuring Automatic Deletion of Fault History............................................................................................... 14

    4 Daily Maintenance.......................................................................................................15

    4.1 Backing Up Tellabs 8000 Intelligent Network Manager Database ..................................................................... 15

    4.1.1 General...................................................................................... ........................................................... 15

    4.1.2 Verifying Scheduled Backups on Windows Database Server in Case Backup Scripts Are Used....... 15

    4.1.3 Verifying Scheduled Backups on Solaris Database Server in Case Backup Scripts Are Used ........... 16

    4.1.4 Verifying Tape Backups................................ ....................................................................................... 17

    4.2 Monitoring Size of Data Devices and Transaction Logs..................................................................................... 17

    4.3 Checking Disk Space Usage and Logfiles on Database Server........................................................................... 19

    4.3.1 Checking Disk Space........................................................................................................................... 19

    4.3.2 Investigating and Removing Core Files from Solaris Database Server .............................................. 19

    4.3.3 Truncating Log File on Solaris Database Server ................................................................................. 19

    4.4 Checking Tellabs 8000 Intelligent Network Manager Servers............................................................................ 19

    4.5 Checking Communication Links and Paths to All Network Nodes .................................................................... 20

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    5

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    6/40

    Table of Contents

    5 Weekly Maintenance .................................................................................................. 21

    5.1 Verifying Database Configuration Parameters .................................................................................................... 21

    5.2 Verifying Status of Scheduled Procedures in Automatic Maintenance Procedures Tool.................................... 22

    5.3 Reorganizing Indexes in Tellabs 8000 Intelligent Network Manager Database ................................................. 23

    5.4 Updating Database Statistics after Reorganizing Indexes................................................................................... 24

    5.5 Checking Tellabs 8000 Intelligent Network Manager Database Consistency .................................................... 25

    5.6 Monitoring Size of Performance Data Size .................................................................................... ..................... 25

    5.7 Testing Restoration of Backup ............................................................................................................................ 26

    5.8 Checking Disk Space of Tellabs 8000 Intelligent Network Manager Computers (and Moving Log Files) ....... 28

    5.9 Checking Disk Space on ThinClient Server ........................................................................................................ 29

    5.10 Checking Disk Space on Route Master ............................................................................................................... 29

    6 Monthly Maintenance.................................................................................................. 31

    6.1 Deleting Audit Log........................................................................................................... ................................... 31

    6.2 Monitoring Size of Fault Tables ............................................................................................. ............................. 31

    6.3 Accounting Management Events......................................................................................................................... 33

    6.4 Private Subnetwork Trunk Accounting Events for Tellabs 8100 Nodes............................................................. 33

    6.5 Recovery Management System ................................................................................................ ........................... 34

    6.5.1 Spontaneous Fault Reporting (GMH Units)........................................................................................ 34

    6.5.2 Spontaneous Fault Reporting (GMU, GMU-A and GMU-M Units) .................................................. 34

    6.5.3 Spontaneous Fault Reporting (Other Units)........................................................................................ 34

    6.6 Maintenance of Element Software in Database............ ....................................................................................... 35

    6.7 Node Configuration Backup ................................................................................................................... ............. 35

    7 Additional Tools for Sun Solaris Server.................................................................... 37

    7.1 Sun Explorer ....................................................................................................... ................................................. 37

    7.2 Installation of Explorer ................................................................................................................................. ....... 37

    7.3 Configuration and Service Tracker (CST) Latest Version................................................................................... 38

    Suggested Weekly Schedule for Automatic Maintenance Procedures........................ 39

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    6

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    7/40

    About the Manual

    About the ManualThis section discusses the objectives and the intended audience of this manual,Tellabs 8000

    Intelligent Network Manager Maintenance Procedures, and consists of the following sections:

    Objectives

    Audience

    Related Documentation

    Conventions

    Documentation Feedback

    Objectives

    This manual provides instructions on the procedures that have to be executed on a regular basis in

    order to maintain the Tellabs 8000 intelligent network manager system.

    Audience

    This manual is designed for system administration personnel.

    Related Documentation

    The new document numbering scheme consists of the document ID, indicated by numbers, and the

    document revision, indicated by a letter. The references in the Related Documentation table below

    are generic and include only the document ID. To make sure the references point to the latest

    available document versions, please refer to the relevant product document program on the Tellabs

    Portal by navigating to www.portal.tellabs.com> Product Documentation > Data Networking >

    Tellabs 8000 Intelligent Network Manager > Technical Documentation.

    Tellabs 8000 Intelligent Network Manager SR2.1

    Software Installation Manual (76.8021-70201)

    Provides instructions on how to install and

    configure Tellabs 8000 intelligent network

    manager in different Tellabs 8000 intelligentnetwork manager computer platforms

    (workstation, servers).

    Tellabs 8000 Intelligent Network Manager SR2.1

    Third party Hardware and Software Requirements

    (76.8021-70222)

    Describes all the needed third party software

    components and versions in the Tellabs 8000

    intelligent network manager system (operating

    systems, service packs, database server versions,

    client versions, web browsers). Describes also

    the recommended computer platforms, LAN

    configurations and minimum performance

    requirements for computers.

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    7

    https://www.portal.tellabs.com/global/portal/home.jsphttps://www.portal.tellabs.com/global/portal/home.jsp
  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    8/40

    About the Manual

    Conventions

    The following typographical conventions are used to help distinguish elements in the text in this

    manual.

    Text Element Explanation

    Bold Indicates the name of an item (window/dialog/button, etc.) you need to select.

    When more than one item needs to be selected in a step, the items to be selected

    are separated by a dash (-).

    Italics Normally italics is used when referring to another document.

    Courier Indicates the contents of a file, a command or a part of a program.

    Document Conventions

    This is a note symbol. It emphasizes or supplements information in the document.

    This is a caution symbol. It indicates that damage to equipment is possible if the instructionsare not followed.

    This is a warning symbol. It indicates that bodily injury is possible if the instructions are notfollowed.

    Documentation Feedback

    Please contact us to suggest improvements or to report errors in our documentation:

    Email: [email protected]

    Fax: +358.9.4131.2430

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    8

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    9/40

    1 Introduction

    1 Introduction

    This document is intended for system administrators of the Tellabs 8000 intelligent network

    manager. It contains step lists for the procedures that must be executed on a regular basis to properly

    maintain the Tellabs 8000 intelligent network manager system.

    Please note that if these system maintenance procedures are not carried out, it may lead tototal management outage. If there isnt a valid database backup available, system recovery tooperational state can take several days or even weeks, and in most cases requires a massiveamount of manual work by Tellabs. This work is not covered by the Support Agreementbut is charged separately at an hourly rate.

    The procedures are classified into four categories, as listed below:

    1. What can be done automatically.

    2. What should be done every day.

    3. What should be done every week.

    4. What should be done every month or whenever necessary.

    These procedures are to be used with the Tellabs 8000 intelligent network manager. In addition to

    these procedures, the normal maintenance tasks for the operating system should be performed. For

    example, events in the Event Viewer such as Application Log, Security Log and System Log shouldbe monitored on a regular basis and truncated before they become too big.

    Instructions in this manual are written for Classic Start menu of Windows Server 2003/2008.

    You can turn on the Classic Start menu in Windows 2003/2008/XP by right clickingStartand

    selectingProperties, clicking the Start Menutab, selecting the Classic Start menuradio button

    and clicking OK.

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    9

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    10/40

    2 Automatic Maintenance Procedures

    2 Automatic Maintenance Procedures

    Tellabs 8000 intelligent network manager provides a way to automate some database maintenance

    and backup procedures that previously had to be done manually. The procedures that can be

    automated are listed below:

    1. Rebuild Indexes (ReorgIndex.cmd) [ID 10001]

    Schedule the rebuilding of indexes to be run before the reclaiming of unused space.

    2. Reclaim Unused Space (ReorgIndex.cmd) [ID 20004]

    3. Update Table Statistics (UpdStat.cmd) [ID 30002]

    Schedule the updating of table statistics to be run before the updating of index statistics.

    4. Update Index Statistics (UpdStat.cmd) [ID 40003]

    5. Check/Fix Table Corruptions (CheckDb.cmd) [ID 60006]

    6. Check/Fix Index Corruptions (CheckDb.cmd) [ID 70007]

    7. Dump Database [ID 80000]

    8. Dump Transaction [ID 90000]

    9. Dump Master [ID 100000]

    10. Resourcelock Housekeeper [ID 110008]

    11. DB Server Housekeeper [ID 120009]12. Server Monitoring Housekeeper [ID 130010]

    The database maintenance procedures (16) can be scheduled to run weekly during low Database

    Server load.

    For the database backup procedures (Dump Database, Dump Transaction and Dump Master),

    there are two alternatives: using the Automatic Maintenance Procedures Tool or using scheduled

    scripts. The procedure for using scheduled scripts is presented inTellabs 8000 Intelligent Network

    Manager Software Installation Manualin chapters Database Backup Procedures for Windows

    2003/2008 Database Serverand Database Backup Procedures for Solaris.

    Note that both alternatives of taking database backup should not be used simultaneously.When configuring the backups to be taken with Automatic Maintenance Procedures Tool,make sure to delete the related AT command scheduled scripts. If there are no other entries,deleting all can be done with command at /delete on the Database Server. If there are otherentries, the deletion must be done only for the related entries. In case of Solaris databaseserver, similar procedure needs to be performed using the crontab utility. On the other hand,if database backup is chosen to be done with scheduled scripts, then the database backupprocedures in Automatic Maintenance Procedures Tool should be disabled from use.

    The database housekeeping procedures Resource Lock Housekeeper and DB Server Housekeeper

    are automatically scheduled to run at the 15th minute of every hour and at 05:59 of every day

    respectively.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    10

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    11/40

    2 Automatic Maintenance Procedures

    2.1 Activating Automatic Maintenance Procedures

    You can activate the automatic maintenance procedures to be run regularly.

    Tool(s) to use Automatic Maintenance Procedures Tool

    Step 1 Make sure that the Automatic Maintenance Procedures Service (AMPS) is enabled and runningon one of the Management Servers.

    If it is not enabled, shut down a Management Server, start the Tellabs 8000 intelligent net-

    work manager installation program with the re-configure option and select also the Auto-

    matic Maintenance Procedures service from the list of services to be installed.

    Step 2 Login with superuser privileges to Tellabs 8000 Intelligent Network Manager Toolbox.

    Step 3 Start the Automatic Maintenance Procedures Tool.

    SelectTools - Automatic Maintenance Procedures Tool in Tellabs 8000 Intelligent Net-

    work Manager Toolbox.

    Step 4 Select the procedure that you wish to schedule.

    Step 5 Click theSchedule button.

    Step 6 Login with thedxxdatalogin name and password and clickOK.

    Step 7 Complete the following details in the openedAMPT Schedulingwindow:

    Enter a name for the scheduled procedure inSchedule Name.

    Select the interval for the frequency of the selected maintenance procedure (Yearly, Monthly,

    Weekly, Daily, Hourly, Every Minute).

    Select the execution time preferably to a low traffic period like late night-hours and during

    weekends etc.

    ClickAdd to add the task to the schedule. The scheduled procedure is shown in the list.

    ClickExit.

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    11

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    12/40

    2 Automatic Maintenance Procedures

    Step 8 Repeat steps 4 to 7 for the other procedures keeping in mind the following:

    Refer the guidelines provided in Appendix Suggested Weekly Schedule for Automatic Main-

    tenance Proceduresfor scheduling the various maintenance procedures.

    For database maintenance procedures (16), schedule each procedure initially with rather

    high interval between them, for example, 23 hours, so that they do not overlap each-other.

    As the execution times vary depending on the network factors, such as size, these inter-

    vals need to be reassessed and adjusted after observing couple of runs and also periodically

    checked as mentioned in the weekly maintenance task.

    During scheduling, follow the sequence guidelines for running different procedures with

    procedures 1 and 3 in2 Automatic Maintenance Procedures. See also chapter5.2 Verifying

    Status of Scheduled Procedures in Automatic Maintenance Procedures Tool.

    The database backup procedures (Dump Database, Dump Transaction and Dump Master)

    should be scheduled only if database backup procedures are not scheduled using scripts.

    The suggested scheduling times and intervals of database backup procedures are as follows:

    Full database dump at 04:00 everyday; Database Transaction dump at 07:00, 11:00, 15:00,19:00 and 23:00 everyday; Database Master dump at 06:00 every sunday.

    If database backup procedures are being scheduled, the backup directory should be set by

    adding the parameter DBBKPDIR = (Sample values are

    G:\backup\ in case of windows and /backup/ in case of Solaris) in the NMS.INI file of

    the management server running AMPS process and all work-stations where AMP Tool is

    used. Note that the AMPS process and the AMP Tool need to be restarted after making the

    change to NMS.INI file to take it into effect.

    The Resourcelock Housekeeper and Server Monitoring Housekeeper schedules should be

    left as pre-configured.

    The DB Server Housekeeper should be scheduled to run daily at 07:30.

    Step 9 ClickExitto close the Automatic Maintenance Procedures Tool.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    12

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    13/40

    2 Automatic Maintenance Procedures

    2.2 Configuring Automatic Maintenance Procedures on DatabaseHigh Availability Setup

    The following configuration needs to be done to enable automatic maintenance procedures to beperformed on both ASE1 and ASE2 on a database high availability setup.

    Step 1 Make sure that the server names are configured correctly on ASE1 as shown below.

    Open the Command Prompt window and start an ISQL session as an sa user on ASE1 andcheck the server name configuration.

    isql -SDBSERVER -Usa Ppassword1> sp_helpserver2> go

    The output should display the hostname of ASE1 as the local entry and hostname of ASE2

    as an ASEnterprise entry.

    If any of the two entries are missing, add them as shown below.

    1> sp_addserver , local2> go3> sp_addserver 4> go

    Enable remote access and login for ASE1.1> sp_configure allow remote access, 12> go3> sp_addremotelogin 4> go

    Step 2 Accomplish the previous step on ASE2 keeping in mind to reverse the roles of ASE1 and ASE2, i.e.ASE2 as local and ASE1 as remote.

    Step 3 Restart the Sybase software on both ASE1 and ASE2.

    Step 4 Verify on ASE1 and ASE2 that there are entries for server names of both ASE1 and ASE2 throughthe Sybase Directory Service Editor tool using the dsedit command. The dsedit tool is a graphical

    user interface which requires X display. Also the dscp command, which is a command line interface,

    can be used for this purpose.

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    13

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    14/40

    3 Fault History Maintenance

    3 Fault History Maintenance

    3.1 Configuring Automatic Deletion of Fault History

    Tool(s) to use Fault Management System

    Description Automatically deletes the faults that are older than a configured period. In

    the long run, the fault history table grows so big that it is necessary to delete

    part of the old faults. Otherwise the performance of Tellabs 8000 intelligent

    network manager system suffers. If you want to save all faults, you must

    always save the latest database backup before deletion of the fault history.

    Deletion of Fault History creates a large amount of transaction log data. It should beperiodically checked that there is enough free space in log device as mentioned in dailymaintenance tasks. For more information, see4.2 Monitoring Size of Data Devices andTransaction Logs.

    The Tellabs 8000 manager can be configured to automatically delete fault history older than a

    configured period of days. Both element and service faults can be configured separately.

    Procedure

    Step 1 Open theConfigure Fault History Deletingoption.

    Select theFault - Configure History Settingsmenu option in the network window of Fault

    Management.

    The Configure Fault History Deletingdialog opens.

    Step 2 Configure the number of days of Element Fault History to be retained.

    Enable theAutomatic Element Fault History Deleting enabledoption.

    Configure the required number of days after which element fault history can be deleted in

    text boxDelete element faults from Fault History older thandays.

    Step 3 Confi

    gure the number of days of Service Fault History to be retained. Enable theAutomatic Service Fault History Deleting enabledoption.

    Configure the required number of days after which service fault history can be deleted in

    text boxDelete service faults from Fault History older than days.

    Step 4 PressOKto exit.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    14

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    15/40

    4 Daily Maintenance

    4 Daily Maintenance

    4.1 Backing Up Tellabs 8000 Intelligent Network Manager Database

    4.1.1 General

    Time Late evening or night time

    Tool(s) to use ISQL and scripts

    Description The Tellabs 8000 intelligent network manager database contains all the

    information on the network, for example, Tellabs 8100 managed access

    system and Tellabs 8600 managed edge system. Therefore, you have to take

    a backup of the Tellabs 8000 intelligent network manager database every

    day. If the Tellabs 8000 intelligent network manager database is corrupted

    or destroyed for some reason, e.g. a hard disk problem, you must restore

    the latest database backup.

    You should take a backup of the Tellabs 8000 intelligent network manager database on the hard disk

    of the database server and on tape at least every day. The backup procedures take the transaction

    logs dump every four hours of the day, and a full dump of the database once every day and master

    database dump once every week and moves these dumps to tape once every day. Study these

    procedures carefully in order to understand them and to be able to check if they function properly onyour computer.

    If you have not installed a backup procedure on your Database Server, see Tellabs 8000 Intelligent

    Network Manager Software Installation Manual, chapterDatabase Backup Procedures for Windows

    2003/2008 Database Serverfor installation instructions. If the Database Server is installed on

    Solaris server, see chapterDatabase Backup Procedures for Solaris for installation instructions. If

    Database backup procedures are being used through automatic maintenance procedures (AMPS), the

    below verification of scheduled scripts is not applicable . The automatic maintenance procedures are

    verified as mentioned in5.2 Verifying Status of Scheduled Procedures in Automatic Maintenance

    Procedures Tool

    4.1.2 Verifying Scheduled Backups on Windows Database Server in CaseBackup Scripts Are Used

    It is recommended to check daily that the scheduled backups are running. To verify running tasks,

    run theat command in Command Prompt:

    C:\>at

    Status ID Day Time Command Line

    1 Each M 23:00 PM g:\backup\dumpdb.bat mon 6

    10 Each Tu 3:00 AM g:\backup\dumptran.bat tue 1

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    15

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    16/40

    4 Daily Maintenance

    11 Each W 3:00 AM g:\backup\dumptran.bat wed 1

    12 Each Th 3:00 AM g:\backup\dumptran.bat thu 1

    13 Each F 3:00 AM g:\backup\dumptran.bat fri 1

    14 Each S 3:00 AM g:\backup\dumptran.bat sat 1

    15 Each Su 3:00 AM g:\backup\dumptran.bat sun 1

    16 Each M 7:00 AM g:\backup\dumptran.bat mon 2

    17 Each T 7:00 AM g:\backup\dumptran.bat tue 2

    18 Each W 7:00 AM g:\backup\dumptran.bat wed 2

    19 Each Th 7:00 AM g:\backup\dumptran.bat thu 2

    The list above is truncated, you should see at least seven full database dumps,five daily transaction

    log dumps and one master database dump scheduled.

    Apart from checking that the scheduling of backups is running and the actual backups are created, the

    log file C:\SYBASE\BACKUP.LOG should be checked for any errors during creation of backups.

    It should be verified that the backup scripts are configured correctly with the right databaseserver and database name. This can be checked in the scripts DUMPDB.BAT, DUMPDB.QRY,DUMPMSTR.BAT, DUMPMSTR.QRY, DUMPTRAN.BAT and DUMPTRAN.QRY.

    4.1.3 Verifying Scheduled Backups on Solaris Database Server in Case Backup

    Scripts Are Used

    Needed Crontab Entries

    The backup script dbdump.sh runs with Solaris cronjob utility. To check the cronjob for the

    scheduled backups, enter the following command.

    #crontab l

    Result:

    #ident "@(#)root 1.21 04/03/23 SMI"## The root crontab should be used to perform accounting datacollection.##10 3 * * * /usr/sbin/logadm15 3 * * 0 /usr/lib/fs/nfs/nfsfind30 3 * * * [ -x /usr/lib/gss/gsscred_clean ] &&/usr/lib/gss/gsscred_clean#10 3 * * * /usr/lib/krb5/kprop_script ___slave_kdcs___0 4 * * * su - sybase -c "/sybase/bkscripts/dbdump.sh data"0 5 * * * su - sybase -c "/sybase/bkscripts/dbdump.sh tape"55 5 * * * su - sybase -c "/sybase/bkscripts/dbdump.sh clean"0 6 * * 0 su - sybase -c "/sybase/bkscripts/dbdump.sh master"

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    16

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    17/40

    4 Daily Maintenance

    0 7 * * * su - sybase -c "/sybase/bkscripts/dbdump.sh tran"0 11 * * * su - sybase -c "/sybase/bkscripts/dbdump.sh tran"0 15 * * * su - sybase -c "/sybase/bkscripts/dbdump.sh tran"0 19 * * * su - sybase -c "/sybase/bkscripts/dbdump.sh tran"0 23 * * * su - sybase -c "/sybase/bkscripts/dbdump.sh tran"

    Apart from checking that the scheduling of backups is running and that the actual backups are

    created, the log file /sybase/log/SYB_BACKUP.log and the log file with relevant timestamp should

    be checked for any errors during creation of backups.

    It should be verified that the backup script dbdump.sh is configured correctly with the rightdatabase server and database name.

    Database Backup Procedure on High Availability Setup

    The primary computer takes the backup. If there are any problems with the primary computer, it

    should be fixed as soon as possible. If using the High Availability software, the secondary computer

    also takes the backups when it is active.

    If there are any problems with the fibres or with one disk storage array, the information is read from

    the other disk storage array.

    4.1.4 Verifying Tape Backups

    The tape should be changed every morning. You should have at least 9 tapes: one for each weekday, the 8th tape should be used every second Monday, and the 9th tape should be used once every

    month. If there is a need to have backups older than a month, the operator can use more tapes for

    this purpose as needed. This procedure eliminates the possibility of total loss of the database if one

    tape is faulty. The tapes should be tagged with their day or number according to their usage to avoid

    any confusion. Store the tapes in a safe place (protected from fire). You should never store backups

    in the same office/place where your Database Server is.

    4.2 Monitoring Size of Data Devices and Transaction Logs

    Time Any time

    Tool(s) to use ISQL, NMS Component Fault Monitoring Window

    Description Needed to check that the transaction logs are not filling up and thus

    preventing the usage of Tellabs 8000 intelligent network manager. It is also

    useful to check how much of the database device is currently used.

    Procedure

    Step 1 Open the Command Prompt window.

    Step 2 Start an ISQL session.isql -SDBSERVER -Usa -Ppassword

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    17

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    18/40

    4 Daily Maintenance

    Step 3 Check how much of the transaction log (free Kbytes in nms_log1) is used with the followingcommand.

    1>sp_helpdb db80002>go

    Result

    The transaction log should neverfill up if everything is functioning fine because it is dumped

    automatically by the backup procedure that you must have set up. If the transaction log fills up, you

    must dump it and find out why it filled up, because normally it should not happen.

    Step 4 Check how much free space there is in the data device (free Kbytes in nms_db1).1>sp_helpdb db80002>go

    Result

    The amount of free space should never drop below 10 Mbytes (10240 Kbytes). If this happens,

    check if it is caused by an excessive number of stored fault history events. If this is the case, delete a

    part of the fault history. Otherwise the network has grown so big that the size of the data device

    must be extended (contact your local Technical Support for Tellabs 8000 system). The audit log

    table can possibly cause the database to fill up too much. Delete or move the data from the audit

    log to another database if necessary.

    Alternatively, it is possible to configure Fault Management to show alarms when a data and log

    device is 50, 60, 70, 80 or 90 percent full. This requires that the thresh.qry script is modi fied to

    match the database name and that the script is run once every time the database devices are altered

    by adding a device or changing the size of a device. This script sets the threshold triggers to the

    database for sending an alarm message to the dbstate table. The NCAM_MONITOR_DB setting

    must be set to TRUE in the nms.ini file of the Fault Mapping Workstation to enable the polling of the

    dbstate table. Finally, the alarm levels must be configured in theConfigure Alarm Limits dialog

    -Add/Edit Alarm Limits dialog.

    Configuring Alarm Levels

    Step 1 Start Fault Management.

    Step 2 Select theOptions - NMS component monitoring menu option in the Fault Management mainwindow.

    Step 3 Double-click on the computer icon at the lower right hand corner of Network View.TheNMS Component Fault Monitoringwindow opens.

    Step 4 In the NMS Component Fault Monitoring window, SelectOptions Configure Alarm Limits

    Step 5 ClickAdd orEditbutton.

    Configure the alarms for different data and log devices and set the corresponding limits usingthe Add New Alarm Limit window.

    After this has been done, FMS shows an alarm message describing which device is filling up andwhich limit has been exceeded.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    18

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    19/40

    4 Daily Maintenance

    4.3 Checking Disk Space Usage and Logfiles on Database Server

    4.3.1 Checking Disk Space

    The disk space usage of all file systems of the Database Server should be checked if they are near full.

    On Solaris Database Server the command df k can be used to check disk space usage. Necessary

    action should be taken to clear space on those file systems which are near full by clearing large and

    old logfiles. On a Windows database server, the sybase log files (%SYBASE%\BACKUP.LOG and

    %SYBASE%\MS.LOG) should be emptied during the maintenance window when database is down.

    4.3.2 Investigating and Removing Core Files from Solaris Database Server

    Corefi

    les are created when a crash has occurred at the operating system level. So, these corefi

    lesshould be analyzed with the help of a solaris administrator to understand what caused the problem.

    In order to list the core dump files from the Sun Solaris Database Server run the following command.

    $ find / name core

    After moving the core files to a different location for investigation, they should be deleted on theserver to save space.

    $ find / name core print | xargs rm

    4.3.3 Truncating Log File on Solaris Database Server

    The running Database Server Log file cannot be deleted as ASE losesfile handle on it, so it must be

    purged by copying null to it as shown below.

    Step 1 Open a terminal connection to the Solaris server.

    Step 2 Go to log file location.$ cd /sybase/log

    Step 3 Copy the logfile to archive.$ cp DBSERVER.log DBSERVER.log.old

    Step 4 Purge the existing log file.$ cp /dev/null DBSERVER.log

    Step 5 Similarly, repeat the abovesteps 2 to 4 for/sybase/log/SYB_BACKUP.log.

    Step 6 If there is no need of old logs for any investigation, they should be deleted or moved to a differentstorage server.

    4.4 Checking Tellabs 8000 Intelligent Network Manager Servers

    Time Any time

    Tool(s) to use Fault Management System

    Description Needed to check that Management, Communication and Route Master

    Server(s) are functioning.

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    19

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    20/40

    4 Daily Maintenance

    Procedure

    Step 1 Start Fault Management.

    Step 2 Check the fault status of Tellabs 8000 intelligent network manager servers.

    Select theOptions - NMS component monitoring menu option in the Fault Management

    main window.

    Double-click on the computer icon at the lower right hand corner of Network View.

    The NMS Component Fault Monitoring window opens.

    Check that there are no faults for any Tellabs 8000 intelligent network manager servers.

    Result

    There should not be any faults in the servers. If there are, try to find out the reason and solve the

    problem, or contact your local Technical Support for Tellabs 8000 system.

    4.5 Checking Communication Links and Paths to All Network Nodes

    Time Any time

    Tool(s) to use Fault Management System

    Description Needed to check that communication links or paths to all Network Nodes

    are working.

    Procedure

    Step 1 Open theNetwork Fault Report for Active or Unacknowledged Faults.

    Select theFault - Network Fault Report Active or Unacknowledged Faults menu op-

    tion in the network window of Fault Management.

    Step 2 Check for any alarms indicating that communication links or paths with the Network Elementsare broken.

    Result

    Check that there are no used Network Nodes with broken communication links or paths.

    If any of them has an alarm, checkfirst that the cables from DXX Server(s) or Communication

    server(s) to the Tellabs 8000 system network are connected correctly. If this does not solve the

    problem, contact your local Technical Support for Tellabs 8000 system.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    20

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    21/40

    5 Weekly Maintenance

    5 Weekly Maintenance

    5.1 Verifying Database Configuration Parameters

    Time Night time or early in the morning

    Tool(s) to use AMPT, FMS and ISQL

    Description Check the usage of Database configuration parameters like Procedure Cache,

    number of open Partitions etc. The values of these parameters need to be

    readjusted based on their usage.

    In addition to periodic maintenance, note that database server parameters need to be revisedin cases of major changes in network size, number of users, large operations done withTellabs 8000 intelligent network manager, upgrades or change of database server hardware.Monitoring the database performance is quite essential in order to find out about needs tochange database server parameters. Refer to Sybase database server documentation forfurther information and contact your Sybase administrator to change database parameters.

    Procedure

    Step 1 Start the Automatic Maintenance Procedures Tool.

    SelectTools - Automatic Maintenance Procedures Tool in Tellabs 8000 Intelligent Net-

    work Manager Toolbox.

    Step 2 Select the procedureDB Server Housekeeper from the list available.

    Step 3 ClickExecute.

    Step 4 Login with thedxxdatalogin name and password and clickOK.Wait until theLast exec status is Successful.

    Step 5 Inspect the logs and perform corrective actions, if any.

    Inspect the logs in output section of the window for any errors.

    In the Fault Management System, open the Network Fault Report for Active or Unac-knowledged Faults and check for anyPrompt Maintenance Alarms (PMA)with problem

    type Database Server configuration required.

    Take corrective actions to increase parameter values by running ISQL commands as sug-

    gested in the Alarming object field description of the above type of alarms.

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    21

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    22/40

    5 Weekly Maintenance

    Step 6 While increasing memory based parameters likeProcedure Cache Size, Additional NetworkMemoryetc, verify that there is enough free memory available to be allocated, as shown below.

    Open a Command Prompt window and start an ISQL session as an sa user and check the

    max memoryparameter.isql -SDBSERVER -Usa Ppassword1> sp_monitorconfig max memory2> go

    The Num_freevalue should be more than the amount of intended increase suggested in

    the alarm.

    If there is enough free memory available, increase the parameters1> sp_configure ,2> go

    Step 7 If there is not enough free memory available to increase the memory based parameters, decreasethe number of sort buffers.

    Check the value of DB parameternumber of sort buffers.1> sp_configure number of sort buffers2> go

    Decrease thenumber of sort buffers to half of its current value displayed in the fieldRunValue.1> sp_configure number of sort buffers, 2> go

    Step 8 ClickExitto close the ISQL session and Automatic Maintenance Procedures Tool and the FMS.

    5.2 Verifying Status of Scheduled Procedures in Automatic

    Maintenance Procedures Tool

    Time Any time

    Tool(s) to use Automatic Maintenance Procedures Tool

    Description Verify the status of the different scheduled procedures in Automatic

    Maintenance Procedures Tool to check if they are running as planned and

    their intervals are not conflicting with each other.

    If your network grows extensively, e.g. new network elements are added or moved all the time,the execution times of different procedures scheduled by Automatic Maintenance Procedures

    Tool increase and may cause conflicts due to overlapping with each-other if the intervals setbetween them are not long enough.

    The execution status of all the scheduled procedures should be checked if there were any problems

    during execution. Also, the execution times should be verified so that they do not overlap each

    other. If there are any overlaps, intervals should be reorganized using steps in2.1 Activating

    Automatic Maintenance Procedures.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    22

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    23/40

    5 Weekly Maintenance

    Procedure

    Step 1 Start the Automatic Maintenance Procedures Tool.

    SelectTools - Automatic Maintenance Procedures Tool in Tellabs 8000 Intelligent Net-work Manager Toolbox.

    Step 2 Select a scheduled procedure from the list available.

    Step 3 Verify the Last exec timeof the procedure if it matches with the scheduling specified.

    Step 4 Verify the Last exec status of the procedure if it is successful.

    Step 5 If theLast exec statusis Failed, execute the procedure and inspect the logs for any errors.

    Select the procedure and click theExecutebutton.

    Login with thedxxdataas login name and password and clickOK.

    After completion of running the procedure, inspect the log in output section of the window

    for any errors. Take corresponding corrective actions for the errors.

    Step 6 Check theLast exec duration to determine how much time the procedure has taken to execute.Based on the execution time, it can be determined if there is any overlap between the different

    scheduled procedures. If there are any overlaps, intervals should be reorganized using steps in

    2.1 Activating Automatic Maintenance Procedures. The intervals can also be reduced if the

    execution times are short and the intervals are wider than necessary.

    Step 7 Similarly, repeat the steps 26 for every scheduled procedure in the list.

    5.3 Reorganizing Indexes in Tellabs 8000 Intelligent Network

    Manager Database

    Time Night time or early in the morning

    Tool(s) to use Automatic Maintenance Procedures Tool

    Description Reorganizing indexes is necessary for keeping the performance of the

    database access as good as possible. If you do not reorganize indexes on a

    regular basis, the performance suffers in the long run.

    If your network grows extensively, e.g. new network elements are added or moved all thetime, we advise you to reorganize the indexes more often, e.g. twice a week, to increase theperformance of the Database Server.

    The execution of the tasks may take some time depending on the size of the network. You can

    see the status of the operation in the Last exec status column. When the operation is completed

    the Last exec status is Successful.

    Procedure

    Step 1 Start the Automatic Maintenance Procedures Tool.

    SelectTools - Automatic Maintenance Procedures Tool in Tellabs 8000 Intelligent Net-work Manager Toolbox.

    Step 2 Select the Rebuild Indexes task.

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    23

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    24/40

    5 Weekly Maintenance

    Step 3 ClickExecute.

    Step 4 Login with thedxxdatalogin name and password and clickOK.Wait until the Last exec status is Successful.

    Step 5 If theLast exec status is Failed, inspect the logs for errors.

    Inspect the logs in output section of the window for any errors.

    Take corresponding corrective actions for the errors.

    Step 6 Similarly, repeat the abovesteps 2 to 5with the taskReclaim Unused Space.

    Step 7 ClickExit to close the Automatic Maintenance Procedures Tool or continue running othermaintenance procedures.

    5.4 Updating Database Statistics after Reorganizing Indexes

    Time Always after indexes have been reorganized

    Tool(s) to use Automatic Maintenance Procedures Tool

    Description Database statistics update is an operation that is needed to improve the

    performance of database access.

    The execution of the tasks may take some time depending on the size of the network. You can

    see the status of the operation in the Last exec status column. When the operation is completed

    the Last exec status is Successful.

    Procedure

    Step 1 Start the Automatic Maintenance Procedures Tool.

    SelectTools - Automatic Maintenance Procedures Tool in Tellabs 8000 Intelligent Net-

    work Manager Toolbox.

    Step 2 Select theUpdate Table Statistics task.

    Step 3 ClickExecute.

    Step 4 Login with thedxxdatalogin name and password and clickOK.Wait until the Last exec status is Successful.

    Step 5 If theLast exec status is Failed, inspect the logs for errors.

    Inspect the logs in output section of the window for any errors.

    Take corresponding corrective actions for the errors.

    Step 6 Similarly, repeat the abovesteps 2 to 5with the taskUpdate Index Statistics.

    Step 7 ClickExit to close the Automatic Maintenance Procedures Tool or continue running othermaintenance procedures.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    24

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    25/40

    5 Weekly Maintenance

    5.5 Checking Tellabs 8000 Intelligent Network Manager DatabaseConsistency

    Time Recommended to be run when the use of the network management system

    is light, e.g. early in the morning.

    Tool(s) to use Automatic Maintenance Procedures Tool

    Description This operation is needed for checking the consistency of the database. If the

    database has inconsistencies you may have serious problems in Tellabs 8000

    intelligent network manager system.

    Procedure

    Step 1 Start the Automatic Maintenance Procedures Tool.

    SelectTools - Automatic Maintenance Procedures Tool in Tellabs 8000 Intelligent Net-

    work Manager Toolbox.

    Step 2 Select theCheck/Fix Table Corruptionstask.

    Step 3 ClickExecute.

    Step 4 Login with thedxxdatalogin name and password and clickOK.Wait until the Last exec status is Successful.

    Step 5 If theLast exec status is Failed, inspect the logs for errors.

    Inspect the logs in output section of the window for any errors.

    Take corresponding corrective actions for the errors.

    Step 6 Similarly, repeat the abovesteps 2 to 5 with the taskCheck/Fix Index Corruptions.

    Step 7 ClickExit to close the Automatic Maintenance Procedures Tool or continue running othermaintenance procedures.

    If Database Server is not running in single user mode, you may receive the following errormessage:Table Corrupt: Page is allocated but not linked Msg 2540, Level 16, State 1:If you want to be sure about the consistency of the database, run the script checkdb in singleuser mode.

    5.6 Monitoring Size of Performance Data Size

    Time Any time

    Tool(s) to use 8000 Performance Management

    Description The PMS data size should be monitored in order not to let databasefill up

    too much.

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    25

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    26/40

    5 Weekly Maintenance

    Procedure

    Step 1 Start8000 Performance Management.From Toolbox selectPerformance Management 8000 Performance Management.

    Step 2 Open Performance Management Configuration.SelectPerformance Performance Management Configuration.

    Step 3 Check the size of the PMS data:See tabs8000 Configurationand 8100 Configurationand check theDatabase size MB fields.

    Result

    It is difficult to give a rule about the acceptable size for the PMS data in database. The acceptable

    size depends on the overall database size, database server type and memory as well as the impact

    on the duration of running of database maintenance scripts. However, tables which are larger than

    several megabytes might cause problems in the SQL server and decrease its performance.

    You can affect the size of the PMS data by setting the corresponding Interval historyand agingdaysvariables in the Performance Management Configuration window.

    5.7 Testing Restoration of Backup

    Time Any time

    Tool(s) to use ISQL

    Description Test the restoration of Tellabs 8000 intelligent network manager database

    backup once a week. Use a second database (e.g. sim_secdb8000) for this

    purpose.

    This procedure is possible only if you have enough space on your Database Server to create asecond database with the same characteristics, e.g. the size of devices, as in your productiondatabase, or if you have another server on which the database dump can be loaded. However,it is highly recommended that a separate isolated Database Server is used to avoid anyaccidental interruption on the production Database Server.

    Windows Server 2008 does not natively support tape backups. SeeTellabs 8000 IntelligentNetwork Manager Software Installation Manual, chapterDatabase Backup Procedures forWindows 2003/2008 Database Serverfor more information. In Windows Server 2008, copy thebackup from the used external media to test restorationof a backup.

    Procedure

    Step 1 Create a database sim_secdb8000 that has the same characteristics as your network database, e.g.the size of devices. See Tellabs 8000 Intelligent Network Manager Software Installation Manual

    for instructions on creating a database.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    26

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    27/40

    5 Weekly Maintenance

    Step 2 From Database Server, restore the database dump from the tape to the disk following the instructionsbelow.

    If you are using a Windows 2003based Database Server, proceed as follows:

    Open the backup program by selectingStart - Programs - Accessories - System Tools -Backup.

    Open the tape.

    Select the dump and clickrestore.

    If using a Solarisbased Database Server, type in the shell:

    $cd /backup$mkdir restore$cd restore$tar xvf /dev/rmt/0

    Step 3 Load the dump intosim_secdb8000.

    Open the Command Prompt window.

    Start an ISQL session.isql -SDBSERVER -Usa Ppassword

    Load the dump into sim_secdb8000.

    If you are using a Windows 2003/2008based Database Server, enter the following

    command:

    1> load database sim_secdb8000 from g:\backup\dumpdb.bkp2> go

    If you are using a Solarisbased Database Server, enter the following command:

    1> load database sim_secdb8000 from /backup/re-store/dumpdb.bkp2> go

    Result

    There should be no errors occurring during the load operation.

    Step 4 In case transaction dumps are being tested too, load the transaction dumps as shown below, oneafter the other in the same order that they were created. If transaction dumps are not being tested,

    skip this step and proceed from step 5 onwards.

    Load the transaction dump into sim_secdb8000.If you are using a Windows 2003/2008based Database Server, enter the following

    command:

    1> load tran sim_secdb8000 from g:\backup\dumptran.bkp2> go

    If you are using a Solarisbased Database Server, enter the following command:

    1> load tran sim_secdb8000 from /backup/re-store/dumptran.bkp2> go

    Repeat the above for each transaction dump, in the same order that they were created.

    Result

    There should be no errors occurring during the load operation.

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    27

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    28/40

    5 Weekly Maintenance

    Step 5 Bringsim_secdb8000online and check the NMS table.1>online database sim_secdb80002>go

    1>use sim_secdb80002>go

    1> select * from nms2> go

    Result

    The command must have been executed and it should display information of the installed Tellabs

    8000 intelligent network manager.

    Step 6 Check if the database backup contains up-to-date information using below commands:1>select max(time) from sim_secdb8000..osrchan2>go

    ResultThe result should be close to the time when the database backup was taken. If not it is likely that the

    backup is older in time than it is thought to be.

    It is very important that you do not perform any hardware actions using this database (e.g.update parameters in a network element) with a workstation unless you are sure that thereis no connection to the production network from the server running the sim_secdb8000database. The production database could be damaged and cut off traffic. You can alsocreate a simulation database instead of a second database in order to avoid this problem.Completing steps 5 and 6 should be enough to check that the restoration of the database isfunctioning properly.

    If you fail to restore your Tellabs 8000 intelligent network manager database backup, contactyour local Technical Support for Tellabs 8000 system.

    5.8 Checking Disk Space of Tellabs 8000 Intelligent Network ManagerComputers (and Moving Log Files)

    Time Any time

    Tool(s) to use Commands and scripts

    Description Needed to check that there is enough disk space available for log writing.

    Procedure

    Step 1 On each computer where you are getting logfiles, e.g. nmstrace.log, you should check that the harddisk is not filling up. No more than 90 per cent of the disk space should be used. To make this step

    easier, you can use the parametery to cut log files at midnight with the following command.nmstrace 0123456 * trace.log y

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    28

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    29/40

    5 Weekly Maintenance

    Step 2 Check if there are any unnecessary logfiles and delete them.

    If you find any unnecessary log files, select the directory of the file and delete the files.

    There may be logs in the %NMSDIR%\LOG folder. In addition, Phoenix.log is located in

    \%NMSDIR%\Java\Extern\Phoenix\Logs.

    Delete other unnecessaryfiles.

    Step 3 As long as there are no errors or unsolved entries in the nms trace logfiles, there is no need to keepthem to occupy disk space, so delete or move the log files to another computer via network.

    5.9 Checking Disk Space on ThinClient Server

    Time Any time

    Tool(s) to use Commands

    Description Needed to check that there is enough disk space available for log writing onThinClient Server. ThinClient Server maintains session data ex. logs for all

    the users accessing NMS in their corresponding directories. The nms logs

    created by these individual sessions should be regularly leaned up in order

    to ensure there is enough disk space on the ThinClient Server.

    Procedure

    Step 1 The disk space usage of allfile systems of the ThinClient server should be checked if they are nearfull. No more than 90 per cent of the disk space should be used. The following main nms directory

    should be checked if there are any excessive log files and they should be deleted.%NMSDIR%\log

    Step 2 Check if there are any excessive logfiles in all the user sessions created in ThinClient server.

    The individual user session nms logs are in the directories C:\shared\users\cit-

    rix_users\\Tellabs 8000\log.

    Make sure the workstation of the corresponding user is not running and delete all excessive

    log files in each directory.

    5.10 Checking Disk Space on Route Master

    Time Any time

    Tool(s) to use Commands

    Description Needed to check that there is enough disk space available for log writing on

    Route Master server.

    Procedure

    Step 1 The disk space usage of all file systems on the Route Master should be checked if they are near full.The command df k can be used to check diskspace usage. No more than 90 per cent of the disk

    space should be used. Necessary action should be taken to clear space on those file systems which

    are near full by clearing large and old logfiles.

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    29

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    30/40

    5 Weekly Maintenance

    Step 2 Empty excessive logs in the directory /var/log/routemaster, if there are any.

    Open a terminal connection to the Route Master server.

    Go to log file location.

    $ cd /var/log/routemaster

    Copy the logfile to archive$ cp ocnm_mon.log ocnm_mon.log.old

    Purge the existing logfile.$ cp /dev/null ocnm_mon.log

    Step 3 Similarly, repeat the abovestep 2for below logs/var/log/routemaster/nsm.log

    /var/log/routemaster/ospfd.log

    Step 4 If there is no need of old logs for any investigation, they should be deleted or moved to a differentstorage server.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    30

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    31/40

    6 Monthly Maintenance

    6 Monthly Maintenance

    6.1 Deleting Audit Log

    Tool(s) to use Tellabs 8000 Intelligent Network Manager Toolbox

    Description Delete the Audit Log entries that are older than one month. In the long run,

    the auditlog table becomes so big that it is necessary to delete part of the

    old entries. Otherwise, the performance of Tellabs 8000 intelligent network

    manager system suffers. If you want to save all entries, you must always save

    the latest database backup before deleting the Audit Log.

    This procedure creates a large amount of transaction log data. Checkfirst that youhaveenough free space in your log device before executing this operation.

    Procedure

    Step 1 Select the Audit Log entry truncation date.

    Select theSecurity - Audit Logs - Manager Audit Log menu option in Tellabs 8000 Intel-

    ligent Network Manager Toolbox.The Audit Logwindow opens.

    SelectOptions - Refresh.

    Select theOptions Delete Events menu option.

    The Deletedialog opens.

    Define the deletion time to one month earlier than the current date.

    Step 2 Delete the faults.

    Click the Okbutton.

    The specified Audit Log events are deleted from the database.

    6.2 Monitoring Size of Fault Tables

    Tool(s) to use ISQL

    Description It is very important to monitor the size of the tables that can grow a lot in

    the database. In some cases, if a table is too big, the SQL server can have

    problems creating indexes, or some entries can become corrupted.

    The performance of the database decreases a lot if some tables are too big.

    Step 1 Start an ISQL session.isql -SDBSERVER -Usa -Ppassword

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    31

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    32/40

    6 Monthly Maintenance

    Step 2 Check the size of the fault tables with the following commands.1>use db80002>go

    1>sp_spaceused faulthist2>go

    1>sp_spaceused fault2>go

    Result

    It is rather difficult to give a rule about the acceptable size of a table. The acceptable table size

    depends on various parameters such as the computer type and the amount of memory. However,

    the tables which are bigger than several megabytes might cause problems in the SQL server and

    decrease its performance.

    If the fault history table is too big, you should keep the entries corresponding to a shorter period

    of time as mentioned in chapter3 Fault History Maintenance.

    If the fault table is too big, start clearing up the faults in your network more efficiently.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    32

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    33/40

    6 Monthly Maintenance

    6.3 Accounting Management Events

    This creates a large amount of transaction log data. Check first that you have enough freespace in your log device using DB command sp_helpdevice before executing this operation.

    Time Any time

    Tool(s) to use Accounting Management

    Description Delete the events that are older than one month. If you want to save all

    events, you must always save the latest database backup before deleting the

    events. Contact your local Technical Support for Tellabs 8000 system for

    more detailed instructions on backing up and restoring a database.

    Procedure

    Step 1 Start theAccounting Management tool.

    Step 2 Click theDeletebutton.

    Step 3 Type a month-old date. All events that occurred before that date is deleted.

    Step 4 ClickOK.

    Step 5 ClickYes.

    6.4 Private Subnetwork Trunk Accounting Events for Tellabs 8100Nodes

    This creates a large amount of transaction log data. Check first that you have enough freespace in your log device using DB command sp_helpdevice before executing this operation.

    Time Any time

    Tool(s) to use Private Subnetwork Trunk Accounting

    Description Delete the events that are older than one month. If you want to save all events

    you must always save the latest database backup before deleting the faulthistory. Contact your local Technical Support for Tellabs 8000 system for

    more detailed instructions on backing up and restoring a database. Note that

    this operation is needed only if you have defined private subnetworks for

    8100 nodes in your Tellabs 8000 system network.

    Procedure

    Step 1 Start thePrivate Subnetwork Trunk Accounting tool by selecting Tools Private SubnetworkTrunk Accounting.

    Step 2 Select a private subnetwork and clickOK.

    Step 3 Open theDelete Events dialog by selecting Options - Delete Events menu option.

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    33

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    34/40

    6 Monthly Maintenance

    Step 4 Define the deletion time in YYYYMMDD format. All events that occurred before that date isdeleted.

    Step 5 ClickOK.

    6.5 Recovery Management System

    6.5.1 Spontaneous Fault Reporting (GMH Units)

    We recommend that you set spontaneous fault reporting systematically when setting parameters to

    trunk interfaces with Node Manager.

    Step 1 Open the unit window for a GMH unit using Node Manager.

    Step 2 Open theFault Block Configurationdialog by selectingUnit - Fault block Configuration.

    Step 3 SetEvent Reporting on.

    Step 4 SetFault Filtering Timeaccording to your needs. The default value is 0.

    6.5.2 Spontaneous Fault Reporting (GMU, GMU-A and GMU-M1 Units)

    Time Any time, after adding a new VC-12, VC-2, VC-12-mc or VC-2-mc trunk

    Tool(s) to use Node Manager

    Description It is necessary to set the spontaneous fault sending on in trunk interfaces.

    This is needed for faster Tellabs 8000 intelligent network manager recovery

    operations. Otherwise the recovery is much slower.

    Procedure

    Step 1 Open the unit window for the GMU that contains the VC-12, VC-2, VC-12-mc or VC-2-mc TTPs ofthe new trunks.

    Step 2 Right-click the mouse to open theFault Block Configurationdialog.

    Step 3 SetEvent Reporting on.

    Step 4 SetFiltering Timeaccording to your needs. The default value is 0.

    Step 5 SetFault Declare Time according to your needs. The default value is 2 s.

    Step 6 SetFault Clear Time according to your needs. The default value is 10 s.

    6.5.3 Spontaneous Fault Reporting (Other Units)

    Fault reporting can be set on with the same principle as for GMH in XCG (trunk interfaces),

    GMM, VMM, and QMH.

    1GMU and GMU-M have been discontinued.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    34

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    35/40

    6 Monthly Maintenance

    6.6 Maintenance of Element Software in Database

    Time Whenever new ESW is available

    Tool(s) to use Tellabs 8000 Intelligent Network Manager Toolbox

    Description It should be taken care on a monthly basis or even more often that the relevant

    versions of element software are loaded into the database and obsolete

    versions are deleted.

    Procedure

    Step 1 In Tellabs 8000 Intelligent Network Manager Toolbox selectTools - 8600 Element SoftwareManagement...

    The8600 Element Software Management window opens.

    Step 2 ClickAdd ESW.The Add ESW File to DB window opens.

    Step 3 ClickSelect FileNavigate to the desired path containing the relevant ESW files and select a fileyou want to add into the database.

    Step 4 Fill in theVersion, File Type,Description,Info Model Versionand Unit Type fields.

    Step 5 ClickCopy to DB.

    Step 6 When the copying is finished, clickExit.

    Step 7 Delete unnecessary files in 8600 Element Software Management window by selecting the filefrom the list and clicking Delete.

    6.7 Node Configuration Backup

    Time Whenever new ESW is available

    Tool(s) to use Tellabs 8000 Intelligent Network Manager Toolbox

    Description In Tellabs 8000 intelligent manager it is possible to store a node configuration

    into the database using Node Configuration snapshot or backup.

    Procedure

    Step 1 In Tellabs 8000 Intelligent Network Manager Toolbox selectNetwork Editor.The Network Editor window opens.

    Step 2 SelectEdit Communication ServersTheNWED:Communication Server Parameters dialog opens.

    Step 3 Select the relevant communication server from theNamedrop-down list, and select the relevantadapter to parametrize fromCommunication Adapterswindow pane.

    Step 4 ClickParametersThe NWED:Tellabs 8600 Adapter dialog opens.

    Step 5 ClickPolling PoliciesTheNWED:Polling Policy Parameters dialog opens.

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    35

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    36/40

    6 Monthly Maintenance

    Step 6 Tick theConfiguration Backupselection box and clickParametersnext to it.The NWED:Configuration Backupdialog opens.

    Step 7 Select eitherPeriodic backuporDaily backupradio button, and fill in the relevant parameters and

    the number of backup files per node.

    Step 8 ClickOKto confirm the parameters and close the dialog.

    Step 9 ClickUpdateon theNWED:Polling Policy Parametersdialog to activate the policy.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    36

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    37/40

    7 Additional Tools for Sun Solaris Server

    7 Additional Tools for Sun Solaris Server

    7.1 Sun Explorer

    Sun Explorer is a data collection tool, which is used for collecting information on the

    configuration, version and logs to help problem solving or system audits. The data is stored in the

    /opt/SUNWexplo/outputdirectory and can be read by the system administrator at any time. The

    data can also be transferred to the Sun Microsystems database from where the Tellabs Technical

    Support personnel can retrieve them when necessary.

    The Explorer software can be downloaded fromhttp://sunsolve.sun.com(click onExplorer Data

    Collectorunder theDiagnostic Tools title). You can download the software by clicking on View

    License Agreement and Download. In addition, there is a link to Frequently Asked Questions

    where extra information on the Explorer software can be found.

    7.2 Installation of Explorer

    Explorer is provided by Sun Microsystems as SUNWexplo.tar.Z, which is a compressed tar kit ready

    to be installed. The installation commands are as follows.

    #cp SUNWexplo.tar.Z /var/tmp#cd /var/tmp#zcat SUNWexplo.tar.Z | tar xvf #pkgadd d . SUNWexplo

    The last command starts the installation process. During the installation you have to answer several

    questions which determine the operation of Explorer. The questions with example answers can be

    found below (Explorer version 4.0).

    Agree to these terms? [y, n] YWhere should this package be installed? [/opt/ SUNWexplo]:/opt/ SUNWexplo

    Contact Information NOTE! Fill all the possible fields.Geographic Region: EMEAPlease Enter the two characters Country code or enter ? for alist of countries. []:? FiWould you like explorer output to be sent to alternate emailaddresses at the completion of explorer? If not, enter asingle only for your reply.Return address for explorer email output: [email protected]

    Are these values ok? [y,n]Do you wish to run explorer once a week? [y,n] y

    At which hour would you like this to run (0 to 23): Select theone best suited for your needs

    At which minute would you like this to run (0 to 59): Selectthe one best suited for your needs

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    37

    http://sunsolve.sun.com/http://sunsolve.sun.com/
  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    38/40

    7 Additional Tools for Sun Solaris Server

    On which day would you like this to run (0 to 6 with 0=Sunday):Select the one best suited for your needs

    If this is a new install of explorer, please run explorer andmail the results to Sun. The command to do this would be:/opt/SUNWexplo/bin/explorer q eWould you like to do this now? [y,n] yDo you want to continue with the installation of [y,n,?] y

    The information is stored in the file called /etc/opt/SUNWexplo/default/explorer and you can edit it

    at any time.

    7.3 Configuration and Service Tracker (CST) Latest Version

    Configuration and Service Tracker (CST) is a software that helps the system administrator to register

    and archive the changes and activities done in the system. CST also calculates availability and

    keeps log of the system architecture.

    CST monitors e.g. the following events:

    hardware, software and patch installations, removals and upgrades

    configuration changes in the operation systems

    shut down, reboot, panic and run-level type of status changes

    cluster level changes

    CST calculates the availability of the system using the above-mentioned events. CST can be

    configured to send an e-mail when a change is noticed.

    CST software can be downloaded free of charge from Sun Download Center at:

    http://www.sun.com/software/download(clickfirst on System Administration, and then on

    System Management).

    An alternative site is http://srsnetconnect3.sun.com, where the installation manuals and other

    documents can be found.

    Tellabs 8000 Intelligent Network Manager SR2.1 76.8021-70211AMaintenance Procedures 2011 Tellabs.

    38

    http://www.sun.com/software/downloadhttp://srsnetconnect3.sun.com/http://srsnetconnect3.sun.com/http://www.sun.com/software/download
  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    39/40

    Suggested Weekly Schedule for Automatic Maintenance Procedures

    Suggested Weekly Schedule for AutomaticMaintenance Procedures

    The following is suggested week-wise scheduling of automatic maintenance procedures. These can

    only be used as a guideline as the running times of various procedures vary based on the size of

    the network, hence they need to be adjusted as per requirement.

    On Windows Database Server

    Mon Tue Wed Thu Fri Sat Sun

    00:00 tape

    backup

    tape

    backup

    tape

    backup

    tape

    backup

    tape

    backup

    tape

    backup

    tape

    backup

    03:00 trans-

    action

    dump

    transac-

    tion dumptransaction

    dump

    trans-

    action

    dump

    transac-

    tion dump

    transac-

    tion dump

    trans-

    action

    dump

    03:30 rebuild

    indexes

    check/fix

    table

    corruptions

    04:30 reclaim

    unused

    space

    05:30 update

    table

    statistics

    check/fix

    index

    corruptions

    06:20 update

    index

    statistics

    06:30 server

    moni-

    toring

    house-

    keeper

    server

    monitor-

    ing house-

    keeper

    server

    monitor-

    ing house-

    keeper

    server

    moni-

    toring

    house-

    keeper

    server

    moni-

    toring

    house-

    keeper

    server

    moni-

    toring

    house-

    keeper

    server

    moni-

    toring

    house-

    keeper

    07:00 trans-

    action

    dump

    transac-

    tion dumptransaction

    dump

    trans-

    action

    dump

    transac-

    tion dump

    transac-

    tion dump

    trans-

    action

    dump

    07:30 dbserver

    house-

    keeper

    db serverhouse-

    keeper

    db serverhouse-

    keeper

    db serverhouse-

    keeper

    db serverhouse-

    keeper

    db serverhouse-

    keeper

    db serverhouse-

    keeper

    11:00 trans-

    action

    dump

    transac-

    tion dumptransaction

    dump

    trans-

    action

    dump

    transac-

    tion dump

    transac-

    tion dump

    trans-

    action

    dump

    15:00 trans-

    action

    dump

    transac-

    tion dumptransaction

    dump

    trans-

    action

    dump

    transac-

    tion dump

    transac-

    tion dump

    trans-

    action

    dump

    19:00 trans-

    action

    dump

    transac-

    tion dumptransaction

    dump

    trans-

    action

    dump

    transac-

    tion dump

    transac-

    tion dump

    trans-

    action

    dump

    76.8021-70211A Tellabs 8000 Intelligent Network Manager SR2.1 2011 Tellabs. Maintenance Procedures

    39

  • 8/10/2019 Tellabs 8000 Intelligent Network Manager

    40/40

    Suggested Weekly Schedule for Automatic Maintenance Procedures

    On Windows Database Server

    22:00 master

    dump

    23:00 full dbdump

    full dbdump

    full dbdump

    full dbdump

    full dbdump

    full dbdump

    full dbdump

    On Solaris Database Server

    Mon Tue Wed Thu Fri Sat Sun

    00:00 rebuild

    indexes

    check/fix

    table

    corruptions

    01:00 reclaim

    unused

    space

    02:00 update

    table

    statistics

    check/fix

    index

    corruptions

    03:00 update

    index

    statistics

    04:00 full db

    dump

    full db

    dump

    full db

    dump

    full db

    dump

    full db

    dump

    full db

    dump

    full db

    dump

    05:00 tape

    backup

    tape

    backuptape backup tape

    backup

    tape

    backup

    tape

    backup

    tape

    backup

    05:55 clean up clean up clean up clean up clean up clean up clean up

    06:00 masterdump

    06:30 server

    moni-

    toring

    house-

    keeper

    server

    monitor-

    ing house-

    keeper

    server

    monitor-

    ing house-

    keeper

    server

    moni-

    toring

    house-

    keeper

    server

    moni-

    toring

    house-

    keeper

    server

    moni-

    toring

    house-

    keeper

    server

    moni-

    toring

    house-

    keeper

    07:00 trans-

    action

    dump

    transac-

    tion dumptransaction

    dump

    trans-

    action

    dump

    transac-

    tion dump

    trans-

    action

    dump

    transac-

    tion dump

    07:30 db

    server

    house-

    keeper

    db server

    house-

    keeper

    db server

    house-

    keeper

    db server

    house-

    keeper

    db server

    house-

    keeper

    db server

    house-

    keeper

    db server

    house-

    keeper

    11:00 trans-

    action

    dump

    transac-

    tion dumptransaction

    dump

    trans-

    action

    dump

    transac-

    tion dump

    trans-

    action

    dump

    transac-

    tion dump

    15:00 trans-

    action

    dump

    transac-

    tion dumptransaction

    dump

    trans-

    action

    dump

    transac-

    tion dump

    trans-

    action

    dump

    transac-

    tion dump

    19:00 trans-

    action

    dump

    transac-

    tion dumptransaction

    dump

    trans-

    action

    dump

    transac-

    tion dump

    trans-

    action

    dump

    transac-

    tion dump

    23:00 trans-

    action

    dump

    transac-

    tion dumptransaction

    dump

    trans-

    action

    dump

    transac-

    tion dump

    trans-

    action

    dump

    transac-

    tion dump