0513_3 Workstation Network Maintenance

Document number/Issue Copyright © Nokia Telecommunications Oy

NTC TAN 0513/3 en

1

NOKIA NMS/2000FOR MANAGING CELLULAR NETWORKS

WORKSTATION NETWORK MAINTENANCE

System Administrator's Guide


NTC TAN 0513/3 en2

The information in this document is subject to change without notice and describes only the product definedin the introduction of this documentation. This document is intended for the use of NokiaTelecommunications' customers only for the purposes of the agreement under which the document issubmitted, and no part of it may be reproduced or transmitted in any form or means without the prior writtenpermission of Nokia Telecommunications. The document has been prepared to be used by professional andproperly trained personnel, and the customer assumes full responsibility when using it. NokiaTelecommunications welcomes customer comments as part of the process of continuous development andimprovement of the documentation.

The information or statements given in this document concerning the suitability, capacity, or performance ofthe mentioned hardware or software products cannot be considered binding but shall be defined in theagreement made between Nokia Telecommunicat ions and the customer. However, NokiaTelecommunications has made all reasonable efforts to ensure that the instructions contained in thedocument are adequate and free of material errors and omissions. Nokia Telecommunications will, ifnecessary, explain issues which may not be covered by the document.

Nokia Telecommunications' liability for any errors in the document is limited to the documentary correction oferrors. Nokia Telecommunications WILL NOT BE RESPONSIBLE IN ANY EVENT FOR ERRORS IN THISDOCUMENT OR FOR ANY DAMAGES, INCIDENTAL OR CONSEQUENTIAL (INCLUDING MONETARYLOSSES), that might arise from the use of this document or the information in it.

This document and the product it describes are considered protected by copyright according to theapplicable laws.

NOKIA logo is a registered trademark of Nokia Corporation.

Other product names mentioned in this document may be trademarks of their respective companies, andthey are mentioned for identification purposes only.

Copyright © Nokia Telecommunications Oy 1997. All rights reserved.

No. of Edited by Author Approved by Previous issuepages (2) approved

M. Ganszauge M. Ganszauge J. Karppelin 15 May 1997101/MiG 22 Jan 1998 22 Jan 1998 22 Jan 1998


NTC TAN 0513/3 en3

TABLE OF CONTENTS

1 WHAT IS NEW. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.1 Changes from T8 to T9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.2 Changes from T9 to T10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 ABOUT THIS MANUAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.1 What you need to know first . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2 How to use this manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.3 Where to find more . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.4 Typographic conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.4.1 Text styles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.4.2 Command line conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.5 Concepts and terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.5.1 Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3 QUICK REFERENCE TO ADMINISTRATIVE TASKS . . . . . . . . . 173.1 Maintenance tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.1.1 Periodic tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.1.2 Occasional tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4 FAULT MANAGEMENT TASKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.1 Modifying alarm lifting rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.2 Adding and deleting alarm pipes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234.3 Checking that network monitoring works . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.3.1 Verifying that the monitored network element is online . . . . . . . 264.3.2 Verifying that the NMS is showing the true state of the network 27

4.4 Blocking alarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284.4.1 Blocking alarms in a specific network element . . . . . . . . . . . . . . 284.4.2 Blocking BTS alarms in the BSC . . . . . . . . . . . . . . . . . . . . . . . . 284.4.3 Disabling and enabling network alarms and measurements in the

workstations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5 PERFORMANCE MANAGEMENT TASKS . . . . . . . . . . . . . . . . . . . 305.1 Monitoring the network performance data. . . . . . . . . . . . . . . . . . . . . . . . . 30

6 TAKING BACKUPS OF THE NOKIA NMS . . . . . . . . . . . . . . . . . . . 326.1 About backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.2 Taking a full backup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.3 Backing up the critical files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396.4 Restoring a full backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396.5 Restoring a critical files backup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416.6 Taking a backup of the files in the Front End . . . . . . . . . . . . . . . . . . . . . . 416.7 Restoring a Front End backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426.8 Taking a backup of the router configuration . . . . . . . . . . . . . . . . . . . . . . . 43


NTC TAN 0513/3 en4

7 UNIX ADMINISTRATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457.1 System shutdown and startup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457.2 Shutting down and starting up the system . . . . . . . . . . . . . . . . . . . . . . . . . 477.3 Rebooting the NMS servers and workstations . . . . . . . . . . . . . . . . . . . . . . 497.4 Shutting down and starting up the Nokia NMS . . . . . . . . . . . . . . . . . . . . . 507.5 Administering MC/ServiceGuard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

7.5.1 Halting the MC/ServiceGuard packages . . . . . . . . . . . . . . . . . . . 517.5.2 Starting the MC/ServiceGuard packages . . . . . . . . . . . . . . . . . . . 527.5.3 Halting an MC/ServiceGuard node . . . . . . . . . . . . . . . . . . . . . . . 537.5.4 Starting an MC/ServiceGuard node . . . . . . . . . . . . . . . . . . . . . . . 547.5.5 Halting the MC/ServiceGuard cluster . . . . . . . . . . . . . . . . . . . . . 557.5.6 Starting the MC/ServiceGuard cluster . . . . . . . . . . . . . . . . . . . . . 567.5.7 Shutting down and restarting a server after halting a node or the

whole cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 577.5.8 Enabling MC/ServiceGuard node switching . . . . . . . . . . . . . . . . 577.5.9 Mounting and unmounting file systems within a MC/ServiceGuard

cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 587.5.9.1 Mounting the disks in clustering mode . . . . . . . . . . . . . 597.5.9.2 Mounting the disks in non-clustering mode . . . . . . . . . 59

7.5.10 Verifying the functionality of the MC/ServiceGuard cluster. . . . 607.6 Managing the Nokia NMS processes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

7.6.1 Configuring Process Supervision and Recovery Service. . . . . . . 617.6.2 Terminating and starting the Process Supervision and Recovery

application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 627.6.3 Terminating and starting a process under supervision. . . . . . . . . 62

7.7 Resource Supervision Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647.7.1 Configuring disk supervision. . . . . . . . . . . . . . . . . . . . . . . . . . . . 647.7.2 Monitoring the disk space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657.7.3 Monitoring log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

7.7.3.1 Log files monitored by the system. . . . . . . . . . . . . . . . . 667.7.3.2 Log files to be monitored by the system administrator . 67

7.7.4 Removing core image files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 687.8 Generic Supervision Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

7.8.1 Supervising the X.25 adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . 697.8.2 Supervising mirrored disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 697.8.3 Supervising the database index tablespaces. . . . . . . . . . . . . . . . . 707.8.4 Supervising the flow of BSC traffic measurements. . . . . . . . . . . 707.8.5 Supervising the workstation message service . . . . . . . . . . . . . . . 707.8.6 Supervising the OSI connections . . . . . . . . . . . . . . . . . . . . . . . . . 71

7.9 Modifying the Alarm Flow Supervision parameters . . . . . . . . . . . . . . . . . 727.10 Checking the processes manually . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 737.11 Monitoring the wcltoday.log File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 747.12 Checking revisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

7.12.1 Creating revision files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 757.12.2 Comparing revisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

7.13 Managing crontab files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 797.14 Tasks on the Nokia NMS servers with no MC/ServiceGuard software. . . 81

7.14.1 Shutting down and starting up the Nokia NMS servers. . . . . . . . 81


NTC TAN 0513/3 en5

7.14.2 Starting up and shutting down the Nokia NMS . . . . . . . . . . . . . . 817.14.3 Closing and opening the X.25 connections . . . . . . . . . . . . . . . . . 847.14.4 Starting up and shutting down the Nokia NMS database . . . . . . 85

8 SYSTEM MODIFICATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 878.1 Setting up a printer in Alarm Viewer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 878.2 Installing customised sounds for Alarm Viewer . . . . . . . . . . . . . . . . . . . . 888.3 Changing the IP address of a server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 898.4 Configuring the TFTP server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 918.5 Changing the NIS configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 928.6 Exporting directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 948.7 Restricting access to certain services on the local host . . . . . . . . . . . . . . . 96

INDEX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

What is new


NTC TAN 0513/3 en6

1 WHAT IS NEW

This chapter outlines the most significant changes that have occurred in theNokia NMS software, related HP-UX operating system software anddocumentation since release T8. If you are an experienced user of the NokiaNMS, we strongly advise you to read this section to get an overview of whatsystem-level changes have been implemented in Nokia NMS release T9 andT10. New features in applications and new applications are described in therespective manuals.

1.1 Changes from T8 to T9

This section describes the changes that have occurred in the Nokia NMSworkstation maintenance procedures and in the documentation between releasesT8 and T9 of the Nokia NMS.

Supervision Framework

Nokia NMS now includes a feature called Supervision Framework whichmonitors the X.25 adapters, BSC traffic measurements, database indextablespaces, mirrored disks, WS NW message service and OSI connections.Formore information see 7.8,Generic Supervision Service.

Oracle database

The Oracle database version used in T9 was 7.2.3. Also, three new scripts werecreated to verify the structure of the database.

1.2 Changes from T9 to T10

The following changes were made between release T9 and T10:

HP-UX operating system

The HP-UX operating system version has been upgraded from 10.01 to 10.20.For more information on what is new for HP-UX 10.20, seeHP-UX 10.20Operating System Help in the HP Help Manager.

Oracle database

The Oracle database version used in T10 is 7.3.2. As a result, the NMS databasefile structure has changed from previous releases. For more information, seeDatabase Maintenance, TAN0514.

NMS documentation

NMS documentation is now available online. Information on basic tasks and userinterfaces previously found in the User’s Guides can now be found in therespective application help. If you have the optional Online Library installed, alldocumentation is available to you via the NMS Online Help. For more

What is new


NTC TAN 0513/3 en7

information, see Section 2.2,Where to find information on system managementin System Management Basic Operating Principles and Procedures, TAN 0732.

New alarm system

The Nokia NMS has a new alarm system that uses pipes and supports alarmcollection from third-party network elements. For more information, seeSystemManagement Basic Operating Principles and Procedures, TAN 0732, FaultManagement Basic Operating Principles and Procedures, TAN 0704 andChapter 4,Fault Management tasks in this manual.

About this manual


NTC TAN 0513/3 en8

2 ABOUT THIS MANUAL

The set of System Administrator's Guides for the Nokia NMS is intended for thesystem administrators of the Nokia NMS. In general, system administrators arethe most privileged users in the network and their responsibility is to set up andmaintain the environment for normal users. The Nokia NMS systemadministrator usually also acts as the system administrator of the UNIX servers(the root user). This manual explains the tasks the system administrator needsto carry out to ensure that the Nokia NMS remains functional at all times.

The descriptions in this manual are release-specific, so we urge you not to useany previous issues of this document with Nokia NMS release T10. Thesupported releases of network element software are S6 for BSCs and M7B forMSCs unless otherwise stated.

This document also deals with the HP 9000 servers and workstations and HP UXversion 10.20, unless otherwise stated. The manual describes the StandardServer Configuration for Workstations which comprises one CommunicationsServer, one Database Server, one Standby Server and one or more ApplicationServers. If your server configuration is different, you may have to carry out thetask on different servers from the ones mentioned in the procedure descriptions.For more information on Nokia NMS server configurations, see theTechnicalReference Guide, TAN 0453.

More specifically, this document covers the following topics:

• Chapter 3,Quick reference to administrative tasks introduces the basicconcepts used in maintaining Nokia NMS workstation network. Thechapter also explains the difference between periodic and occasional tasks.

• Chapter 4,Fault Management tasks explains the basic procedures relatedto monitoring and investigating alarms from the network elements, andmodifying the alarm system operation.

• Chapter 6,Taking backups of the Nokia NMS provides instructions fortaking and restoring a full backup of the Nokia NMS workstations, andtaking and restoring the most critical files in the system.

• Chapter 7,UNIX administration describes the tasks related to the built-insupervision and recovery functionality of the Nokia NMS and outlines themost common tasks related to regular system administration of the WSnetwork.

• Chapter 8,System modifications describes the most common tasks relatedto the reconfiguration of the WS network.

Note:

The chapters include administration procedures for servers both with andwithout the HP MC/ServiceGuard software. If the procedures are different

About this manual


NTC TAN 0513/3 en9

depending on whether you have or do not have MC/ServiceGuard, the procedureheadings clearly show if you should carry out the procedure or not.

Chapter 7,UNIX administration also includes a special section 7.14,Tasks on theNokia NMS servers with no MC/ServiceGuard software.

2.1 What you need to know first

This document assumes that you are familiar with the following areas:

• The concepts of the Nokia NMS and GSM/PCS/DCS networks in general

• A text processing utility, such as vi. This is required for you to be able toedit certain configuration files.

• The system administration tasks of the UNIX operating system. Anoverview of the responsibilities of the system administrator can be foundin the HP-UX System Administration Concepts.

• This manuald complements information contained in the following NokiaNMS/2000 documents, and you should also familiarise yourself with theinformation contained in these documents.

• Database Maintenance, TAN 0514

• Technical Reference Guide, TAN 0453

• Expanding and Reconfiguring Networks, TAN 0432

2.2 How to use this manual

This manual can serve both as a guide to the system maintenance concepts andas a quick task reference. To conveniently locate the information you need,follow these guidelines:

What You NeedWhat

What to Look for in theManual

Examples

Informationabout the basicconcepts,descriptions andfeatures available.

About... 6.1,About backups

Descriptions ofindividual tasksand relatedwindows anddialogs.

...ing 4.4.2,Blocking BTSalarms in the BSC

About this manual


NTC TAN 0513/3 en10

Table 1.How to Use This Manual

Note that the fastest way to get started is to have a look at the task reference inChapter 3,Quick reference to administrative tasks.

2.3 Where to find more

When you perform the tasks described in this document, you may need to referto other documentation for further information. The document list belowcontains manuals that you will find useful as well as a brief description of theircontents.

NMS manuals

• For information on system management concepts, principles and systemmanagement applications for the Nokia NMS refer toSystem ManagementBasic Operating Principles and Procedures, TAN 0732.

• For information on integrating the Nokia BSS with the Nokia NMS,installing and configuring hardware, creating X.25 connections andconfiguring I/O systems, please refer toDCN Integration, TAN 0839.

• For more information on NMS database functions, database recovery,database backups and other database administration tasks that the systemadministrator should perform, seeDatabase Maintenance, TAN 0514.

• For information on hardware, NMS file and directory structure, NMSapplications and processes, configuration files and optional features, seeTechnical Reference Guide, TAN 0453.

• For further information on other NMS manuals, seeNMS/2000Description of Documentation, TAN 442.

Oracle manuals

• For conceptual information on the information contained in other Oraclemanuals, refer toOracle7 Server Concepts Manual.

• For more information on managing the an Oracle 7 server, refer to theOracle7 Server Administrator’s Guide.

• For more information on the messages generated by the Oracle 7 server,the SQL processor, PL/SQL Server Manager, Export and Import utilitiesrefer toOracle7 Server Messages.

Quick hands-ontask reference.

How to...or To... To take a backup ofthe critical files

What You NeedWhat

What to Look for in theManual

Examples

About this manual


NTC TAN 0513/3 en11

• For more details on how to use Oracle 7 server utilities for data transfer,maintenance and database administration, see theOracle7 Server UtilitiesUser’s Guide

• For more information on using SQL*Net V2.1, seeUnderstandingSQL*Net

Hewlett-Packard documents

• For more information on the available on HP-UX manuals refer toHP-UX10.0 Documentation Mapand the HP-UX manual pagemanuals (5) .

• For further information on NFS administration, please refer to theInstalling and Administering NFS Services manual.

• For further information on managing MC/ServiceGuard, please refer to theManaging MC/ServiceGuard manual.

• For further information on HP-UX system administration tasks and solvingHP-UX-related problems, please refer to theHP-UX SystemAdministration Tasks manual.

2.4 Typographic conventions

Several types of typographic conventions are used in this manual to describedifferent actions and restrictions. The tables below present the conventions usedin this manual.

2.4.1 Text styles

The following table presents the typefaces and their indications.

Text Style Explanation

Initial Upper-CaseLettering

• Application names

• Hardware components

• Names of windows and dialogs

Italicised text • Emphasis

• State, status or mode

Initial Upper-CaseLetters in Italics

• Referenced documents

• Referenced chapters within a document

Initial Upper-CaseLettering in Bold

• All components on a window or dialog in the userinterface, for example, "ChooseExit from theFile menu".

About this manual


NTC TAN 0513/3 en12

Table 2.Text Styles in This Document

2.4.2 Command line conventions

Command line prompts

The following UNIX command line prompts are used to indicate which usershould enter the command(s). Note that you should not include the prompt in thecommand.

Table 3.Command Line Prompts

Environment variables

This document assumes that the following environment variables have the valuesindicated below:

Table 4.Environment Variables

The Courier font • User names

• File and directory names

• Names of database tables

• Counters in database tables

• Parameters

• System output

UPPER-CASELETTERING

• Keys on the keyboard.

<bracketed text> • The text within the brackets presents informationabout a parameter. Replace the whole expressionwith the correct value.

Shading • Further information about command-line

Prompt As Which User

# As the root user

username>% As the user indicated byusername.

% As any user.

Variable Value

$OMCCONFPATH /global/NokiaOMC/conf/global

$OMCROOT /usr/local/NokiaOMC/<omc_build>

Text Style Explanation

About this manual


NTC TAN 0513/3 en13

Line breaks

For layout purposes, long command lines may sometimes be split onto two ormore separate lines. This is indicated as follows:

With UNIX commands and UNIX file descriptions, a backslash (\ ) at the endof a printed line is used to indicate that the command continues on the followingline. For example:

# cat /etc/cmcluster/pkgcomm/control.sh.log | grep \"crontab" | more

With MML commands, everything between the initialZ and the semicolon (; )constitute a single line. For example:

ZWTP:OMU:AS7_U,<PIU-index>,<track>::X25,<PCM-type>,<PCM-number>,TSL,0;

Optional parameters

With all types of commands, square brackets ([ ] ) are used to indicate that theportion of the command enclosed in brackets is optional or not always required.For example:

ZUSU:<unit>[,<index>];

The above example indicates that if you are dealing with several units of thesame type (for example, the BDCUs), it is necessary to also specify the index.With single units (for example, the OMU), the index parameter is not necessary.

Conditional parameters

With all types of commands, the braces and the vertical bar ({ | } ) are usedto indicate that the portion of the command enclosed in brackets is conditional.For example:

# cmmodpkg { -e | -d } <package>

The above example indicates that you must select between the-e and-d optionswhen you enter the command.

Text variables

The following mark-up for string variables is used throughout this document torepresent certain system concepts. When you encounter one of these variables,replace the string within the brackets (including the brackets) with the actual textrelevant in each context.

About this manual


NTC TAN 0513/3 en14

Table 5.Text variables

2.5 Concepts and terminology

This subsection explains the concepts, terms and abbreviations used in thisdocument.

Variable Explanation

<omc_build> This variable refers to the name of the release-specific directory of the Nokia NMS. For the actualname, refer to Chapter Release Information in theNMS Customer Release Notes, which can be foundin the Nokia NMS release binders.

<Communications> This variable refers to the name of theCommunications Server.

<Database> This variable refers to the name of the DatabaseServer.

<Standby> This variable refers to the name of the StandbyServer.

<App_X> This variable refers to the name of the ApplicationServer N. Note that there is often more than oneApplication Server, so the commands presented inthese contexts must be executed on all ApplicationServers.

<domain_name> This variable refers to the domain name of NetworkInformation Service.

<server> This variable refers to the name of a server.

Term Explanation

Application Server A server which has the Nokia NMS softwareinstalled locally. This server runs the graphicalNMS applications.

Comm&DB Server A combined Communications and DatabaseServer which runs the Nokia NMS database andalso functions as a communications gatewaybetween the workstation network and othernetwork elements.

Communications Server A server that functions as a communicationsgateway between the workstation network andother network elements.

Database Server A server that runs the Nokia NMS database.

About this manual


NTC TAN 0513/3 en15

HP 9000 server Abbreviation used to refer to a HP 9000 G, Hand K-Class servers. The HP 9000 servers usedwith the Nokia NMS have the following roles:

• Communications Server

• Database Server

• Standby Server

Depending on the configuration, a single HP9000 server may perform only a part of one ofthese functions or even several of thesefunctions.

HP 9000 workstation Abbreviation used to refer to a HP 9000 Series700 workstation. The HP 9000 Series 700workstations used with the Nokia NMS havethe following role:

• Application Server

NFS Network File System. The NFS allows filesystems to be accessed from different servers.

NFS client A server which can mount a file system fromanother server.

NFS server A server whose disks can be mounted fromother servers.

NIS Network Information Service. The NISprovides generic database access facilities forthe distribution of information, such aspassword and group files, within a network.

NIS client A server which receives NIS information fromone of the NIS servers.

NIS master A server which contains the main NIS maps anddistributes information about users and groupsto the NIS network.

NIS slave A server that takes over the tasks of the NISmaster if the NIS master itself is not functional.

Server configuration Describes the type of network connection, harddisk configuration and the number and type ofHP 9000 servers. For more information on eachof these configurations, see theTechnicalReference Guide.

Standby Server A server which monitors the Communicationsand Database Servers or Comm&DB Serverand replaces it in case of server failure. StandbyServer has its own connections to the networkelements.

X terminal A graphical terminal which can displaygraphical applications that run on anApplication Server.

Term Explanation

About this manual


NTC TAN 0513/3 en16

Table 6.Terminology used in this document

2.5.1 Abbreviations

The following table presents the abbreviations and acronyms which appear inthis document:

Table 7.Abbreviations used in this document

X.25 An interface between the DTE and the DCE ina packet switched network using X.21 at thephysical layer.

Abbreviation Explanation

AS Server role: Application Server

BSC Base Station Controller

BTS Base Transceiver Station

CDS Server role: Comm&DB Server

CS Server role: Communications Server

DAT Digital Audio Tape

DB Database

DS Server role: Database Server

FE Front End

HP Hewlett-Packard

IP Internet Protocol

IS Intermediate System

MML Man-Machine Language

NFS Network File System

NIS Network Information Service

PID Process ID

RNW Radio Network

SAM System Administrator Manager

SS Server role: Standby Server

SW Software

CDE Common Desktop Environment

WS Workstation

Term Explanation

Quick reference to administrative tasks


NTC TAN 0513/3 en17

3 QUICK REFERENCE TO ADMINISTRATIVETASKS

Workstation network maintenance is the general concept used to refer to all theadministrative tasks you have to perform to ensure that the Nokia NMS remainsfunctional at all times. It includes both preventive and corrective procedureswhich together minimise the possibility of data loss. The aim is to guarantee thatthe NMS workstations have correct and up-to-date information about the cellularnetwork, and that the information can be exploited to the fullest.

The following section,Maintenance tasks, gives you a brief overview of thetasks involved in keeping the Nokia NMS functional and efficient. We suggestthat you familiarise yourself with this section to gain a general idea of the tasksassociated with workstation network maintenance. For detailed informationabout the actual tasks, refer to the appropriate chapters later in this manual. Theydescribe each maintenance area in more detail and provide you with step-by-stepinstructions on how to use the Nokia NMS maintenance applications.

3.1 Maintenance tasks

This section describes some of the procedures involved in keeping the NokiaNMS system functional and efficient. The maintenance procedures can bedivided into two categories:

• Periodic tasks: tasks that have to be carried out at regular intervals.

• Occasional tasks: tasks that have to be carried out when needed.

We advise you to review the tables below to obtain an overview of workstationnetwork maintenance procedures.

3.1.1 Periodic tasks

This section lists the tasks that have to be regularly carried out by the systemadministrator or automatically by the system.

What to do? When to do it? Where to find instructions?

Check the reasonsfor automaticprocess restarts

Daily 7.10,Checking the processesmanually

Check the statusof the MC/ServiceGuardcluster

Daily or if there areproblems

7.5,Administering MC/ServiceGuard

Free disk spacetaken by log files

Weekly. 7.7.3,Monitoring log files



NTC TAN 0513/3 en18

Table 8.Periodic tasks

3.1.2 Occasional tasks

This section lists the tasks that have to be carried out by the system administratorwhen needed. The frequency of the tasks to be carried out below may varygreatly.

Take a backup ofthe critical files

The frequencydepends on the usage

6.3,Backing up the critical files

Take a full backupof the file systems

Every time you makechanges to the NokiaNMS system

6.2,Taking a full backup


Monitor theavailable disk space

If there are problems 7.7.2,Monitoring the diskspace

Block and filteralarms

If a network elementis excessivelygenerating alarms.

4.4,Blocking alarms

Change the IPaddress

If you shift thesystem is to anotheraddress range.

8.3,Changing the IP addressof a server

Change the NFSconfiguration

If you add or removehosts.

8.6,Exporting directories

Change the NISconfiguration

If you add or removehosts to/from the NISdomain.

8.5,Changing the NISconfiguration

Check revisions When needed 7.12,Checking revisions

Compare revisions When needed 7.12.2,Comparing revisions

Configure disksupervision

Default settings areset during thecommissioning.Reconfigure whenneeded.

7.7.1,Configuring disksupervision

Configure alarmlifting rules

When needed 4.1,Modifying alarm liftingrules

Configure alarmpipes

When addingnetwork elementsthat use new alarminterfaces.

4.2,Adding and deleting alarmpipes

Configure a printerfor Alarm Viewer

When needed 8.1,Setting up a printer inAlarm Viewer




NTC TAN 0513/3 en19

Configure the TFTPserver

If TFTP serverneeded in filetransfer

8.4,Configuring the TFTPserver

Configure automaticprocess supervision

Default settings areset during thecommissioning.Reconfigure whenneeded.

7.6.1,Configuring ProcessSupervision and RecoveryService

Create revision files When needed 7.12.1,Creating revision files

Halt an MC/ServiceGuardpackage, node orcluster

When shutting downservers or taking afull backup

7.5,Administering MC/ServiceGuard

Installing newsounds for AlarmViewer

When needed 8.2,Installing customisedsounds for Alarm Viewer

Restrict access toworkstations

If additional securityrequired.

8.7,Restricting access tocertain services on the localhost

Restore a criticalfiles backup

In case of anemergency or for testpurposes.

6.5,Restoring a critical filesbackup

Restore a file systembackup

In case of anemergency or for testpurposes.

6.4,Restoring a full backup

Shut down and startup HP 9000workstations

When needed 7.2,Shutting down andstarting up the system

Shut down and startup the HP 9000servers

When needed 7.2,Shutting down andstarting up the system

Shut down the NMS Only if instructed byNMS documentationto do so.

7.4,Shutting down andstarting up the Nokia NMS

Start up the NMS When needed. 7.4,Shutting down andstarting up the Nokia NMS

Verify that anetwork elementmonitored is online

When needed 4.3.1,Verifying that themonitored network element isonline

Verify thatmeasurementinformation iscorrect

When needed 5.1,Monitoring the networkperformance data




NTC TAN 0513/3 en20

Table 9.Occasional tasks

Each subsection describes an individual maintenance area and gives youreferences to the more detailed sections in this manual.

Verify thatmeasurements arecoming

When needed 5.1,Monitoring the networkperformance data

Verify that the NMSreflects the true stateof the network

When needed 4.3.2,Verifying that the NMSis showing the true state of thenetwork


Fault Management tasks


NTC TAN 0513/3 en21

4 FAULT MANAGEMENT TASKS

This chapter describes Fault Management tasks related to the NMS and thenetwork elements controlled by it. The following, topics are covered:

• Modifying alarm lifting rules

• Adding and deleting alarm pipes

• Checking that network monitoring works

• Blocking alarms

4.1 Modifying alarm lifting rules

The alarm lifting rules are defined in the configuration filearcobjmodelmx.cfin $OMCCONFPATH on the Communications Server.

The lifting rules have the following syntax:

# ( LEGAL_ALARM_LIFTS ""

# ( <source object class> "<target object class>" )

# )

( LEGAL_ALARM_LIFTS ""

( 24 "4" ) # TRX alarms are lifted to BTS

( 10 "108" )# alarms from FU are lifted to HLR

)

Example 1.Lifting rules in thearcobjmodelmx.cf file

To change alarm lifting rules

Caution:

Usually there is no need to modify the default alarm lifting rules. If, however,you wish to change the rules, we recommend that you contact your local Nokiarepresentative first.

1. Log into the Communications Server as theomc user.

2. Change to the$OMCCONFPATH directory and make a backup copy of thearcobjmodelmx.cf configuration file:

omc>% cp arcobjmodelmx.cf arcobjmodelmx.cf.bak

3. Open thearcobjmodelmx.cf configuration file in that directory. Use atext editor to change the contents of the file.



NTC TAN 0513/3 en22

4. In thearcobjmodelmx.cf configuration file, look for the section titledLEGAL_ALARM_LIFTS. As you can see in example above there is one linefor each alarm lifting rule in the configuration file.

5. Change the lifting rules as follows:

• If the source object class is already included in the file and has alifting rule assigned to it, and you want to change those objects’target object class, you need to change the target object class entry inthe respective line. For example, if there is a lifting rule for allChannel (5) alarms to be lifted to the Transceiver (24), and you wantto lift the Channel alarms to the BTS (4), you need to change the line

( 5 "24" )

to

( 5 "4" )

• If the source object class is not included in the file, you need to adda new row to theLEGAL_ALARM_LIFTS section, where you definethe source object class and the target object class as shown inexample on the previous page.

Note that the target object class must be a superior within the distinguishedname of the source object class. Note also that the alarm lifting rules areread hierarchically and use the first match.

6. Save the changes and close the text editor.

7. Find out the process IDs of thearcpcsmx processes and kill thearcpcsmx processes. Give the following commands:

omc>% mx | grep arcpcsmxomc>% kill -INT <pid> <pid> ...

Process Supervision and Recovery Service (wpmanamx) will restart thealarm processor processes automatically.



NTC TAN 0513/3 en23

4.2 Adding and deleting alarm pipes

Alarm pipes are defined in thearcpipmx.cf configuration file on the serverwhich runs the alarm pipe processes. An example of thearcpipmx.cf file isgiven below:

#

# Alarm pipe for WS alarms.

#

(ALARM_PIPE "0 <Comm>"

(COLLECTOR "arccolmx \

-forward -pipeId 0 \

-interface WS \

-writeBufferId ARCCOL.0")

(PROCESSOR "arcpcsmx \

-pipeId 0 \

-readBufferId ARCCOL.0 \

-writeBufferId ARCPCS.0")

(FORWARDER "arcfrwmx \

-pipeId 0 \

-readBufferId ARCPCS.0")

)

#

# Alarm pipe for Q3 alarms.

#

(ALARM_PIPE "1 <Comm>"

(COLLECTOR "arcq3cmx \

-forward \

-pipeId 1 \

-writeBufferId ARCCOL.1")


-pipeId 1 \

-readBufferId ARCCOL.1 \

-writeBufferId ARCPCS.1")


-pipeId 1

-readBufferId ARCPCS.1")

)

<other alarm pipes...>

Example 2.Example of the configuration file for alarm pipes.

Note:

The internal alarm pipes with pipe identifier 0 should not be modified in anyway.



NTC TAN 0513/3 en24

Because of MC/ServiceGuard, thearcpipmx.cf configuration file is stored inthree places in the NMS. The following table tells you where the files are located:

Every time thearcpipmx.cf file is changed, the changes have to be made toall the three files.

Table 10.Alarm pipe locations in the NMS

When a network element that uses a different type of alarm interface from theexisting one(s) is added under the supervision of the NMS, you need to install anew alarm pipe to ensure that the alarms generated in the network element areforwarded to the right target.

To add new alarm pipes

Warning:

You should only add a new alarm pipe if there does not exist a pipe for theparticular alarm interface.

The Nokia NMS supports alarm collection from several types of third-partynetwork elements. For more information on these, refer toFault ManagementBasic Operating Principles and Procedures, TAN 0704.

We also recommend that you contact your local Nokia representative before youstart modifying the alarm pipe configuration.

1. Create the buffer for the alarm pipe. Each alarm pipe has to have buffersbetween its processes.

omc>% tubeezmx create <buffername> 65536 omc sysop 700

2. Change to the$OMCCONFPATH directory and make a backup copy of thearcpipmx.cf configuration file:

omc>% cp arcpipmx.cf arcpipmx.cf.bak

Directory Server Name

$OMCCONFPATH Communications Server

$OMCCONFPATH/cs Role-specific directory (CommunicationsServer)

$OMCCONFPATH/cds Role-specific directory (Comm&DB Server)



NTC TAN 0513/3 en25

3. Update the contents of the$OMCCONFPATH/arcpipmx.cf configurationfile. Add the following section to the file:

(ALARM_PIPE "<no.> <Comm>"

(COLLECTOR "<collector_process> \

-forward -pipeId <no.> \

-interface WS \

-writeBufferId ARCCOL.<no.>")


-pipeId <no.> \

-readBufferId ARCCOL.<no.> \

-writeBufferId ARCPCS.<no.>")


-pipeId <no.> \

-readBufferId ARCPCS.<no.>")

)

Remember to take backup copies and update the files$OMCCONFPATH/cs/arcpipmx.cf and$OMCCONFPATH/cds/arcpipmx.cf , too.

4. To start the processes of the new alarm pipe, executearcstamx on allworkstations where the new alarm pipe should run.

To delete an alarm pipe

Warning:

You should only delete an alarm pipe if there are no more network elements thatuse the particular alarm interface.

The Nokia NMS supports alarm collection from several types of third-partynetwork elements. For more information on these, refer toFault ManagementBasic Operating Principles and Procedures, TAN 0704.

We also recommend that you contact your local Nokia representative before youstart modifying the alarm pipe configuration.

To remove an alarm pipe, carry out the following steps on the CommunicationsServer:

<no.> ID for the alarm pipe

<collector_process> Name of the alarm collector process,which can be:

arcq3cmx

arccolmx



NTC TAN 0513/3 en26

1. Log in as theomc user and then find out which processes belong to thealarm pipe in question by entering the following command:

omc>% ps -fp 99999 ; ps -ef | grep ’pipeId <no.>’

This command lists the processes which belong to the alarm pipe inquestion. The process ID is the number given in thePID column.

2. Kill the processes by entering

omc>% kill -TERM <pid> <pid> ...

3. Delete the buffers that belong to the alarm pipe you want to remove byentering the following commands for each buffer:

omc>% tubeezmx delete ARCCOL.<no.>omc>% tubeezmx delete ARCPCS.<no.>

4. Change to the$OMCCONFPATH directory and make a backup copy of thearcpipmx.cf configuration file:

omc>% cp arcpipmx.cf arcpipmx.cf.bak

5. Open the file$OMCCONFPATH/arcpipmx.cf in a file editor and deletethe section that defines the alarm pipe in question. For more informationon how the entries in this configuration file are structured, see theexamples on the previous pages.

Remember to take backup copies and remove the section in question alsoin the files$OMCCONFPATH/cs/arcpipmx.cf and $OMCCONFPATH/cds/arcpipmx.cf .

4.3 Checking that network monitoring works

From time to time, you should make sure that the alarm status in the NMS and anetwork element is up-to-date. The alarm status in the NMS and network elementmay be out of sync if, for example, the O&M connection between the NMS andnetwork element has been down, or if you have made any modifications to theNMS or network elements.

4.3.1 Verifying that the monitored network element is online

To verify that you are receiving alarms from the network elements to the NMS,open the Alarm Viewer application. You should see new alarms or warningsappear in the Alarm Viewer every now and then. You can check the time whenthe alarm has been generated to verify that there has not been a backlog of alarmswhich have been buffered somewhere for a long time.

If your network comprises only few network elements and practically nothingseems to be happening in your network (you see no alarms or warnings beinggenerated in your network), you may want to generate an alarm yourself in anetwork element to make sure that everything is functioning properly.



NTC TAN 0513/3 en27

Generating alarms is also a way to verify the status of a particular networkelement.

To generate an alarm, you can change the Spare Message Bus toTEST state fora short while, which causes an Incorrect Working State alarm. Open an MMLsession to the network element and first find out the index of the Spare MessageBus by entering

ZUSI:MB;

To change the Spare Message Bus toTEST state and back toSPARE state, givethe following commands:

ZUSC:MB,<index>:TE;ZUSC:MB,<index>:SP;

Some network elements also support Alarm Flow Supervision, whichautomatically generates an alarm if a network element is not online. For moreinformation, see 7.9,Modifying the Alarm Flow Supervision parameters in thismanual andSystem Management Basic Operating Principles and Procedures,TAN 0732.

4.3.2 Verifying that the NMS is showing the true state of thenetwork

You can verify that the NMS is showing the true state of the network eitherdirectly from the Top-level User Interface or by opening an MML session.

To check the current alarm status of one network element as seen by theNMS

1. Open the Alarm History application by selectingAlarm History →Search Criteria from the pop-up menu.

2. Select bothacknowledged and non-acknowledged alarms from theAlarmAck State pane.

3. Click theOther Criteria button, select theOrder by Time option andclick theSearch button

To check the alarm status in the network element you can open an MML sessionto the network element. Display active alarms with the MML command asfollows:

ZAHO; (use ZEOL; if you intend to check BTS alarms)

<index> The index of the Spare Message Bus



NTC TAN 0513/3 en28

The differences between the alarms in the NMS and network elements could alsobe caused by alarm reclassification in the NMS, or alarm blocking in the networkelement. If there is a severe mismatch between the NMS and network elementalarms, you should correct the situation with the Alarm Upload application.

4.4 Blocking alarms

If you are carrying out some maintenance work in a network element, it isadvisable to block all the alarms coming from that network element in order toprevent excessive number of alarms from arriving to the database. The alarmblock can be set using an MML session to the network element.

4.4.1 Blocking alarms in a specific network element

The following table presents the MML commands which can be used to block athe alarms in a network element.

Table 11.Blocking Alarms in a Network Element

4.4.2 Blocking BTS alarms in the BSC

It is possible to block the alarms coming from the BTSs in the parent BSC. Thiswill prevent the alarms from arriving to the NMS. The BTS alarm block can beset using an MML session to the parent BSC.

The following table presents the MML commands which can be used to block athe BTS alarms in a BSC.

Table 12.Blocking BTS Alarms in a BSC

MML Command Description

ZABB:<alarm number>:,; Blocks the alarms of a specific number

ZABU:<alarm number>:,; Unblocks the alarms of a specific number

ZABO; Lists all blocked alarms in a networkelement.

MML Command Description

ZEOB:<alarm number>; Blocks the BTS alarms of a specific numberin the BSC

ZEOU:<alarm number>; Unblocks the BTS alarms of a specificnumber in the BSC

ZEOE; Lists the blocked BTS alarms in the BSC



NTC TAN 0513/3 en29

4.4.3 Disabling and enabling network alarms andmeasurements in the workstations

Network measurements and alarms should always be disabled before thedatabase is shut down. When the alarms and measurements are disabled, theelements start buffering events to avoid data loss. The alarms and measurementsmust also be enabled after the database startup when buffering is no longernecessary and forwarding the alarms and measurements to the database can beresumed.

To disable measurements and alarms

1. Log into the Communications Server as theomc user.

2. Enter the following command to find out the process ID of theedmmanmxprocess:

omc>% mx | grep edmman

3. Enter the following command to disable network alarms andmeasurements:

omc>% kill -USR1 <edmmanmx_PID>

When edmmanmx process receives theUSR1 signal, it ceases to forward networkalarms and measurements to the database. However, the processmust stay up tobe able to resume event forwarding.

To enable measurements and alarms

After the database startup the alarms and measurements have to enabled.

1. Log into the Communications Server as the omc user.

2. Enter the following command to find out the process ID of theedmmanmxprocess:

omc>% mx | grep edmman

3. Enter the following command to enable network alarms andmeasurements:

omc>% kill -USR1 <edmmanmx_PID>

When edmmanmx process receives theUSR1 signal, it resumes forwardingnetwork alarms and measurements to the database. Foreddmanmx to be able toresume event forwarding, itmust not be killed.

<edmmanmx_PID> Process ID of theedmmanmx process

<edmmanmx_PID> Process ID of theedmmanmx process

Performance Management tasks


NTC TAN 0513/3 en30

5 PERFORMANCE MANAGEMENT TASKS

This chapter describes performance management tasks related to the NMS andthe network elements controlled by it. The following, topics are covered:

• Monitoring the network performance data

5.1 Monitoring the network performance data

If the O&M connection between the NMS and a network element has been downor if you have made some modifications in the NMS or network elements, youshould check that network performance data is available in the NMS.

To verify that measurements from the BSC are transferred to the NMS

1. SelectPerformance Mgmt → DB Contents from the BSC pop-up menu.

2. Click theSelect All button to select all the available measurement types

3. Click theFrom button in theTime pane and set the date to current date andthe time to the beginning of the day, for example. Keeping the periodrelatively short speeds up the verification process. However, you shouldnot set the time to shorter than your measurement interval or you will notbe able to see any measurements coming in.

4. Click File → Find.

5. If you do not see any measurements coming, you should also check themeasurement schedule, output interval and the state of the measurement.Some observation reports are generated only when the requirements for theobservation are met.

Generic Supervision Service also monitors the transfer of measurementsfrom BSCs. For more information, refer to 7.8.4,Supervising the flow ofBSC traffic measurements in this manual.

To verify that BSC measurements are not buffered in the BSC

1. Click with the mouse button on the right on top of the BSC and selectMML Session

2. Give the following command:

ZIFO:OMU,MEASUR:1&&300;

Performance Management tasks


NTC TAN 0513/3 en31

You will see similar output to the one below:

298 TRANSFERRED OK/-- - - - 1998-01-18 08:01:00 1998-01-18 08:01:09

299 TRANSFERRED OK/-- - - - 1998-01-18 08:15:59 1998-01-18 08:16:07

300 TRANSFERRED OK/-- - - - 1998-01-18 08:30:59 1998-01-18 08:31:09

Example 3.Output fromZIFO:OMU,MEASUR:1&&300; command

If measurements are not buffered, the buffer is empty and there are no filesin FULL state.

To verify that measurement information is correct in the NMS

1. Click with the mouse button on the right on top of the network element youwant to check and selectPerformance Mgmt→ Administration from thepop-up menu.

2. Select a measurement by clicking it.

3. SelectAction → Interrogate. The application informs you, if there areany discrepancies between the network element and NMS concerning thismeasurement or observation type.

4. Repeat steps 2 and 3 for all measurement types on the list.

Taking backups of the Nokia NMS


NTC TAN 0513/3 en32

6 TAKING BACKUPS OF THE NOKIA NMS

This chapter describes the procedures associated with taking and restoringbackups of the files in the Nokia NMS system. This procedure is divided into thefollowing parts:

• About backups

• Taking a full backup

• Backing up the critical files

• Restoring a full backup

• Restoring a critical files backup

For more information on backup principles, refer toSystem Management BasicOperating Principles and Procedures, TAN 0732.

This chapter also contains instructions on how to take backups of the files in theNokia NMS Front End.



NTC TAN 0513/3 en33

6.1 About backups

It is highly recommended that you take regular backups of the system to avoiddata loss in case of a system failure. The backup frequency depends on thesystem usage.

The following procedures explain both the full backup and the backup of themost critical files in the system. The subsections give instructions on how to takethe backups both with and without the MC/ServiceGuard software.

6.2 Taking a full backup

A full backup saves a copy of all the file systems in the workstation network ontoDATs, from where it can be restored in case something goes wrong.

To take a full backup, all applications which are currently running on the server,need to be stopped. In the Nokia NMS environment, this means shutting downthe database and all the Nokia NMS processes. Only after this has been done canthe files be copied onto the backup medium.

The backup procedure presented here takes separate copies of the root filesystem and of the disks mounted under the/d directory hierarchy (/d is not amount point!). Please note that one DAT may not be enough to store all the filesin the system.

To take a full backup with MC/ServiceGuard in use

1. On the Communications Server, halt the MC/ServiceGuard packages:

# cmhaltpkg -v db# cmhaltpkg -v comm

Halting the packages may take several minutes.

Halting thecomm package on the Communications Server also halts theomcdb package on the Database Server and thespare package on theStandby Server.

2. Insert a DAT into the tape drive on the Communications Server and copythe root file system onto the tape. If you have a compressing DAT drive,give the following commands on the Communications Server:

# cd /# tar cvf /dev/dat ./

If you do not have a compressing DAT drive, you can speed up the backupprocess considerably by compressing the files while moving them onto theDAT:

# tar cvf - ./ | compress > /dev/dat



NTC TAN 0513/3 en34

3. To access the other disks, you have to remount them. Give the followingcommands:

# /etc/cmcluster/sgsdskmx.sh -e mount db# /etc/cmcluster/sgsdskmx.sh -e mount comm

4. Copy all the files from/d onto the tape, or copy different file systems ontodifferent tapes:

• If you have a compressing DAT drive, give the following commandson the Communications Server to copy all the files onto the sameDAT:

# cd /# tar cvf /dev/dat ./d

If you do not have a compressing DAT drive, you can speed up thebackup process considerably by compressing the files while movingthem onto the DAT:

# tar cvf - ./d | compress > /dev/dat

Note:

When the tape becomes full, the system asks you to insert anothertape.

• As an alternative, you can copy the file systems mounted under/d/global and/d/dbXX each onto separate tapes. This may help youto keep track what files are stored where and make restoring the fileseasier. For example, to copy the files under/d/global onto aseparate DAT, insert a new tape into the DAT drive on theCommunications Server and do the following:

# cd /d# tar cvf /dev/dat ./global

Remove the tape when all files from/d/global have been copiedand insert another tape for another file system.

5. Unmount the disks before starting up the Nokia NMS and Nokia NMSdatabase:

# /etc/cmcluster/sgsdskmx.sh -e umount db# /etc/cmcluster/sgsdskmx.sh -e umount comm

6. Enable package switching for the packages:

# cmmodpkg -e db comm spare omcdb

You can view the status of the MC/ServiceGuard cluster by giving thecommandcmviewcl and verify that the packages are starting as they



NTC TAN 0513/3 en35

should. The startup usually takes a while, so you may have to repeat thecommand a few times.

If the MC/ServiceGuard packages do not start up after you have enabledthe package switching attributes, you have to restart the packages manuallyby giving the following command:

# cmrunpkg -v -n <server> <package>

You have to specify the server if the primary node for the package isdifferent from the server on which you are giving the command.

To take a full backup with MC/ServiceGuard not in use

Note:

You should follow the instructions in this section if your system does not havethe MC/ServiceGuard software installed. We assume that you have aComm&DB Server only.

1. Log into the Comm&DB Server as theroot user.

2. Close the connections to the network elements:

• If you use X.25 connections, give the following commands on theComm&DB Server:

# x25stop -d /dev/x25_0# exit

Wait a while for the processes to flush the channels.

Note:

If you have several X.25 cards, you must repeat thex25stopcommand for each X.25 card, replacing the device specification asnecessary, for example:

# x25stop -d /dev/x25_1

• If you use LAN connections kill the OTS processes. Give thefollowing commands:

# ps -ef | grep ots

Make a note of the Process IDs (PIDs) of theotsamd andotslogdprocesses. Kill the processes:

# kill -TERM <otsamd_pid> <otslogd_pid>



NTC TAN 0513/3 en36

3. Shut down the Nokia NMS background processes on all servers:

% su - omcomc>% mx | grep wpmana

Make a note of the PID of thewpmanamx process and kill the process:

omc>% kill -TERM <wpmanamx_pid>

4. Stop thecron daemon by giving the following command on theComm&DB Server as theroot user:

# /sbin/init.d/cron stop

5. Shut down the database:

# su - oracleoracle>% svrmgrlSVRMGR> connect internalSVRMGR> shutdown immediate

If shutting down the database seems to take an excessive amount of time,the RDBMS may be performing a rollback or another time-consumingoperation. You can view the activities the RDBMS carries out bymonitoring thealert_omc.log file. Enter the following command:

oracle>% tail -f $ORACLE_HOME/rdbms/log/alert_omc.log

If the RDBMS is carrying out a time-consuming task, simply wait for it tocomplete.

When the database has been shut down using the immediate option, start itup and shut it down again:

SVRMGR> startup openSVRMGR> shutdown normalSVRMGR> exit

You should receive a notification that the database has been shut down.Then exit theoracle prompt by entering:

oracle>% exit

6. Unmount the file systems in/d/global and/d/dbXX :

# cd /# umount /d/global /d/db01 /d/db02 [ /d/dbXX ]

<otsamd_pid> The PID of theotsamd process.

<otslogd_pid> The PID of theotslogd process.

<wpmanamx_pid> The PID of thewpmanamx process



NTC TAN 0513/3 en37

7. Copy all the files on the root file system onto a DAT. If you have acompressing DAT drive, enter the following commands on theCommunications Server as theroot user:

# tar cvf /dev/dat ./

If you do not have a compressing DAT drive, you can speed up the backupprocess by compressing the database while moving it onto the DAT. Enterthe following commands on the Comm&DB Server as theroot user:

# tar cvf - ./ | compress > /dev/dat

The workstation backup may take several hours. You will be prompted foranother DAT if the one in the DAT drive becomes full.

8. Mount the file systems back under/d/global and/d/dbXX :

# mount -ae

9. Copy all the files from/d onto the tape, or copy different file systems ontodifferent tapes.

• If you have a compressing DAT drive, give the following commandson the Communications Server to copy all the files onto the sameDAT:

# cd /# tar cvf /dev/dat ./d

If you do not have a compressing DAT drive, you can speed up thebackup process considerably by compressing the files while movingthem onto the DAT:

# tar cvf - ./d | compress > /dev/dat

Note:

When the tape becomes full, the system asks you to insert anothertape.

• As an alternative, you can copy the file systems mounted under/d/global and/d/dbXX each onto separate tapes. This may help youto keep track what files are stored where and make restoring the fileseasier. For example, to copy the files under/d/global onto aseparate DAT, insert a new tape into the DAT drive on theCommunications Server and do the following:

# cd /d# # tar cvf /dev/dat ./global

Remove the tape when all files from/d/global have been copiedand insert another tape for another file systems.



NTC TAN 0513/3 en38

10. If you use X.25 connections proceed to step 11.

If you use LAN connections, take the following steps to restore the normalmode of operation:

• Reboot the server:

# shutdown -r -y 0

Note:

The backup procedure with LAN connections ends here. When theserver restarts it restores the network element connections and bringsthe Nokia NMS back online.

11. When the backup is finished, start thecron daemon by entering thefollowing command as theroot user:

# /sbin/init.d/cron start

12. Start up the database:

# su - oracleoracle>% svrmgrlSVRMGR> connect internalSVRMGR> startup open

You should receive a notification that the database has been started up.Then exit theoracle prompt by entering:

oracle>% exit

13. Open the X.25 connections by entering the following commands on theComm&DB Server:

# x25init -c /etc/x25/x25config_0# exit

Note:

If you have several X.25 cards, you must repeat thex25init commandfor each X.25 card, replacing the device specification as necessary, forexample:

# x25init -c /dev/x25_1

14. Restart the Nokia NMS processes on all servers:

# su - omcomc>% omkickmx -v



NTC TAN 0513/3 en39

6.3 Backing up the critical files

The most critical files in the workstation configuration are the following:

• Data files in/usr/oracle/dbs

• Database archive log files in/usr/oracle/dbs/arc

• Nokia NMS configuration files in/usr/local/NokiaOMC/conf

• Network views in/usr/local/NokiaOMC/views

• Customisation files in/usr/local/NokiaOMC/custom

• Home directories of the users in/home

The NMS software automatically copies these directories from theCommunications Server to the Standby Server every night. We advise you tocheck daily if the automatic backup operation has been successful. This can bedone by verifying the time stamps of the files on the backup disk. The timestamps should show the time when the backup was last taken. The backupprocedures are run by thecron daemon foromc, root andoracle users. If anyproblems occur, check the crontab entries of each the users with thecrontab -l command. For more information, see the 7.13,Managing crontab files.

The database files can be backed up taking advantage of the features offered bythe HP MirrorDisk/UX software. For information on how to take an offlinebackup of the database files, please refer toDatabase Maintenance, TAN 0514.

To take a backup of the critical files

1. Insert the backup tape into the DAT drive on the Communications Server.

2. Log into the Communications Server as theroot user.

3. Take a backup of the critical files in the directories by giving followingcommand:

# tar cvfh /dev/dat /usr/local/NokiaOMC/conf \ /usr/local/NokiaOMC/views \ /usr/local/NokiaOMC/custom \ /home

6.4 Restoring a full backup

This subsection gives you instructions on how to restore a full file systembackup. We assume that you have taken backups as described in 6.2,Taking afull backup.



NTC TAN 0513/3 en40

Note:

These instructions assume that the backup is restored to a system which hassuffered a disk failure or has in some other way been disabled. This means thatthe Nokia NMS is not running and that the Nokia NMS database is down.However, we assume that the HP-UX operating system is functional. We alsoassume that the disk and hardware configuration are the same as when thebackup was taken.

To restore a full backup

To restore a full backup of all the files in the workstation network of the NokiaNMS you should have plenty of free space on one of the disks. The backup isrestored to a temporary directory and the necessary files are copied from there totheir correct locations. We recommend that you use the backup disks, forexample.

1. Insert the backup DAT into the DAT drive. If you did not compress thedata files, give the following commands as theroot user to restore thebackup:

# cd <disk_with_free_space># tar xvfp /dev/dat

If you have compressed the data files when taking the backup, you shouldnow uncompress and restore them with the following command:

# uncompress < /dev/dat | tar xvfp -

Note:

If you want to restore a specific file, give the following commands:

# cd /# tar xvfp /dev/dat/ <file>

Note:

If you get an error message saying that the files you are trying to restoreare busy, you have to rename the files on the hard disk with themvcommand.

Repeat the above step for all directories where you have data files.

<file> File or directory namewith full relative path specification



NTC TAN 0513/3 en41

2. Move or copy the required files to their correct locations. Remember to usethe -p flag with thecp command to preserve permission and ownershipsettings.

3. If you overwrite any operating system files we recommend that you rebootthe servers. For instructions, refer to Chapter 7,UNIX administration.

6.5 Restoring a critical files backup

This subsection gives instructions on how to restore a critical files backup. Thebackup is assumed to have been taken as described in 7.8.2,Supervisingmirrored disks.

To restore a critical files backup

1. Insert the backup tape of the critical files into the DAT drive on theCommunications Server.


3. Restore the critical files by giving the following command:

# cd /# tar xvfp /dev/dat

Note:

If you only want to restore specific directories or files, enter the followingcommands:

# cd /# tar xvfp /dev/dat/ <file>

6.6 Taking a backup of the files in the Front End

The Nokia NMS Front End contains duplicated hard disks that hold the systemdata. Even with this precaution, it is advisable to take regular backups of thesystem files in the Front End onto a diskette or one of the hard disks in the FrontEnd. You can take a backup of the system files by:

• Disabling file updates from the PIUs

• Creating a backup build

• Enabling file updates from the PIUs

For exact instructions, see the tasks below.

<file> File or directory namewith full relative path specification



NTC TAN 0513/3 en42

To take a backup of the current build onto a hard disk

1. Disable file updates to the disk from the units:

ZDUP:OMU:DISK;ZDUP:FCMU:DISK;ZDUP:PFMU:DISK;

2. Take a backup copy of the build:

ZWQS:ALL:NAME=<backup_name>,DIRE=<backup_dir>,:;

3. If you wish to update theLFILES directory of the backup to the level ofthe active build, give the following command:

ZWQS:DATA;

4. Enable file updates to the disk:

ZDUR:OMU:DISK;ZDUR:FCMU:DISK;ZDUR:PFMU:DISK;

To take a backup of LFILES onto a floppy disk

1. Give the following commands:

ZIWY:S:SYSTEM=<FE_NAME>,UNIT=OMU,PATH=-LFILES/,DRIVE=WDU-S;ZIWY:D:SYSTEM=<FE_NAME>,UNIT=OMU,DRIVE=FDU-N0,PATH=/;ZIBC;

6.7 Restoring a Front End backup

You can either activate the backup build on the other hard disk or restore thebuild from a diskette.

To restore a Front End backup from a diskette

1. Give the following commands:

ZIWY:S:SYSTEM=<FE_NAME>,UNIT=OMU,PATH=/,DRIVE=FDU-N0;ZIWY:D:SYSTEM=<FE_NAME>,UNIT=OMU,DRIVE=WDU-S,PATH=-LFILES/;ZIBC;

2. Copy the files from theSYSTEM drive onto theBACKUP drive:

ZIWY:S:SYSTEM=<FE_NAME>,UNIT=OMU,DRIVE=WDU-S,PATH=-LFILES/;ZIWY:D:SYSTEM=<FE_NAME>,UNIT=OMU,

<backup_name> Any string up to eight characters.

<backup_dir> Any string up to eight characters.



NTC TAN 0513/3 en43

DRIVE=WDU-B,PATH=-LFILES/;ZIBC;

To activate the Front End backup

1. If you have to take the backup into use, give the following command:

ZWSD:STAT=FB;

2. Restart the Front End:

ZUSS:SYM:C=DSK,F=TOT;

6.8 Taking a backup of the router configuration

In case the router configuration is accidentally lost, it is good to have a backupof the router configuration. The task sequence below shows how to take backupcopies of running-configuration or startup-configuration files with the TFTPserver (tftpd ).

To take a backup of the router configuration


2. Create the configuration file into thetftp user's home directory. The filename must be exactly the same as the one you are going to use when youmake copy of router's configuration file, for example,router-confg . Ifthe TFTP service is not available, please refer to 8.4,Configuring theTFTP server.

# touch /usr/tftpdir/router_conf/router-confg

3. Change the file owner totftp:guest and give the user read and writepermissions.

# chown tftp:guest /usr/tftpdir/router_conf/\router-confg# chmod 600 /usr/tftpdir/router_conf/router-confg

4. Copy the configuration file to the TFTP server

router# copy running-config startup-configremote host []? <IP_address>Name of configuration file to write [router-confg]?/router_conf/router-confg

Note:

Ensure that the full path name the client is requesting from the server existswithin thetftp user’s home directory.



NTC TAN 0513/3 en44

Write file /router_conf/router-confg on host111.222.122.92?[confirm]Building configuration...Writing /router_conf/router-confg !! [OK]router# exit

5. Copy the configuration file from the TFTP server. When you have copiedthe configuration file from the TFTP server to the startup-configuration,the router has to be restarted to activate the changes.

router# copy tftp startup-configAddress of remote host [255.255.255.255]? <IP_address>Name of configuration file [router-confg]?/router_conf/router-confgConfigure using router-confg from 111.222.122.92?[confirm]Loading router-confg from 111.222.122.92 (viaEthernet0): ![OK - 1172/32723 bytes][OK]

6. Activate the new start-up configuration by restarting the router:

router# reloadProceed with reload? [confirm]

UNIX administration


NTC TAN 0513/3 en45

7 UNIX ADMINISTRATION

This section describes the most common system administration tasks related tothe HP-UX operating system and the Nokia NMS. This chapter cover thefollowing topics:

• System shutdown and startup

• Shutting down and starting up the system

• Rebooting the NMS servers and workstations

• Shutting down and starting up the Nokia NMS

• Administering MC/ServiceGuard

• Managing the Nokia NMS processes

• Resource Supervision Service

• Generic Supervision Service

• Checking the processes manually

• Monitoring the wcltoday.log File

• Checking revisions

• Managing crontab files

• Tasks on the Nokia NMS servers with no MC/ServiceGuard software

Note:

For information on tasks to be carried out on servers without the MC/ServiceGuard software can be found in Section 7.14,Tasks on the Nokia NMSservers with no MC/ServiceGuard software.

7.1 System shutdown and startup

Some operating system processes may reduce the available system resourcesover time. Therefore it may be necessary to reboot the servers and workstationsoccasionally.

This chapter gives instructions on how to halt or restart the system using theshutdown command and describes what happens when the server is brought toa halt or when it is rebooted.

The instructions given in this chapter apply to all server configurations.

UNIX administration


NTC TAN 0513/3 en46

About the shutdown command

Using theshutdown command is the correct way to reboot or halt the system.When you enter theshutdown command to halt or reboot the servers, theshutdown program shuts the system down in an orderly and cautious manner.There are a few things that should be borne in mind when you use the shutdowncommand:

• If you entershutdown without specifying any command line arguments,the system is brought down tosingle-user mode. This mode is often usedfor system maintenance tasks. Insingle-user mode you can either enter/sbin/reboot to reboot the system or/sbin/reboot -h to bring thesystem to a halt.

Note that youshould not runshutdown -r to reboot the system when youare already insingle-user mode but use the reboot command as describedabove.

• If you specify -h, the system is halted after the system is brought tosingle-user mode.

• If you specify -r, the system is rebooted immediately after it has beenbrought tosingle-user mode.

Note:

You must not enter the/sbin/reboot command directly in multi-user modeto reboot or halt the system.

In the example sequence below, we assume that the current runlevel is 3, whichis the default runlevel used by the Nokia NMS servers. Theshutdown programgoes through the following steps:

1. The PATH environment variable is reset. By default it is set to/usr/bin:/usr/sbin:/sbin . The value of the variable is modified laterwhen the process kill scripts are run.

2. TheIFS (Internal Field Separator) environment variable is set to tab, newline and space. The Bourne shell and other programs uses this variable tosplit the command line arguments passed to it into distinct argumentswhere any of the characters are found.

3. Theshutdown program checks to see if the current user is authorised toexecute theshutdown command.

4. The current working directory is set to the root directory (/ ). This is doneto ensure that file systems can be unmounted.

5. All superblocks on the mounted file systems are updated with thesynccommand to ensure the integrity of the file systems.

UNIX administration


NTC TAN 0513/3 en47

6. The real user ID is set to that of the superuser.

7. Theshutdown program sends a broadcast message to all users logged onthe system telling them to log out.

8. The program executes the/sbin/rc script to bring down any subsystems.The steps are roughly the following:

• ThePATH environment variable is set to/sbin.

• The script determines the current runlevel

• The script executes the process kill scripts in/sbin/rc<runlevel>.d directory. By running these scripts with thestop parameter all processes are killed in an orderly fashion. Whenthe server is being halted or rebooted, the scripts for the intermediatelower runlevels and the new (target) runlevel will be executed. Inother words, with the current runlevel being 3, the scripts forrunlevels 2, 1 and 0 are run.

In the Nokia NMS environment the following processes (amongothers) are stopped at this stage if they are running:

• MC/ServiceGuard cluster

• CDE login process

• Nokia NMS processes

• Oracle SQL*Net processes

• NMS database processes

• NMS time synchronisation processes

• NFS server subsystem

• System message logging daemon

• X.25 connections

• NIS subsystem

• other processes/subsystems...

9. The system is halted or rebooted with/sbin/reboot depending onwhether-h or -r was specified when theshutdown command was given.

7.2 Shutting down and starting up the system

This section describes how to shut down and start up the NMS servers(Communications Server or Comm&DB Server, Database Server and StandbyServer), as well as the NMS workstations (Application Servers).

UNIX administration


NTC TAN 0513/3 en48

To shut down the NMS servers

1. Log into all servers from the server console as theroot user.

2. Shut down the Communications Server:

# shutdown -h -y 0

3. Repeat the above command on the Standby Server and then on theDatabase Server.

4. When the servers have been halted and the system informs you that youcan switch off the server, turn off the power in the following order.

• Switch off the power on the servers.

• Switch off the power on all external disks, screens and other externaldevices.

To start up the NMS servers

1. Power on all the external devices (disks, screens, and other devices). Waituntil the external devices have warmed up.

2. Power on the Communications Server, the Standby Server and theDatabase Server.

3. Verify that all NMS processes are running. For instructions refer to 7.10,Checking the processes manually.

4. If you have MC/ServiceGuard in use, verify that the MC/ServiceGuardcluster starts up successfully:

# cmviewcl

The table below shows how the MC/ServiceGuard packages have beendefined to run on the different servers. The table sums up the supportedNokia NMS server configurations and their MC/ServiceGuard packages(primary nodes).

Table 13.MC/ServiceGuard packages on the Nokia NMS servers

To shut down an NMS workstation

1. Log into the Application Server from the console as theroot user.

Server Configuration CDS CS DB SS

CDS + SS (1 + 1) db + comm - - spare

CS + DB (2 + 0) - comm db + omcdb -

CS + DB + SS (2 + 1) - comm db + omcdb spare

UNIX administration


NTC TAN 0513/3 en49

2. Shut down the Application Server:

# shutdown -h -y 0

3. Repeat steps 1 and 2 on all other Application Servers that need to be shutdown.

4. When the workstation has been halted and the system informs you that youcan switch off the workstation, turn off the power in the following order.

• Switch off the power on the workstation.

• Switch off the power on all external disks, screens and other externaldevices.

To start up an NMS workstation

1. Power on all the external devices (disks, screens, and other devices). Waituntil the external devices have warmed up.

2. Power on the workstation.


7.3 Rebooting the NMS servers and workstations

This section describes how to reboot the NMS servers (the CommunicationsServer or Comm&DB Server, Database Server and Standby Server) andworkstations (Application Servers).

To reboot the NMS servers

1. Log into all servers from the server console as theroot user.

2. Reboot the Communications Server:

# shutdown -r -y 0

3. Repeat the above command on the Standby Server and then on theDatabase Server.


5. If you have MC/ServiceGuard in use, you should verify the status of theMC/serviceGuard cluster.

# cmviewcl

When the servers are up and running, you should also reboot the ApplicationServer(s). For more information, see below.

UNIX administration


NTC TAN 0513/3 en50

To reboot the NMS workstations

1. Log into the Application Server from the console as theroot user.

2. Reboot the Application Server:

# shutdown -r -y 0

3. Repeat Steps 1 and 2 on all other Application Servers that need to berebooted.


7.4 Shutting down and starting up the Nokia NMS

As mentioned in Chapter 7,UNIX administration, thewpmanamx process takescare of process supervision and recovery. With Nokia NMS release T10, thewpmanamx process is normally supervised by MC/ServiceGuard, which is whyyou have to shut down the Nokia NMS by halting thecomm package.

To terminate all the Nokia NMS processes

Warning:

Terminating all NMS processes will temporarily make the NMS unusable. Donot terminate the processes unless a maintenance manual or some other NMSdocumentation specifically instructs you to do so. Also note that if you manuallyterminate all the processes under supervision, they also have to be restartedmanually. See Subsect1 7.6.3,Terminating and starting a process undersupervision for details.

1. Enter the following command as theroot user on the node running thecomm package:

# cmhaltpkg -v comm

The above command will cause Process Supervision and Recovery to behalted as well as the Nokia NMS processes.

2. If you need to access the disks included in thecomm MC/ServiceGuardpackage, you have to remount the disks with the following command:

# /etc/cmcluster/sgsdskmx.sh -e mount comm

To start up the Nokia NMS

MC/ServiceGuard supervises thewpmanamx process which is responsible for themonitoring of the other NMS processes. The instructions given in this sectionpresuppose that you have stopped the Nokia NMS processes as described above.

UNIX administration


NTC TAN 0513/3 en51

1. If you have mounted the disks belonging to thecomm package while theMC/ServiceGuard daemon has been down, you have to unmount the disksbefore starting up thecomm package and the Nokia NMS processes. Tounmount the disks give the following command:

# /etc/cmcluster/sgsdskmx.sh -e umount comm

2. To start all the Nokia NMS processes, give the following command on theCommunications Server:

# cmrunpkg comm

This starts thecomm MC/ServiceGuard package and the Nokia NMSprocesses.

7.5 Administering MC/ServiceGuard

This chapter describes the basic management procedures for MC/ServiceGuardin the Nokia NMS environment.

The basic concepts are presented inSystem Management Basic OperatingPrinciples and Procedures, TAN 0732. For more information on MC/ServiceGuard, see the HP-UX man pages on thecmhaltpkg , cmrunpkg ,cmmodpkg, cmhaltcl and cmruncl commands, or refer to the HP manualManaging MC/ServiceGuard.

MC/ServiceGuard affects many of the routine maintenance tasks within theNokia NMS workstation network. If you intend to shut down the database or killthe Nokia NMS processes, you have to do it by closing down the MC/ServiceGuard services in one of the ways presented below.

7.5.1 Halting the MC/ServiceGuard packages

Many routine system administration tasks require that you halt the MC/ServiceGuard packages. If you shut down the Nokia NMS database, for example,without first stopping MC/ServiceGuard, the MC/ServiceGuard daemon will tryto restart the database as soon as it finds out that the database is down. Thereforethe procedure for shutting down the Nokia NMS processes or Nokia NMSdatabase must include the command for halting an MC/ServiceGuard package orthe whole cluster.

The general syntax for halting an MC/ServiceGuard package is given below:

# cmhaltpkg [ -n <node_name> ] [ -v ] <package>

<node_name> The name of the server

<package> The name of the package (comm, db, omcdb or spare )

UNIX administration


NTC TAN 0513/3 en52

If you do not specify the node, the package will be halted regardless of where itis currently running. If the node is specified, the package will only be halted if itis running on the specified node.

Halting the MC/ServiceGuard package also makes the disks belonging to thepackage inaccessible. After halting the MC/ServiceGuard package you canremount the disks if you need to access them.

To halt the MC/ServiceGuard packages

1. Log into the Communications Server as theroot user and give thefollowing commands:

# cmhaltpkg -v db comm

2. After halting the packages you have two options:

• You can proceed to halting the nodes or the whole cluster. For moreinformation, refer to 7.5.3,Halting an MC/ServiceGuard node or7.5.5,Halting the MC/ServiceGuard cluster.

• You can continue working on the node. To access the disks ownedby the MC/ServiceGuard cluster they have to be remounted. Give thefollowing commands:

# /etc/cmcluster/sgsdskmx.sh -e mount db# /etc/cmcluster/sgsdskmx.sh -e mount comm

For more information on mounting and unmounting disks on theNokia NMS servers running MC/ServiceGuard, refer to 7.5.9,Mounting and unmounting file systems within a MC/ServiceGuardcluster.

7.5.2 Starting the MC/ServiceGuard packages

The general syntax for starting an MC/ServiceGuard package is given below:

# cmrunpkg [ -n <node_name> ] [ -v ] <package>

Thecmrunpkg command starts the specified package within a cluster. If you donot specify the node with the-n option, the package starts running on the serverwhere you give the command. Sometimes it may be necessary to specify thenode, if one of the servers is down and the package has to be moved from oneserver to another, for example. If you have stopped an MC/ServiceGuardpackage and remounted the disks belonging to the package, restarting a MC/ServiceGuard package requires that you first unmount the disks and then restartthe MC/ServiceGuard package.

The MC/ServiceGuard packages can also be started by enabling the nodeswitching attributes on the packages. For more information on how to enable thenode switching attributes, refer to 7.5.8,Enabling MC/ServiceGuard nodeswitching.

UNIX administration


NTC TAN 0513/3 en53

To start the MC/ServiceGuard packages


If you have not mounted the disks while the packages have been down,proceed to Step 3.

2. Unmount the disks before restarting the packages:

# /etc/cmcluster/sgsdskmx.sh -e umount db# /etc/cmcluster/sgsdskmx.sh -e umount comm

3. Restart the packages by enabling the node switching attributes on thepackages:

# cmmodpkg -e db comm

The command starts the packages on their primary nodes.

4. Verify that the packages start up normally. Give the following command:

# cmviewcl

The startup takes some time. You may have to repeat the command a fewtimes to see that the packages start up normally.

5. If the MC/ServiceGuard packages do not start up after you have enabledthe package switching attributes, you have to restart the packages manuallyby giving the following command:

# cmrunpkg -v [ -n <server> ] <package>

You must specify the server if the primary node for the package is differentfrom the server on which you are giving the command.

7.5.3 Halting an MC/ServiceGuard node

Instead of halting an MC/ServiceGuard package, you can halt or restart an MC/ServiceGuard node. You can halt the MC/ServiceGuard daemon on a specificnode by giving the following command:

# cmhaltnode [ -f ] [ -v ]

The cmhaltnode command causes the node to stop its MC/ServiceGuarddaemon, optionally halting all packages in the process if you specify the-f flag.If you have not halted the packages at an earlier stage on the Communications orthe Database Server, halting the node with the-f flag will cause MC/ServiceGuard to switch the package(s) from the current node to the adoptivenode.

To halt an MC/ServiceGuard node

1. Log into the server as theroot user

UNIX administration


NTC TAN 0513/3 en54

2. Halt the node by giving the following command:

# cmhaltnode -v

Note:

If you halt one of the primary nodes (CS, DS or CDS) and force the shutdown of the packages with the-f flag, MC/ServiceGuard moves the MC/ServiceGuard package(s) to the adoptive node after halting the primarynode.

3. After halting the node you have two options:

• You can proceed to halting another node, the whole cluster orshutting down the server. For more information on halting thecluster, refer to 7.5.5,Halting the MC/ServiceGuard cluster and7.5.7,Shutting down and restarting a server after halting a node orthe whole cluster.

• You can continue working on the node. To access the disks ownedby the MC/ServiceGuard cluster they have to be remounted. Give thefollowing command:

# /etc/cmcluster/sgsdskmx.sh -e mount <package>

For more information on mounting and unmounting disks on theNokia NMS servers running MC/ServiceGuard, refer to 7.5.9,Mounting and unmounting file systems within a MC/ServiceGuardcluster.

7.5.4 Starting an MC/ServiceGuard node

The general syntax for starting an MC/ServiceGuard node is given below:

# cmrunnode [ -v ] [ <node> ]

The cmrunnode command causes the node to join itself to the MC/ServiceGuard cluster.

To start an MC/ServiceGuard node

1. Log into the server in the cluster as theroot user.

If you have not mounted the disks while the node has been down, proceedto Step 3.

2. Unmount the disks before restarting the node:

# /etc/cmcluster/sgsdskmx.sh -e umount <package>

UNIX administration


NTC TAN 0513/3 en55

Repeat the command for all the packages whose disks you have mountedwhile the node has been down.

3. Start the node by giving the following command:

# cmrunnode -v

If you have halted the package(s) for which the current server is as aprimary node, proceed to Step 4.

If there are packages that are currently running on an adoptive node, theyhave to be manually stopped on the adoptive node and moved back to theprimary node. For instructions on how to halt and start packages, see 7.5.1,Halting the MC/ServiceGuard packages and 7.5.2,Starting the MC/ServiceGuard packages.

4. Restart the package(s) by enabling the node switching attributes on thepackages:

# cmmodpkg -e <packages>

5. Verify that the node and the packages start up normally. Give the followingcommand:

# cmviewcl

The startup takes some time. You may have to repeat thecmviewclcommand a few times to see that the packages start up normally.

6. If the MC/ServiceGuard packages do not start up after you have enabledthe package switching attributes, you have to restart the packages manuallyby giving the following command:

# cmrunpkg -v [ -n <server> ] <package>

You must specify the server if the primary node for the package is differentfrom the server on which you are giving the command.

7.5.5 Halting the MC/ServiceGuard cluster

The general syntax for halting the MC/ServiceGuard cluster is given below:

# cmhaltcl [-f] [-v]

Thecmhaltcl command causes all nodes configured in the MC/ServiceGuardcluster to stop their MC/ServiceGuard daemons, optionally halting all packagesin the process if you specify the-f flag. The command halts all the MC/ServiceGuard daemons running on the current system and shuts down the NokiaNMS database and the Nokia NMS processes.

To halt the MC/ServiceGuard cluster


UNIX administration


NTC TAN 0513/3 en56

2. Halt the MC/ServiceGuard cluster by giving the following command:

# cmhaltcl -v

The command halts the cluster if the packages have been halted at anearlier stage. If you have not halted the packages before giving thecommand, you have to specify the-f flag.

3. After halting the cluster there are two options:

• You can proceed to shutting down or restarting the servers. Forinstructions, refer to 7.5.7,Shutting down and restarting a serverafter halting a node or the whole cluster.

• You can continue working on the node. To access the disks ownedby the MC/ServiceGuard cluster they have to be disowned andremounted. Give the following commands:

# /etc/cmcluster/sgsdskmx.sh disown <package># /etc/cmcluster/sgsdskmx.sh -a mount <package>

7.5.6 Starting the MC/ServiceGuard cluster

The general syntax for starting up the MC/ServiceGuard cluster is given below:

# cmruncl [ -v ]

The cmruncl command causes all nodes in a cluster to start their MC/ServiceGuard daemons. On the Nokia NMS servers, thecmruncl commandmounts the required file systems and starts the Nokia NMS database as well asthe Nokia NMS processes. However, if the disks have been disowned while thecluster has been halted, they have to be included in the cluster. For instructionson how to carry out these operations, see 7.5.9,Mounting and unmounting filesystems within a MC/ServiceGuard cluster and 7.5.8, Enabling MC/ServiceGuard node switching.

To start the MC/ServiceGuard cluster


If you have not mounted the disks while the cluster has been down, proceedto Step 4.

2. Unmount the disks before restarting the cluster:

# /etc/cmcluster/sgsdskmx.sh -a umount <package>

Repeat the command for all the packages whose disks you have mounted.

3. Include the disowned disks back in the MC/ServiceGuard cluster:

# /etc/cmcluster/sgsdskmx own <package>

UNIX administration


NTC TAN 0513/3 en57


4. Start the cluster by giving the following command:

# cmruncl -v

5. Verify that the cluster and the packages start up normally. Give thefollowing command:

# cmviewcl

The cluster startup takes some time. You may have to repeat the commanda few times to see that the packages start up normally.

7.5.7 Shutting down and restarting a server after halting anode or the whole cluster

After halting an MC/ServiceGuard node or the cluster you can shut down orreboot the Nokia NMS servers.

To shut down a server

1. Log into the server as theroot user.

2. Bring the server to a halt. Give the following command:

# shutdown -h -y 0

To reboot a server

1. Log into the server as the root user.

2. Reboot the server. Give the following command:

# shutdown -r -y 0

For more information on shutting down and starting up the Nokia NMS serversrefer to 7.1,System shutdown and startup and 7.3,Rebooting the NMS serversand workstations.

7.5.8 Enabling MC/ServiceGuard node switching

For MC/ServiceGuard to be able to switch a package onto a different node, youhave to enable the switching attributes for the package. You also have to enablenode switching after restarting the MC/ServiceGuard cluster for the software towork correctly. You have to do this every time if you have halted a package andrestarted it again.

The general syntax for enabling the node switching attributes on a package is asfollows:

# cmmodpkg { -e | -d } [ -n <node_name> ] [ -v ] <package>

UNIX administration


NTC TAN 0513/3 en58

The cmmodpkg command above enables the ability of an MC/ServiceGuardpackage to switch to an adoptive node in case of node failure and starts thepackage on the primary node if the package has been halted. You can also usethe command to prevent a package from running on a certain node by using the-d flag.

To enable MC/ServiceGuard package switching attributes


Enable the node switching attributes on the packages by giving thefollowing command:

# cmmodpkg -e db comm

The command also starts the packages on their primary nodes if they havebeen halted at an earlier stage.

7.5.9 Mounting and unmounting file systems within a MC/ServiceGuard cluster

Nokia NMS release T10 includes the/etc/cmcluster/sgsdskmx.sh scriptfor manipulating the access to volume groups and file systems configured in aMC/ServiceGuard package or cluster. You can use the/etc/cmcluster/sgsdskmx.sh script to mount and unmount file systems, if you need to accessthe hard disks after halting a MC/ServiceGuard package or cluster.

Depending on which commands you use to halt the MC/ServiceGuard packagesyou have to specify the-a or -e flag after the /etc/cmcluster/sgsdskmx.sh command. Below is the command line syntax of the script:

# /etc/cmcluster/sgsdskmx.sh { -e | -a } <cmd> <package>

The table below explains the command line arguments of the/etc/cmcluster/sgsdskmx.sh script:

<cmd> Command (mount , umount , own, disown )

<package> The name of the package (comm, db, omcdb or spare )

Option Meaning

-a The flag to be used if the disks are not owned by the MC/ServiceGuard cluster,non-clustering mode (if you havehalted the whole MC/ServiceGuard cluster with thecmhaltcl command, for example).

UNIX administration


NTC TAN 0513/3 en59

Table 14.Flags to Be Used with the /etc/cmcluster/sgsdskmx.sh Script

Table 15.Commands Understood by thesgsdskmx.sh Script.

7.5.9.1 Mounting the disks in clustering mode

The following two task sequences below show you how to mount and unmountthe disks belonging to the MC/ServiceGuard packages. You should use thisprocedure when you have halted a package or node.

To mount the disks after halting a package or node


2. Mount the disks belonging to the halted package:

# /etc/cmcluster/sgsdskmx.sh -e mount <package>

Repeat the command for all the packages whose disks you are going tomount. The disks are still owned by the cluster after completing the stepsabove, but they can be accessed normally.

To unmount the disks owned by the MC/ServiceGuard cluster





7.5.9.2 Mounting the disks in non-clustering mode

The following two task sequences below show you how to mount and unmountthe disks belonging to the MC/ServiceGuard packages. You should use thisprocedure when you have halted the whole MC/ServiceGuard cluster.

-e The flag to be used if the disks are currently owned by theMC/ServiceGuard cluster,clustering mode (if you havehalted an MC/ServiceGuard package withcmhaltpkgcommand, for example).

Command Meaning

mount Mount the disks configured in the package

umount Unmount the disks configured in the package.

own Change the volume group toclusteringmode.

disown Change the volume group tonon-clustering mode.

Option Meaning

UNIX administration


NTC TAN 0513/3 en60

To mount the disks after halting the cluster


2. Change the disks to non-clustering mode:

# /etc/cmcluster/sgsdskmx.sh disown <package>

Repeat the command for all the packages whose disks you are going tomount.


# /etc/cmcluster/sgsdskmx.sh -a mount <package>

Repeat the command for all the packages whose disks you are going tomount. The disks are not owned by the cluster after completing the stepsabove.

To unmount the disks not owned by the MC/ServiceGuard cluster





3. Change the disks back to clustering mode:

# /etc/cmcluster/sgsdskmx.sh own <package>


7.5.10 Verifying the functionality of the MC/ServiceGuardcluster

You can view the status of the MC/ServiceGuard cluster by giving the commandcmviewcl and verify that the packages are running as they should. For moreinformation on which packages should be running on the servers, refer toSystemManagement Basic Operating Principles and Procedures, TAN 0732.

To check the status of the MC/ServiceGuard cluster

1. Log into the Communications Server as theroot user


# cmviewcl

The output should show that the required MC/ServiceGuard packages arerunning. The example below is taken from the Standard ServerConfiguration for Workstations (2+1).

UNIX administration


NTC TAN 0513/3 en61

CLUSTER STATUS

<name> up

NODE STATUS STATE

<hostname> up running

PACKAGE STATUS STATE PKG_SWITCH NODE

comm up running enabled <hostname>

NODE STATUS STATE



omcdb up running enabled <hostname>

db up running enabled <hostname>

NODE STATUS STATE



spare up running enabled <hostname>

Example 4.MC/ServiceGuard Status Report

7.6 Managing the Nokia NMS processes

The functionality of the Nokia NMS system is based on a number of processeswhich run on the servers. Some processes are transparent to the user in the sensethat they are started, run and terminated in the background by the Nokia NMSsystem. Other processes are started by the user, for example, by selectingUtils→ Fault Mgmt → Alarm Monitor... on the Top-level User Interface.

For more information on process supervision in the Nokia NMS refer toSystemManagement Basic Operating Principles and Procedures, TAN 0732.

7.6.1 Configuring Process Supervision and Recovery Service

The Process Supervision and Recovery application only monitors the processeswhich have been defined to be under its supervision. These definitions are storedin the wpmanamx.cf configuration file, located in the$OMCCONFPATH/<server> directory.

Note:

A default configuration file is supplied during commissioning, and normally thesystem administrator has no need to edit the file.

SeeTechnical Reference Guide, TAN 0453 for the syntax of this file.

UNIX administration


NTC TAN 0513/3 en62

7.6.2 Terminating and starting the Process Supervision andRecovery application

The Process Supervision and Recovery application can be terminated and startedmanually without disturbing other processes.

If you terminate the Process Supervision and Recovery application manually,you must also restart it manually.

Note:

You should only follow the instructions in this section if your system does nothave the MC/ServiceGuard software installed. If your system has MC/ServiceGuard you cannot terminate the Process Supervision & Recoveryapplication alone.

To terminate the Process Supervision and Recovery application

To terminate only the Process Supervision and Recovery application, take thefollowing steps:

1. Give the following command to determine the process ID of thewpmanamx process:

omc>% mx | grep wpmana

2. Send anINT signal to thewpmanamx process:

omc>% kill -INT <PID of wpmanamx>

Entering the above command will terminate the Process Supervision andRecovery application without disturbing the processes under itssupervision.

To start the Process Supervision and Recovery application

If Process Supervision and Recovery was terminated manually, it can berestarted by entering:

omc>% omkickmx -v -p wpmanamx

If Process Supervision and Recovery was terminated in an abnormal way, thecron daemon will start it after the time period specified in the crontab file. TheProcess Supervision and Recovery application will not be started automaticallyif one is running already or if it has been terminated manually.

7.6.3 Terminating and starting a process under supervision

Individual processes on the server can also be terminated and restarted manually.Note that if you terminate a NMS process, the functions it normally performs are

UNIX administration


NTC TAN 0513/3 en63

temporarily disabled. Thus, it is advisable to terminate processes only when youare specifically instructed to do so by a NMS maintenance manual or other NMSdocumentation.

Note:

If you manually terminate a process under supervision, it has to be manuallyrestarted. SeeTo restart a terminated process below for details.

Manually terminating and restarting a process under supervision requires you todetermine the command line arguments the process needs to start again. You cando this by checking the reference file of the process. See below for detailedinstructions.

When you terminate a process manually, Process Supervision and Recovery willterminate all processes which depend on this process. These "child" processeswill automatically be restarted as soon as the "parent" process is running again.

To terminate a process under supervision

Before terminating a NMS process, you must check the reference file of theprocess to be able to restart it. This has to be done before terminating the process,as the reference file will be removed when the process is terminated.

The reference file specifies the path, the process file name and the command linearguments required to restart the process. To find out this information, take thefollowing steps:

1. Log in as theomc user.

2. Change the working directory to$OMCCONFPATH/<server>. Thisdirectory contains thewpmlibmx.cf configuration file, which specifiesthe directory where the reference files are stored.

3. Check the wpmlibmx.cf configuration file for th.e directoryspecification. The directory entry should be found on the last line of thefile. It may look something like the following:

(directory "/etc/daemons/refs")

This example would mean that the reference files are stored in the/etc/daemons/refs directory.

4. Change to the directory specified in thewpmlibmx.cf configuration fileand enterls . You will see the reference files for all the processes currentlyrunning. The names of the reference files are of the form<name of theserver>.<process name>.<process ID>

5. Now you are ready to terminate the process. You can do this by sendingthe process theTERM signal as theomc user:

UNIX administration


NTC TAN 0513/3 en64

omc>% kill -TERM <pid>

The reference file of the process will be removed and the process will beterminated.

Note:

If you wish to change the arguments of a process which is running in aserver, theTERM signal has to be used. If you terminate the process byusing some other signal, the process will typically be recovered using theoriginal arguments.

To restart a terminated process

To be able to restart a NMS process correctly, you must use the appropriatecommand line arguments. Note that all processes do not require additionalarguments.

1. To restart a NMS process, enter the following command as theomc user:

omc>% omkickmx -v -p <process>

7.7 Resource Supervision Service

All the tasks related to hard disk supervision are presented in this section

7.7.1 Configuring disk supervision

Disk supervision parameters are stored in thespymanmx.cf configuration file.The configuration file is stored in the$OMCCONFPATH/<server> directory. Formore information on the configuration file, seeTechnical Reference Guide, TAN0453.

Thespymanmx.cf configuration file defines which file systems are monitoredby Resource Supervision, together with the alarm points and alarm limits foreach alarm class. The working principle of these parameters is basically the sameas in database supervision:

<pid> Process ID

Field Name Description

<alarm class>_ENABLE Specifies whether a certain alarm class isenabled for an individual file system

<alarm class>_POINT The usage percentage when an alarm isgenerated

<alarm class>_LIMIT The usage percentage when the alarm iscancelled

UNIX administration


NTC TAN 0513/3 en65

Table 16.Disk supervision parameters inspymanmx.cf

The alarms associated with disk supervision are as follows:

Table 17.Disk supervision alarms

The system administrator can use thespymanmx.cf configuration file tospecify the file systems that needs to be monitored by Resource Supervision. Tomodify the parameters to suit the needs of your system, use a text editor to editthe $OMCCONFPATH/<server>/spymanmx.cf configuration file. See theTechnical Reference Guide for the syntax of this file.

7.7.2 Monitoring the disk space

You should continuously monitor the available disk space. To achieve theoptimum performance, the usage of database disks should never exceed 80% ofthe total capacity. You can monitor the disk usage with Disk Supervision whichwill send an internal alarm if the specified alarm point is exceeded. Refer toSubsect1 7.7.1,Configuring disk supervision for instructions.

If there is less disk space than recommended, try to clean up the disk by lookingfor unnecessary files, core files or temporary files which could be deleted. Thesetasks are explained below.

7.7.3 Monitoring log files

The HP-UX operating system writes information into certain log files which willgrow unrestricted from a server restart to the next restart. Normally, theworkstations may run without restarts for lengthy periods. Therefore it isnecessary to truncate and possibly archive log files.

The system keeps track of certain log files, and you do not have to pay anyspecial attention to them. Some log files, however, are not monitored by thesystem, and it is up to you to ensure that they have not grown excessively. Thefollowing subsections cover both log types in more detail.

Alarm No. Alarm Class

9314 Warning

9312 Warning

9304 Warning

9205 Minor

9104 Major

9009 Critical

UNIX administration


NTC TAN 0513/3 en66

7.7.3.1 Log files monitored by the system

The files listed in this section are monitored by the system, which ensures thatthey do not unnecessarily occupy hard disk space.

The wcltoday.log file

The NMS uses the UNIX system log to record system management events(command log). This log is located in the/var/adm directory and its name iswcltoday.log . The log files of the last two weeks (14 days) are stored in thesame directory and renamed towclYYMMDD.log . YYMMDD is the date for thelog records. The system stores information on some events into this log. Theseevents are, for example:

• Startup and termination of processes (wpmanamx)

• Startup and termination of supervision programs (sfwmgrmx )

• Startup and termination of backup (dbbackmx )

• MML Session starts as well as commands (wsesammx)

• Object creations (nedmanmx)

• Radio Network events (cevmgrmx )

The renaming is carried out every night.

Other log files

The following log files are also monitored by the system:

• Information on the SQL*Net TCP/IP Network Servers operations, such asorasrv restarts and connection requests from outside, is stored into the /usr/oracle/tcp/log/orasrv.log file. This file is purged every night.

• The Oracle RDBMS alert file /usr/oracle/rdbms/log/alert_<ORACLE_SID>.log stores information on all Oracle errorsituations as well as database closings and openings. It is moved to/usr/oracle/rdbms/log/alert_<ORACLE_SID>.old every 14 days.

• The Oracle Listener log file/usr/oracle/tcp/log/tnslsnr.log

• The $OMCLOGDIR/werlog<number>.log files on each server storeinformation about actions of the Nokia NMS processes. Their sizes aremonitored by the system, and extensive growth is not possible.

Temporary files are often created in the/tmp and/usr/tmp directories. Thereis an NMS feature which automatically removes the old temporary files (whichare older than two days) from these directories. We advise you not to store anypermanent data into these two directories.

UNIX administration


NTC TAN 0513/3 en67

7.7.3.2 Log files to be monitored by the system administrator

The following files should be checked at least every month to ensure that theyhave not grown excessively:

Table 18.Files to Be Monitored by the System Administrator

Note:

If any one of the above listed files grows excessively, its contents should beexamined for any potential configuration problems.

To clear a log file

If a periodic check reveals that one of the above log files is taking up too muchdisk space and you want to clear the log file, take the following steps.

File Name Description

/var/adm/syslog/syslog.log

This log file is managed by thesyslogdprocess and it stores informationaccording to the settings in the/etc/syslog.conf file.

/usr/etc/yp/ypserv.log Theypserv daemon writes itsmessages, log diagnostics and errormessages into this log file. This is onlydone on the NIS server. You can use theypwhich command to find out themaster NIS server on the WS network.

/usr/etc/yp/ypxfr.log Theypxfr process copies NIS mapsfrom a NIS server to the local host. Itwrites all its messages to this log file onthe NIS server.

/var/adm/shutdownlog This log file stores all reboots and haltsof the system as well as the user ID ofthe user who gave the command.

/var/sam/log/samlog All actions by System AdministrationManager (SAM) are logged into this logfile.

/etc/wtmp Thewtmp log holds information on alllogins and logouts. The log is used bycommands such aslast , who, writeandlogin .

/var/adm/sulog This log file stores information onremote and console logins.

/etc/cmcluster/pkg/<package>/control.sh.log

MC/ServiceGuard package log file onworkstations equipped with the MC/ServiceGuard software.

UNIX administration


NTC TAN 0513/3 en68

1. Log in as theroot user.

2. Rename the existing log file. For example:

# mv <log_file> <log_file>.old

3. Create a new file for logging purposes by entering:

# touch <log_file>

For <log_file> substitute the original name of the log file you renamedin step 1.

4. Ensure that the file permissions are correct by enteringls -l<log_file> for both the file you renamed in step 1, and the empty fileyou created in step 2. If the ownership and permissions are not correct, usechown to change the ownership and thenchmod to modify the permissionsof the new file.

5. For further study, you can move the<log_file>.old onto a tape or ontoanother medium. For example:

# tar cvf /dev/dat <log_file>.old

6. Remove the old version of the file.

# rm <log_file>.old

7. If the file is managed bysyslogd , informsyslogd of the file change byentering:

# kill -HUP `cat /etc/syslog.pid`

7.7.4 Removing core image files

In certain error situations, such as memory violations and illegal instructions inthe process code, the system creates a core image of the memory area of the ill-behaved process and dumps it in its working directory. The resulting image filesare titled core , and they can be quite large. Although these files containinformation about the error and can be used to trace the cause of the interrupt,they can often be disregarded and removed.

To remove core image files

1. Give the following command as theroot user:

# cd /; find . -name core -type f -print \-exec rm -i {} \;

You will be asked to verify the deletion of each file if thefind commandfinds anycore files.

<log_file> Name of the log file

UNIX administration


NTC TAN 0513/3 en69

7.8 Generic Supervision Service

The parts supervised by Generic Supervision Service (Supervision Framework)are monitored by separate programs, which can also be executed from thecommand line for testing purposes.

7.8.1 Supervising the X.25 adapters

Thesfwx25mx.perl program monitors the X.25 adapters installed on the localNMS workstation. Generic Supervision Service controls the execution of theprogram according to the parameters set up in thesfwmgrmx.cf configurationfile. The program executes thex25ifstate command which is used forprobing the X.25 interface cards installed on the NMS workstations. If thex25ifstate detects any anomalies in the X.25 interfaces, thesfwmgrmx.cfprogram passes an error message on to Generic Supervision Service, whichinforms the user of the possible error by generating an internal alarm anddisplaying the error message.

You can also run the program from the UNIX command line. Below you see anexample of the output when run from the UNIX command line.

(version "1.0"

(object "/etc/x25/x25config_0" (info "ok") (status "0")))

Example 5.Sample output from thesfwx25mx.perl script.

The program does not require any command-line parameters.

7.8.2 Supervising mirrored disks

Thesfwmirmx.perl program monitors all the mirrored logical volumes on theNokia NMS workstations. The program informs Generic Supervision Service ifmirroring has not been activated for a particular logical volume, if a mirror diskfails or if the status of a mirrored disk cannot be determined. The program usesthe lvdisplay command to report the status of the logical volumes.

If you run the program from the UNIX command line (no parameters required),the program prints out its messages to the standard output. Below you see anexample of the output when run from the UNIX command line.

(version "1.0"

(object "/dev/vg00/lvol1" (info "ok") (status "0"))

(object "/dev/vg01/lvol5" (info "Mirroring is not active on logicalvolume /dev/vg01/lvol5") (status "1"))

(object "/dev/vgdb01/lvol1" (info "ok") (status "0"))

(object "/dev/vgdb02/lvol1" (info "ok") (status "0"))

(object "/dev/vgglobal/lvol1" (info "ok") (status "0"))

Example 6.Sample output from thesfwmirmx.perl script

UNIX administration


NTC TAN 0513/3 en70

7.8.3 Supervising the database index tablespaces

Thesfwdbimx.perl program monitors the index space allocation of the NokiaNMS database tables. The program supervises that the amount of allocatedtablespaces does not exceed a given percentage and that the size ofNEXT_EXTENTS to be allocated does not exceed a given percentage of freetablespace. The program cannot be run from the UNIX command line.

For more information on other database supervision-related topics, refer toDatabase Maintenance, TAN 0514.

7.8.4 Supervising the flow of BSC traffic measurements

The sfwmeamx.perl program monitors that all the BSCs that have trafficmeasurements activated transfer the measurement data to the Nokia NMS. Theprogram looks for measurement data in the Nokia NMS database and checks thestop times of the measurements. If the measurements seem to be late, theprogram informs Generic Supervision Service, which in turn will generate aninternal alarm for the user.

If you run the program from the UNIX command line, the program prints out itsmessages to the standard output. You should use the-f option if you run theprogram from the command line on another server than the Database Server.Below you see an example of the output.

(version "1.0"

(object "BSC" (intid "102645") (info "Traffic measurement data is atleast 60 minutes late.") (status "1"))



(object "BSC" (intid "130388") (status "0"))

Example 7.Sample output from thesfwmeamx.perl script

7.8.5 Supervising the workstation message service

The wnesfwmx program monitors that all locally configured WorkstationNetwork Communication Service Manager applications work properly. Theprogram supervises only the local NMS connections not the gatewayconnections.

UNIX administration


NTC TAN 0513/3 en71

If you run the program from the UNIX command line, the program prints out itsmessages to the standard output. Below you see an example of the output whenrun from the UNIX command line.

#

# amazon Mon Feb 3 14:16:05 1997

#

(version "1.0"

(object "wnemgrmx connection from amazon to goljat"

(info "")

(status "1")

)

(object "wnemgrmx connection from amazon to sarah"

(info "")

(status "0")

)

)

# end of output

Example 8.Sample output from thewnesfwmx program.

In the example above, the workstation message service connection togoljathas failed.

7.8.6 Supervising the OSI connections

OSI Connections Supervision Service (sfwosimx ) supervises the OSIconnections from the Nokia NMS WSs to an IS. With older configurationsequipped with the Nokia NMS FE, OSI Connections Supervision Servicemonitors the CONS X.25 connections only. With newer Nokia NMS systems,OSI Connections Supervision Service monitors the CLNS LAN connections.When supervising the connections the Generic Supervision Service application(sfwmgrmx ) executes thesfwosimx application at given intervals. Thesfwosimx application, in turn, uses thesfwosimx.perl script whoseexecution of the script is controlled by command line arguments.

UNIX administration


NTC TAN 0513/3 en72

If you run thesfwosimx.perl script from the UNIX command line, theprogram prints out its messages to the standard output. You have to be logged onas theroot user to run the script. Below you see an example of the output whenrun from the UNIX command line.

#

# amazon Mon Feb 3 14:17:05 1997

#

(version "1.0"

(object "primary X.25"

(info "ok")

(status "0")

)

(object "secondary X.25"

(info "Secondary route to front end failed")

(status "2")

)

)

# end of output

Example 9.Sample output from thesfwosimx.perl script

7.9 Modifying the Alarm Flow Supervision parameters

Alarm Flow Supervision monitors the transfer of alarms from network elementsto the NMS and generates an internal alarm if the alarm flow is delayed too muchor interrupted. Alarm Flow Supervision can be used with BSCs running versionS6 of the BSC software and MSCs running version M8 of the MSC software.

You can edit thearcsupmx.cf file to change the supervision parameters tobetter meet the supervision needs of your network. Below you see an example ofthe contents of the configuration file:

(objectClass "3" # BSC

(maxAllowedAlarmDelay "20") # 20 minutes

)

(alarmCheckInterval "120") # 120 seconds

Example 10.Contents of thearcsupmx.cf file

If the default delay of 20 minutes is not appropriate for the particular type ofnetwork element, you can set the delay longer or shorter.

The check interval defines how often network elements are scanned and whetheran alarm should be generated. The value is given in seconds and can be changedif the default value is not appropriate.

There may also be need to add individual network elements under supervisionwith a maximum alarm delay of their own. If you know that a certain networkelement should normally send an alarm at certain intervals, it would make sense

UNIX administration


NTC TAN 0513/3 en73

to create a separate entry for the particular network element. Create an entry forthe network element as in the example below:

(intId "<NE_internal_ID>")

(maxAllowedAlarmDelay "<minutes>")

Example 11.Entry for an individual network element

You should list all individual elements before themaxAllowedAlarmDelayparameter, because there can be only one common value for all individualnetwork elements.

7.10 Checking the processes manually

The Process Supervision and Recovery application monitors the Nokia NMSprocesses which have been defined under its supervision. If Process Supervisionand Recovery detects that a process has been terminated in an abnormal way, itattempts to restart the process. For more information about the processsupervision, see 7.6,Managing the Nokia NMS processes above.

The NMS Process Supervision and Recovery application normally takes care ofmost tasks associated with process supervision. However, there are a number oftasks that the system administrator has to carry out to guarantee the fullfunctionality of the Nokia NMS.

If you want to verify that Process Supervision and Recovery is working normallyand that all the processes are running properly, there are several ways ofmonitoring the processes manually. See the following subsections for details.

To check that all processes are running

The processes which should be running in a server are listed in the file$OMCCONFPATH/<hostname>/wpmanamx.cf . To check which processes arerunning, take the following steps:


omc>% mx

2. If one or several of the processes listed in thewpmanamx.cf file are notrunning, start them:

omc>% omkickmx -v -p <process>

The following commands can be used to verify the functionality of the NokiaNMS processes:

<process> name of the process that you want to start

UNIX administration


NTC TAN 0513/3 en74

Table 19.Process verification commands.

To check automatic process restarts

Automatic process restarts and recoveries by Process Supervision are logged intothewcltoday.log file on each server.

1. You can check the contents of the log file for any restart entries on theserver by entering the following command:

omc>% grep WPMANA /var/adm/wcltoday.log

If the command displays restart entries, you should make sure that all the NokiaNMS processes (ending in-mx ) in the /etc/daemons/refs directory (or inanother directory specified in the$OMCCONFPATH/<server>/wpmlibmx.cffile) have their reference files in the directory.

You should also check thewerlog error log file of the problematic process.They can provide valuable information about the problem situation. The errorlog files are located in the$OMCLOGDIR directory on each server. For moreinformation error logs, seeTechnical Reference Guide, TAN 0453.

7.11 Monitoring the wcltoday.log File

You should check the/var/adm/wcltoday.log file daily. The file should notcontain any serious error messages. The Process Supervision and Recovery(wpmanamx), Database Connection Supervision (dbkeepmx ) and GenericSupervision Framework (sfwmgrmx ) applications add entries to the log, whichprovides valuable information for troubleshooting cases. The file also includesinformation about any network changes. The Radio Parameter Event Manager(cevmgrmx ) application adds an entry for every object creation event on the

Command Description

mx Lists the active NMS processes. This command is usefulin every server of the configuration.

mxo Lists the Oracle processes. This command is useful on theDatabase Server. You should see the entries for thefollowing processes:• ora_pmon_omc

• ora_lgwr_omc

• ora_arch_omc

• ora_dbwr_omc

• ora_ckpt_omc

• ora_smon_omc

x25stat -v Displays the status of the X.25 connections. Thiscommand is useful on the Communications and StandbyServers.

UNIX administration


NTC TAN 0513/3 en75

network to this file. This file usually contains log entries for the Nokia NMSprocesses, such as

• database backup (dbbackmx ),

• database cleanup (do-delete ),

• resource supervision (spymanmx, spysizmx , spyanamx ),

• database functions (dbrmuimx ),

• radio network management (rnwmgrmx ),

• BTS hardware management (hwnhndmx),

• MML sessions (wsesammx) and

• MML command execution (exemmlmx).

If you find any log entries caused by errors, you should investigate them verycarefully.

7.12 Checking revisions

The Nokia NMS has a feature which allows you to check workstation softwarerevisions and control changes in the software.

With this tool, you can view the existing versions of the software and check whathas been changed and what are the present revisions. It is also possible tocompare the present situations at two different sites, for example.

The revision control tool consists of two parts. The first part, thetorquemx.shshell script, can be used to create revision files which can be analysed later.

The actual analysis is carried out using thetorchemx.sh script. It allows youto compare different revision situations and to track the Change Note status, forexample.

Details about the usage of the revision tracking tools are given in the followingsubsections.

7.12.1 Creating revision files

As mentioned above, thetorchemx.sh script is used to create revision files,which can later be used to analyse changes to the system software. The tool canbe used in two ways:

• To create the original revision file

• To create revision snapshot files

UNIX administration


NTC TAN 0513/3 en76

The original revision file can only be created once, and only by theroot user.The file is automatically named asoriginal_revisions.txt . As the namesuggests, it can be used to compare software revisions with the originalconfiguration.

The revision snapshot files can be created later, and compared with the originalrevision file or other snapshot files. Revision snapshot files can be namedrevisions_sitename.date , for example.

The revision files are saved in the$OMCROOT/lib/build_log directory.

Note:

You can entertorquemx.sh -help to display a help screen.

To create the original revision file

To create the original revision fileoriginal_revisions.txt , take thefollowing steps. Note that you have to be logged in as theroot user.

1. Change the working directory as follows:

# exec /bin/csh# cd $OMCROOT/lib/torche

2. Give the following command to create the original revision file:

# ./torquemx.sh

Creating the revision file may take a while.

Note:

If you interrupttorchemx.sh by pressing CTRL + C, all the temporary files inthe $OMCROOT/lib/build_log directory are deleted. Some files, however,are stored into the/tmp directory and must be deleted manually at certain timeintervals.

The output file is automatically namedoriginal_revisions.txt and storedin the $OMCROOT/lib/build_log directory. If a file of this name alreadyexists, an error message will be displayed.

This file reflects the software revision situation, and it includes the name,version, checksum and date for the revisions. For more information on how tocompare revisions, see Section 7.8.2, Comparing Revisions.

UNIX administration


NTC TAN 0513/3 en77

To create revision snapshot files

When you give a file name as a parameter to thetorquemx.sh script, it willcreate a revision snapshot file. This file presents the software revision situationat the time of the script execution.

To create a revision snapshot file, give the following commands as theomc user:


omc>% cd $OMCROOT/lib/torche

2. Give the following command to create the original revision file:

omc>% ./torquemx.sh <file name>

The torquemx.sh tool will asks you to confirm the file name. If you donot accept this file name, the process will be cancelled.

Creating the revision file may take a while.

The file is saved in the$OMCROOT/lib/build_log directory. You can createas many snapshots as you wish. A serial number is added to the file name so thatno existing files can be overwritten.

7.12.2 Comparing revisions

The torchemx.sh tool allows you to compare software revisions. There arethree ways in which the tool can be used:

• The current software revisions can be compared with the original ones

• The current software revisions can be compared with the revisions in agiven revision snapshot file

• The software revisions in a revision snapshot file can be compared with therevisions in another snapshot file

Note that thetorchemx.sh tool cannot be used to create revision files. Forinformation on how to create revision files, see Section 7.8.1, Creating RevisionFiles.

To compare revisions

The software revisions can be compared using thetorchemx.sh tool. There area number of command line parameters which can be used to specify the type ofaction you want.

To compare software revisions, take the following steps:


% cd $OMCROOT/lib/torche

UNIX administration


NTC TAN 0513/3 en78

2. Specify the type of comparison you want to carry out. See the Options andthe Examples subsections for details. For example:

% ./torchemx.sh -ch > revision_report.january1997

A revision report is saved to the given file. For information on how tointerpret the report, see Section 7.8.2.1, Contents of the Revision Report.

Note:

You can entertorchemx.sh -help to view a help screen.

Options of the torchemx.sh script

If you do not specify any file names, the current revisions in the software will becompared with the original revisions which were created using thetorquemx.sh tool. You can also choose whether you want to all files or onlythe changed revisions. If the software has been changed or if some files havebeen edited, information on the revision date is given.

Since the report can be quite lengthy, it is a good idea to redirect the output ofthe torchemx.sh script to a printer or a file using the standard UNIXcommands.

The following command line parameters can be given to specify the type ofcomparison:

Table 20.Command Line Parameters fortorchemx.sh

Usage examples of the torchemx.sh script

The following examples present some uses of thetorchemx.sh script:

Command LineParameter(s)

Description

-all Current revisions of all files and binaries are comparedwith the original revisions. If a file has been edited but therevision has not changed, the date is shown.

-ch As above, with the exception that only the files andbinaries whose revisions have changed or which havebeen edited are shown.

<filename> The given revision file is compared with the originalrevisions. The-all and-ch options can be used tospecify which files are shown.

<filename1><filename2>

Two revision files are compared with each other. You canuse this option to compare the SW revision at twodifferent sites, for example, or check which ChangeNotes have been installed and which have not. The-alland-ch options are applicable as above

UNIX administration


NTC TAN 0513/3 en79

• Comparing the current software revisions with the original revisions

torchemx.sh -all

• Comparing a revision snapshot file with the original revisions so thatonly the changed and edited files are displayed:

torchemx.sh -ch <filename>

• Comparing<filename1> with <filename2> , showing only thechanged and edited files:

torchemx.sh -ch <filename1> <filename2>

Contents of the revision report

The following table shows how the changes in the software versions are reflectedin the revision report.

Table 21.Contents of the Revision Report

7.13 Managing crontab files

The user crontabs are different for each user and the tasks defined in the crontabsare dependent on the server’s role. To modify theomc, root or oracle user'scrontab file, you have to be logged on as the user whose crontab file you intendto modify. The user must also be able to write into the working directory.

With MC/ServiceGuard, the user crontabs are located in the role-specificdirectories in$OMCCONFPATH/<role>. If you need to modify the user crontabfiles, you have to edit the crontabs in the role-specific directories.

With MC/ServiceGuard, the execution of the user cron jobs is controlled by theMC/ServiceGuard control scripts in the/etc/cmcluster/<package>

Status Report

Revision of a file has not beenchanged

The existing revision is printed

A file did not exist in the previousrevision file

The present revision is displayedtogether with a message whichindicates that the file did not exist in theprevious revision file

The revision of a file has changed The old and the new revisions areshown

The revision cannot be found The message revision unsolved isdisplayed

<role> The name of the role-specific directory under$OMCCONFPATH: aps, cs , ds , cds or sps .

UNIX administration


NTC TAN 0513/3 en80

directory. The control scripts always reactivate the user crontab files from therole-specific directories when the MC/ServiceGuard packages switch nodes, orif the packages or the whole MC/ServiceGuard cluster are halted and thenrestarted.

Even if theglobal disk is visible on all servers and you can edit the crontabs onone server, you have to activate the crontabs on the server that corresponds to itsrole. If you are on the Communications Server, for example, you can activate thecrontabs in the$OMCCONFPATH/cs directory.

An example of a crontab entry is given below:

18 0 * * * csh -c 'nice $OMCROOT/bin/runmemx' > /dev/null 2>&1

Example 12.Example of a Crontab Entry

To modify a user's crontab file

1. Log into the Communications Server as the user whose crontab file youwant to modify.

2. Change to a role-specific directory:

% cd $OMCCONFPATH/<role>

3. Edit the user’s crontab filecrontab.<user>.cf in the directory with atext editor.

4. Repeat Steps 2 and 3 with the other server roles in your serverconfiguration. You may have to edit as many as four crontab files for oneuser.

5. Change to the role-specific directory of the current server.

% $OMCCONFPATH/<role>

6. Remove the active crontab and install the new crontab:

% crontab -r% crontab crontab.<user>.cf

7. Verify that the changes you made are correct:

% crontab -l | more

8. Activate the rest of the modified crontabs on the other servers in yourconfiguration by logging into them and repeating steps 5 - 7 on them.

UNIX administration


NTC TAN 0513/3 en81

7.14 Tasks on the Nokia NMS servers with no MC/ServiceGuard software

This section describes tasks to be carried out on servers without the HP MC/ServiceGuard software.

Note:

You should only follow the instructions in this section if your system does nothave the MC/ServiceGuard software installed. We also assume that your serverconfiguration includes the Comm&DB Server only.

7.14.1 Shutting down and starting up the Nokia NMS servers

The shutdown and startup procedures are the same for servers with and withoutthe MC/ServiceGuard software. For instructions on how to start up the NokiaNMS servers, refer to Section 7.2,Shutting down and starting up the systemabove.

7.14.2 Starting up and shutting down the Nokia NMS

To shut down the Nokia NMS

Note:

If you use a LAN for connecting to network elements and shut down the NokiaNMS, you have to reboot the Comm&DB Server to restore Nokia NMS. Forinformation on restarting servers, refer to 7.3,Rebooting the NMS servers andworkstations.

The following steps should be carried out on each server which runs the NokiaNMS processes.


2. Close the connections to the network elements:

• If you use X.25 connections, give the following commands on theComm&DB Server:

# x25stop -d /dev/x25_0# exit

Wait a while for the processes to flush the channels.

UNIX administration


NTC TAN 0513/3 en82

Note:

If you have several X.25 cards, you must repeat thex25stopcommand for each X.25 card, replacing the device specification asnecessary, for example:

# x25stop -d /dev/x25_1

• If you use LAN connections kill the OTS processes. Give thefollowing commands:

# ps -ef | grep ots

Make a note of the Process IDs (PIDs) of theotsamd andotslogdprocesses. Kill the processes:

# kill -TERM <otsamd_pid> <otslogd_pid>

3. Change to be theomc user and find out the process ID of thewpmanamx(Process Supervision and Recovery) process:

# su - omcomc>% mx | grep wpmana

The process ID is indicated by the parameterPID in the command output.Note the process ID for use in the following step.

4. Enter the following command to terminate thewpmanamx process:

omc>% kill -TERM <wpmanamx_pid>

When the wpmanamx process receives theTERM signal, it beginsterminating other NMS processes. Wait a while for the processes toterminate.

5. Enter the following command to check whether some processes are stillrunning:

omc>% mx

If you see entries for NMS processes, terminate them manually byentering:

omc>% kill <pid>

<otsamd_pid> The PID of theotsamd process.

<otslogd_pid> The PID of theotslogd process.

<wpmanamx _pid > The PID of thewpmanamx process

UNIX administration


NTC TAN 0513/3 en83

When themx command no longer reports running processes, the processeshave terminated.

6. Check that none of the processes left stale reference files when they wereterminated by entering:

omc>% cd /etc/daemons/refsomc>% ls

If the directory listing displays reference files, remove them by entering:

omc>% rm *

Repeat the above steps on all servers which run NMS processes.

To start up the Nokia NMS

The Nokia NMS processes can be started using theomkickmx command. Thisprogram is responsible for providing the correct environment and related settingsfor the user and background processes and for starting the background processes.The configuration is based on the workstation supervision configurationgenerated during the commissioning of the Nokia NMS.

1. The omkickmx program activates all the processes configured for theNMS to operate. To start all NMS processes, take the following step. Astheomc user, give the following command:

omc>% omkickmx -v

Note:

You should not normally run this program from the command line,because it causes the processes currently running to be restarted and theprogram itself does not check for duplicate startups.

2. All the Nokia NMS processes are started. If the X.25 connection has beendown, you should start it after starting up the Nokia NMS processes. As theroot user, open the X.25 connections:

omc>% su - root# x25init -c /etc/x25/x25config_0

<pid> The PIDs of the processes still running

UNIX administration


NTC TAN 0513/3 en84

Note:

If you have several X.25 cards, you must repeat thex25init commandfor each X.25 card, replacing the device specification as necessary, forexample:

# x25init -c /etc/x25/x25config_1

7.14.3 Closing and opening the X.25 connections

You have to shut down and start up the X.25 connections before you wish to shutdown the Nokia NMS and after the Nokia NMS has started up. The order of thesetasks is important because by closing the X.25 connections before shutting downthe Nokia NMS ensures that no events are lost while the Nokia NMS is off-line.This is also why the X.25 connections have to be restored only after the NokiaNMS has been started up.

To close the X.25 connections

The following steps should be carried out on each server which has X.25 cardsinstalled.


2. Enter the following command to close the X.25 link:

# x25stop -d /dev/x25_<no.>

Note:

If the server has several X.25 adapters installed, you should repeat theabove command for each X.25 adapter on the server, changing the cardnumber as necessary.

3. Repeat the above steps on each server equipped with X.25 adapters.

To open the X.25 connections

The following steps should be carried out in each server which has X.25 cardsinstalled.


2. Enter the following command to open the X.25 link:

# x25init -c /etc/x25/x25config_<no.>

<no.> Number of the X.25 card, normally beginning from 0.

UNIX administration


NTC TAN 0513/3 en85

Note that if the server has several X.25 cards installed, you should repeatthe above command for each X.25 card in the server, changing the cardnumber as necessary.

Repeat the above steps in each server which has X.25 cards installed.

7.14.4 Starting up and shutting down the Nokia NMSdatabase

There are several cases in which the database has to be shut down:

• Certain system maintenance and upgrade tasks which make changes in thesystem configuration

• Offline backup

• The servers need to be shut down or rebooted

Generally speaking, the database should be started back up as soon as possibleto enable it to start storing network events again.

To shut down the database

Note that when you shut down the database, all connections to the database willbe lost, and network events will not be stored into the database. Because of this,the X.25 links to the network elements should be closed and the Nokia NMSprocesses should be shut down in order to enable the network elements to startbuffering events.

Warning:

You should shut down the database only when a task described in a manual orother NMS documentation instructs you to do so. Shutting down the database onother occasions will cause data loss.

Take the following steps to shut down the database:

1. Log into the Database Server as theoracle user.

2. Enter the following commands to shut down the database:

# su - oracleoracle>% svrmgrlSVRMGR> connect internalSVRMGR> shutdown immediate

If shutting down the database seems to take an excessive amount of time,the RDBMS may be performing a rollback or another time-consuming

<no.> Number of the X.25 card, normally beginning from 0.

UNIX administration


NTC TAN 0513/3 en86

operation. You can view the activities the RDBMS carries out bymonitoring the alert_omc.log file. Enter the following command:

oracle>% tail -f $ORACLE_HOME/rdbms/log/\alert_omc.log

If the RDBMS is carrying out a time-consuming task, simply wait for it tocomplete

When the database has been shut down using the immediate option, start itup and shut it down again:

SVRMGR> startup openSVRMGR> shutdown normalSVRMGR> exit

You should receive a notification that the database has been shut down.Then exit the oracle prompt by entering:

oracle>% exit

The immediate option terminates the current database sessions directly, withoutwaiting for each database user to log out. After the database has been shut downusing this method, it is started up again. Then, shutting it down normally ensuresthat all data is written from the redo log files into the database to avoid data loss.

To start up the database

To start up the database, take the following steps:

1. Log into the Database Server as theoracle user.

2. Enter the following commands to start up the database:

oracle>% svrmgrlSVRMGR> connect internalSVRMGR> startup open

You should receive a notification that the database has been started up.Then exit the oracle prompt by entering:

oracle>% exit

System modifications


NTC TAN 0513/3 en87

8 SYSTEM MODIFICATIONS

This chapter describes various changes that may have to made to the workstationnetwork configuration parameters. For a more comprehensive review ofnetwork-related topics, refer toDCN Management, TAN 0377. This chaptercovers the following sections:

• Setting up a printer in Alarm Viewer

• Installing customised sounds for Alarm Viewer

• Changing the IP address of a server

• Restricting access to certain services on the local host

• Changing the NIS configuration

• Exporting directories

8.1 Setting up a printer in Alarm Viewer

The Alarm Viewer application has a feature for printing alarms to a local printer.This requires a printer that is able to print messages sent in ASCII format.Normal line printers are most suitable for this purpose.

To set up a printer for Alarm Viewer

1. Log into the Application Server as theroot user.

2. Start the HP System Administration Manager (SAM).

3. In the SAM Areas window, select Printers and Plotters.

4. In the Printers and Plotters window, select LP Spooler.

5. In the Printers and Plotters→ LP Spooler window, select Printer andPlotters.

6. In the spooler window, selectActions → Add Local Printer/Plotter →Add Parallel Printer/Plotter or Add Serial Printer/Plotter dependingon the port you are going to use.

7. Select the appropriate interface card and clickOK . If there are severalpossible interfaces, you also have to specify the port number.

8. Enteralarm as the printer name and set the model todumb.

9. If the print spooler was not running before you added the printer, SAMprompts you to start the spooler. ClickYes.

10. Print out a test print from Alarm Viewer to make sure that the installationwas successful. For instructions on how to print from Alarm Viewer, seethe Alarm Viewer help.



NTC TAN 0513/3 en88

8.2 Installing customised sounds for Alarm Viewer

With Alarm Viewer you can use customised sounds to inform the users aboutincoming alarms. To use audio requires a workstation or X station with Audiohardware. Audio hardware is built into all Series 700 workstations except the720, 730, and 750, and the 705 does not include audio software.

To use audio on an X station, you need either an HP ENVIZEX or ENTRIA Xstation that includes an audio accessory kit.

To install customised sounds for Alarm Viewer

1. Log into the Application Server as the user who is going to use thecustomised sounds.

2. Open anxterm session and ensure that theNCSWAVE and AUDIOenvironment variables have been set (theAUDIO variable should not havethe valuenone either).

% env | grep NCSWAV% env | grep AUDIO

If the variables have not been set, insert the following two lines into theuser’s.cshrc file:

setenv NCSWAVE <path_to_sound_files>setenv AUDIO <audio_server>

Note:

You have to restart the whole X session to activate the changes made to the.cshrc file.

3. Start Alarm Viewer from the Top-level User Interface and selectConfiguration → Change Configuration...

4. In the Current Alarm and Regions Settings window, select ChangeSound... You see a list of available sound files. Change the sound for thetype of alarm or warning by clicking the desired audio file button.

If you have problems with making the audio work, seeSolving NMS/2000Problems, TAN 0470.

<path_to _sound_files> The path to the directory where thesound files are located,~/sounds , forexample.

<audio_server> The name of the workstation used forplaying back the sounds.



NTC TAN 0513/3 en89

8.3 Changing the IP address of a server

If you change the IP address of a NMS workstation, you must edit several fileson the servers. You also have to update the NIS maps when you change the IPaddresses of the WSs or the MC/ServiceGuard packages. The following tablesums up the changes to be made on different occasions.

Table 22.IP Address Changes to Be Made on Different Servers

To change the IP address of a server

Note:

This procedure cannot be used for changing the hostname of a server.

To modify the network configuration files, take the following steps:

1. Log in to the NIS master server as theroot user.

2. Edit the/etc/hosts file and replace the IP address entries for all the WSswhose IP address changes. The following table is an example of thecontents of the file.

# IP address hostname comment

127.0.0.1 localhost # loopback

111.222.111.001 <Comm> # Communications Server111.222.250.001 <Comm>2 # Redundant LAN on

# the Communications Server

111.222.111.005 <DB> # Database Server

111.222.111.002 <Standby> # Standby Server111.222.250.002 <Standby>2 # Redundant LAN on

# the Standby Server

111.222.111.003 <App_X> # Application Server

111.222.111.004 <XTerm_X> # X-Terminal

Example 13.Contents of the/etc/hosts File

IP Addressto bechanged

Edithosts

Editnetconf

Editcmclconf.ascii

Edit /etc/cmcluster/pkgcomm/control.sh

Edit /etc/cmcluster/pkgdb/control.sh

Runcmapplyconf

UpdateNIS

CS ✓ ✓ ✓ ✓ ✓

DS ✓ ✓ ✓ ✓ ✓

SS ✓ ✓ ✓ ✓ ✓

AS ✓ ✓ ✓

commpkg

✓ ✓ ✓

db pkg ✓ ✓ ✓



NTC TAN 0513/3 en90

3. Log into the server, whose IP address is to be changed, as theroot user.

4. Edit the/etc/rc.config.d/netconf file and replace the IP addressentry for the workstation. The following file listing is an example of thecontents of the file.

<...>

HOSTNAME="tarzan"

OPERATING_SYSTEM=HP-UX

LOOPBACK_ADDRESS=127.0.0.1

<...>

INTERFACE_NAME[<IF_number>]=lan<IF_number>

IP_ADDRESS[<IF_number>]="<IP_address>"

SUBNET_MASK[<IF_number>]="<subnet_mask>"

BROADCAST_ADDRESS[<IF_number>]="<broadcast_address>"

LANCONFIG_ARGS[<IF_number>]="ether ieee"

<...>

Example 14.Sample of the/etc/rc.config.d/netconf File

5. Halt the MC/ServiceGuard cluster:

# cmhaltcl -f -v

Wait a while for the cluster to halt.

6. On all servers, edit the/etc/cmcluster/cmclconf.ascii , /etc/cmcluster/pkgcomm/control.sh, and/etc/cmcluster/pkgdb/control.sh files. Replace the old IP addresses with the new ones.

7. To update the NIS maps, runypmake on the NIS master server as therootuser:

# /var/yp/ypmake

8. Mount the cluster lock disk for updating the configuration files:

# cd /etc/cmcluster# ./sgsdskmx.sh disown db# ./sgsdskmx.sh -a mount db

9. Update the MC/ServiceGuard binary files on the Communications Server:

# cmapplyconf -C /etc/cmcluster/cmclconf.ascii \ -P /etc/cmcluster/pkgcomm/pkgcommconf.ascii \ -P /etc/cmcluster/pkgdb/pkgdbconf.ascii \

<IF_number> The number of the LAN interface

<IP_address > The IP address to be assigned to the LANinterface

<subnet_mask> The bit mask for the network

<broadcast_address> Broadcast address for the LAN interface



NTC TAN 0513/3 en91

-P /etc/cmcluster/pkgsps/pkgspareconf.ascii \ -P /etc/cmcluster/pkgomcdb/pkgomcdbconf.ascii

10. Copy the updated configuration files to all other servers in the MC/ServiceGuard configuration:

# rcp -r /etc/cmcluster root@<server_name>:/etc

11. Unmount the cluster lock disk:

# cd /etc/cmcluster# ./sgsdskmx.sh -a umount db

12. Reboot the servers (one right after another) by repeating the followingcommand on all servers in the MC/ServiceGuard configuration:

# shutdown -r -y 0

8.4 Configuring the TFTP server

Configuring TFTP (Trivial File Transfer Protocol) server (tftpd ) on yoursystem allows you to make files available to remote clients that support TFTP.This protocol is needed when taking a backup of the router configuration, forexample.

Caution:

Implementations of TFTP present a security problem since no user authorisationmechanism exists within the protocol.

The combination of the home directory and file access permissions enforcesstrict security on what remote systems can read and write to your system viaTFTP. Furthermore, implementing these features through the/etc/passwdentry forces the system administrator to make a conscious decision about whatfiles are accessible with TFTP. This prevents the administrator from accidentallyconfiguring the TFTP server to be insecure.

To configure the TFTP server


2. Add the TFTP protocol entry to the/etc/services file:

ftp 69/udp # ARPA Trivial file transfer protocol

3. Add the following entry to the/etc/inetd.conf file:

tftp dgram udp wait root /usr/lbin/tftpd tftpd

4. Reconfigure/etc/inetd :

# /usr/sbin/inetd -c



NTC TAN 0513/3 en92

5. Add the usertftp to the/etc/passwd file:

tftp:*:510:301:tftp directory: \/usr/tftpdir: /bin/false

6. Add the groupguest to the/etc/group file:

guest:*:301:tftp

7. Create the home directory for the user tftp . Create the directory and thefile owned by the usertftp , and ensure that the directory and the file givesthe usertftp read, write, and execute permissions. For example:

# mkdir -p /usr/tftpdir/router_conf# chown -R tftp:guest /usr/tftpdir/router_conf# chmod -R 700 /usr/tftpdir/router_conf# touch /usr/tftpdir/router_conf/router-confg# chown tftp:guest /usr/tftpdir/router_conf/\router-confg# chmod 700 /usr/tftpdir/router_conf/router-confg

8. Place files that you want to make available via TFTP in the home directoryof the usertftp . If you want to give remote systems permission to retrievea file via TFTP, the file must not only exist in the home directory of theusertftp , but also be readable by the usertftp .

9. Verify yourtftpd installation. Create a file and use thetftp program toperform a file transfer: Place the file in the home directory,/usr/tftpdir . Ensure that the file is readable by the usertftp . For example:

# cd /tmp# echo "Hello, I feel good!" > /usr/tftpdir/testfile# chown tftp /usr/tftpdir/testfile# chmod 400 /usr/tftpdir/testfile# tftp localhosttftp> get testfiletftp> quit

Compare the files:

# diff /tmp/testfile /usr/tftpdir/testfile

diff should not report any differences.

8.5 Changing the NIS configuration

In the Standard Server Configuration for Workstations, the CommunicationsServer is usually configured as the NIS Master, the Standby Server as an NISclient, and one of the Application Servers must be configured as an NIS slave. Ifworkstations or servers in other subnetworks connect to the Nokia NMS, each ofthese subnetworks also needs a separate NIS server.



NTC TAN 0513/3 en93

To change the NIS configuration

1. Log into the NIS master as theroot user.

2. Find out the name of your NIS domain and write it down. Give thefollowing command:

# domainname

3. Stop the NIS services on all servers.

• Give the following commands on the NIS servers as theroot user:

# /sbin/init.d/nis.server stop# /sbin/init.d/nis.client stop

• Give the following commands on the NIS clients as theroot user:

# /sbin/init.d/nis.client stop

4. Remove the old NIS data from the NIS servers. Delete the directory thatcontains the NIS maps.

# rm -rf /var/yp/<domain_name>

5. Disable the automatic NIS startup at reboot on all servers. Edit the/etc/rc.config.d/namesvrs file as below:

NIS_MASTER_SERVER=0NIS_SLAVE_SERVER=0NIS_CLIENT=0

6. Reboot the servers. For instructions on how to reboot the servers refer to7.3,Rebooting the NMS servers and workstations.

7. Define the NIS server configuration as desired by editing the/etc/rc.config.d/namesvrs file. Bear in mind what was said about thenumber of masters, slaves and clients above.

• On a NIS master server edit the file as follows:


• On a NIS slave server edit the file as follows:


• On a NIS client edit the file as follows:

<domain_name> The name of the NIS domain



NTC TAN 0513/3 en94


8. Re-initialise the NIS.

• Give the following commands on the NIS master as theroot user:

# domainname <domain_name># ypinit -m

• Give the following commands on the NIS slaves as theroot user:

# domainname <domain_name># ypinit -s <NIS_master>

9. Restart the NIS processes.

• Give the following commands first on the NIS master and then on theNIS slaves as theroot user:

# /sbin/init.d/nis.server stop# /sbin/init.d/nis.server start# /sbin/init.d/nis.client stop# /sbin/init.d/nis.client start

• Give the following commands on the NIS clients as theroot user:

# domainname <domain_name># /sbin/init.d/nis.client stop# /sbin/init.d/nis.client start

8.6 Exporting directories

By default, the NFS mount request server (mountd ) does not allow anyone tomount directories from the server. To permit one or more hosts to NFS-mount adirectory or a file system, it must be exported. If you want to change the NFSconfiguration of your workstation network by adding or removing hosts, youhave to specify the directories to be exported in the/etc/exports file and listthe servers that are allowed to mount the exported directories.

To export a directory


2. Edit the/etc/exports file and add any hosts to the list or remove theones that should not be able to mount the directory in question.

<dir> -anon=65534,access=<Standby>:<App_X>,root=<Standby>

<NIS_master> The name of the NIS master server



NTC TAN 0513/3 en95

3. Run/usr/sbin/exportfs to make changes valid:

# /usr/sbin/exportfs -a

4. Edit the/etc/fstab file on the remote systems. If you added the serverto the list of permitted hosts, insert the required parameters to mount thefile systems at system startup. If you removed any hosts from the list ofpermitted hosts, delete the mount requests for the directories from the/etc/fstab file. An example of the/etc/fstab file on a remote serveris given below.

# directory mnt pnt type options bu fsck

# ---------------------------------------------------------------

/dev/vg00/lvol1 / hfs defaults 0 1

/dev/vg00/lvol1 /var vxfs delaylog 0 2

fbar:/d/global /m/global nfs rw,retry=100,bg,suid 0

Example 15.Sample of the/etc/fstab File

To unexport a directory

It may be necessary on some occasions to remove a directory from the list ofexported directories. To unexport a directory, take the following steps:

1. Log into the NFS server as theroot user and check which NFS clientshave mounted the directory you want to unexport:

# /usr/sbin/showmount -a

2. On every NFS client that has the directory mounted, enter the followingcommand to see the process IDs and user names that are using the mounteddirectory:

# /usr/sbin/fuser -u <server_name>:<dir>

3. Instruct any users to change to another directory and kill any processesusing the directory:

# /usr/sbin/fuser -ck <local_mount_point>

<dir> The name of the directory (or file system) madeavailable via NFS.

<dir> The full name and path of the directory (orfile system) made available via NFS.

<server_name> The name of the server containing theexported file system.

<local_mount_point> The name of the directory acting as amount point for the exported directory on alocal workstation.



NTC TAN 0513/3 en96

4. On every NFS client that has the directory mounted, edit the/etc/fstabfile and comment out or remove the line containing the mount parametersfor the unexported directory.

5. On every NFS client that has the directory mounted, give the followingcommand to unmount the directory:

# /usr/sbin/umount <servername>:<dir>

6. On the NFS server, edit the/etc/exports file comment out or removethe line that contains the parameters for the directory you want to unexport.

7. On the NFS server, enter the following command to unexport thedirectory:

# /usr/sbin/exportfs -u <dir>

8.7 Restricting access to certain services on the local host

On some occasions, it may be necessary to restrict remote access to certaininternet services on the local host.

You can restrict the access to the local host by editing the/var/adm/inetd.sec file. You can define rules that control the access of particular hostsor networks to the internet services on the local host. The internet super-server(inetd ) reads this file at startup and when a remote host connects to the localhost requesting a certain service,inetd checks the address of the host againstthe rules defined in the/var/adm/inetd.sec and allows or denies access tothe service requested.

The /var/adm/inetd.sec file syntax is as follows

<service_name> { allow | deny } [ <IP_address> ] [ <name> ]

<service_name> The name of the services defined in the/etc/services , or with RPC services,/etc/rpc file.

<IP_address> The IP address of a host or network

<name> The name of a host or network.



NTC TAN 0513/3 en97

You can use a text editor or SAM to edit the/var/adm/inetd.sec file. Seebelow for a sample of the file.

# allow rlogin from these two hosts

login allow 192.168.1.3 192.168.1.4

# deny ftp access from the network 191.215.0.0

ftp deny 191.215.*

s

# deny all access to the shell service (remsh)

shell deny *

# deny telnet from these hosts

telnet deny tarzan jane boy

Example 16.Sample of the/var/adm/inetd.sec File

The services not listed in the/var/adm/inetd.sec fall back to the defaultsecurity settings offered by the service itself.


NTC TAN 0513/3 en98

INDEX

A

accessinetd.sec 96restricting 96

addingalarm pipes 23

alarm liftingmodifying rules 21

alarm pipes 23adding 24deleting 25

Alarm Viewercustomising sounds 88setting up printer 87

alarmsblocking 28disabling 29enabling 29

Application Server 14

B

backups 32critical files 39Front End 41restoring a Front End backup 42restoring a full backup 39restoring critical files 41routers 43taking a full backup 33

blockingalarms 28

C

Changes from T8 to T9 6checking

cluster 60network element status 26NMS status 27process restarts 74processes manually 73

closingX.25 connections 84

clusterhalting 55

starting 56clustering mode 59Comm&DB Server 14Communications Server 14comparing

revisions 77configuring

customised sounds for Alarm Viewer 88disk supervision 64IP addresses 89NFS 94NIS 92printer for Alarm Viewer 87TFTP 91

core dumpsremoving files 68

creatingrevision snapshots 77

critical filesbacking up 39restoring backup 41

crontabsexamples 80managing 79

D

databaseshutting down 85starting up 85

Database Server 14deleting

alarm pipes 23core image files 68

disablingnetwork alarms and measurements 29

E

edmmanmxdisabling network alarms and

measurements 29enabling network alarms and

measurements 29enabling

network alarms and measurements 29node switching 57


NTC TAN 0513/3 en99

F

Fault Management 21Front End

backing up 41restoring backups 42

full backup 33restoring 39

G

Generic Supervision Service 69

H

haltingclusters 55nodes 53packages 52

hard disksmonitoring free space 65

HP 9000 server 15HP 9000 workstation 15

I

IP addresseschanging 89

L

log filesmonitored by system 66monitored by system administrator 67monitoring 65Oracle Listener 66Oracle RDBMS 66other 66wcltoday.log 66

M

MC/ServiceGuard 51, 52, 53, 54, 55, 57, 60checking status 60clustering mode 59cmhaltcl 55cmhaltnode 53cmhaltpkg 51cmmodpkg 57cmruncl 56cmrunnode 54

cmrunpkg 52cmviewcl 60mounting disks 58non-clustering mode 59sgsdskmx.sh 58

MC/ServiceGuard Packages on the NokiaNMS Servers 48

measurementschecking the BSC 30checking the NMS 31disabling 29enabling 29

modifyingAlarm Flow Supervision 72crontab files 79

monitoringdisk space 65log files 65PM data 30wcltoday.log 74

mountingdisks in cluster 58disks in clustering mode 59disks in non-clustering mode 59

N

network moitoringalarm status 27

network monitoringchecking elements 26verifying functionality 26

NFS 15exporting directories 94

NFS client 15NFS server 15NIS 15, 92

reconfiguring 92NIS client 15NIS master 15NIS slave 15NMS database

shutting down 85starting up 85

NMS processeswpmanamx 73

nodecmhaltnode 53halting 53node switching 57restarting server 57starting 54


NTC TAN 0513/3 en100

node switchingenabling 57

Nokia NMSshutting down 50starting up 50

non-clustering mode 59

O

Occasional tasks 18opening

X.25 connections 84OSI Connections Supervision Service 71

P

packagescmhaltpkg 51halting 52starting 53

Performance Management 30Periodic tasks 17Process Supervision and Recovery Service 61,

73terminating 62

processeschecking manually 73recovering 61restarting 64supervision 61terminating supervised processes 62verification commands 74wpmanamx 62

R

recoveringprocesses 61

reportsrevision 79

Resource Supervision Service 64alarms 65configuring 64parameters 65spymanmx.cf 64

restoringfull backup 39

revision report 79revisions

checking 75comparing 77creating with torquemx.sh 75

routertaking a backup of the configuration 43

S

server configuration 15shutdown

Nokia NMS 50Nokia NMS without MC/ServiceGuard 81without MC/ServiceGuard 81

shutting downNMS database 85Nokia NMS 50

Standby Server 15starting

clusters 56nodes 54packages 53processes under supervision 62

starting upNMS database 85Nokia NMS 50

startupNokia NMS 50

Supervision Framework 6, 69BSC traffic measurements 70database index tablespaces 70mirrored disks 69OSI connections 71sfwdbimx.perl 70sfwmeamx.perl 70sfwmirmx.perl 69sfwosimx 71sfwosimx.perl 72sfwx25mx.perl 69wnesfwmx 70Workstation Message Service 70X.25 adapters 69

T

terminatingProcess Supervision and Recovery Service

62processes under supervision 62

To change the NIS configuration 93To Change the Workstation IP Address 89To Check That All Processes Are Running 73To Check the Automatic Process Restarts 74To check the status of the MC/ServiceGuard

cluster 60To Clear a Log File 67


NTC TAN 0513/3 en101

To close the X.25 connections 84To compare revisions 77To create revision snapshot files 77To create the original revision file 76To enable MC/ServiceGuard package

switching attributes 58To export a directory 94To halt an MC/ServiceGuard node 53To halt the MC/ServiceGuard cluster 55To halt the MC/ServiceGuard packages 52To modify a user's crontab file 80To mount the disks after halting a package or

node 59To mount the disks after halting the cluster 60To open the X.25 connections 84To Remove Core Image Files 68To restart the HP 9000 workstations 50To restore a critical files backup 41To shut down an HP 9000 workstation 48To shut down the database 85To shut down the Nokia NMS 81To start an MC/ServiceGuard node 54To start the MC/ServiceGuard cluster 56To start the MC/ServiceGuard packages 53To Start the Process Supervision and Recovery

Application 62To start up the database 86To start up the HP 9000 servers 48To start up the HP 9000 workstation 49To start up the Nokia NMS 50, 83To take a backup of the critical files 39To take a backup of the new build 42To Terminate a Process Under Supervision 63To terminate all the Nokia NMS processes 50To terminate the Process Supervision and

Recovery application 62To unexport a directory 95To unmount the disks not owned by the MC/

ServiceGuard cluster 60To unmount the disks owned by the MC/

ServiceGuard cluster 59To verify that BSC measurements are not

buffered in the BSC 30To verify that measurement information is

correct in the NMS 31To verify that measurements from the BSC are

transferred to the NMS 30torchemx.sh 75, 77

examples 78torquemx.sh 77truncating

log file 67

W

wcltoday.log 66, 74monitoring 74

workstation network maintenance 17Workstation Network Modifications 87

X

X terminal 15X.25 16X.25 connections

closing 84opening 84


NTC102

Documents

0513_3 Workstation Network Maintenance