EMC Control Center Troubleshooting

This document presents descriptions and remedies to problems you may encounter installing or using components of EMC ControlCenter version 5.0.3. Topics include:

◆ Introduction............................................................................................2◆ System Overview...................................................................................5

Procedures◆ Troubleshooting Principles ................................................................12◆ Diagnosis and Remedy.......................................................................15

Reference ....................................................................................................40◆ Agent Names and Codes....................................................................41◆ Port Settings .........................................................................................45◆ Trace and Log Files..............................................................................45◆ Error Messages.....................................................................................47◆ Tools.......................................................................................................47

EMC ControlCenterVersion 5.0.3

Troubleshooting GuidelinesP/N 300-000-302

REV A02

6/17/02

1

2

Introduction

IntroductionTroubleshooting involves diagnosing and often simultaneously repairing a system malfunction. A range of behaviors, from undesirable (for example, poor performance from traffic bottlenecks) to nonperformance (for example, functional errors, data inaccessibility, or corruption) may prompt you to act.

EMC ControlCenter™ aids in troubleshooting and fixing the storage environment it oversees. The scope of this manual, however, is the troubleshooting of ControlCenter software itself. While other ControlCenter manuals cover the successful installation and functioning of ControlCenter, this guide covers unexpected failures that could disable normal ControlCenter operation, describing symptoms that may include unusual behavior in the GUI.

This manual identifies categories of possible unacceptable behaviors in ControlCenter, and recommended user reactions to them. It attempts to identify occurrences by known symptoms. It does not cover—but may provide pointers to—preventative actions if they concern the same system failures.

Audience This document is intended for use by ControlCenter system administrators.

About this Document This document provides the two basic types of diagnostic and remedy information: procedures and references.

Procedures

Commonly encountered problems, along with their remedies, are described with specific procedures. In certain cases, specific remedies cannot be predefined, and a strategy or an escalation path to EMC Customer Support is given.

See Troubleshooting Principles on page 12 and Diagnosis and Remedy on page 15.

Reference

Tables and examples describing diagnostic objects such as logs, initialization files, and the like are provided here.

See Reference on page 40.

EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines

Introduction

RelatedDocumentation

To avoid problems, the user should adhere to specifications provided in the EMC ControlCenter version 5.0.3 Installation and Configuration Guide, EMC ControlCenter Version 5.0.3 Release Notes, EMC ControlCenter/Open Edition Support Matrix, and EMC ControlCenter Performance and Scalability Guidelines.1

You should have the ControlCenter version 5.0.3 documentation set when troubleshooting, or when preparing to call an EMC representative or EMC Customer Support. It contains the following:

1. EMC ControlCenter version 5.0 Tutorial CD, P/N 053-000-939 Rev A01, provides a tour of ControlCenter features.

2. Within the ControlCenter application, Online Help is provided through the main menu Help, with submenus Contents, Index, Search and Error Message Help. This is a searchable resource with more detailed information than the print documentation.

3. The following printed documents are provided with version 5.0.3:

• EMC ControlCenter version 5.0 Introduction, P/N 300-000-295 Rev A01

• EMC ControlCenter version 5.0.3 Installation and Configuration Guide, P/N 300-000-296 Rev A02

• EMC ControlCenter version 5.0.3 Installation Map, P/N 300-000-297 Rev A02

• EMC ControlCenter version 5.0 Quick Reference Guide, P/N 300-000-298 Rev A01

• EMC ControlCenter version 5.0.3 Release Notes, P/N 300-000-303 Rev A02

4. In addition to those listed above, the following documents are provided in PDF format on EMC ControlCenter version 5.0.3 Documentation/Help CD, P/N 053-000-938 Rev A02:

• EMC ControlCenter version 5.0 User Guide, P/N 300-000-299 Rev A01

• EMC ControlCenter version 5.0 OS/390 Tape Cartridge Release Notes, P/N 300-000-304 Rev A01

• EMC Symmetrix Storage Concepts Guide, P/N 300-000-306 Rev A01

1. EMC ControlCenter/Open Edition Support Matrix and EMC ControlCenter Performance and Scalability Guidelines are available on Powerlink (http://powerlink.emc.com).

EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines 3

4

Introduction

• EMC ControlCenter version 5.0 Integration Packages Product Guide, P/N 300-000-300 Rev A01

• EMC ControlCenter version 5.0 Migration Guidelines, P/N 300-000-301 Rev A01

• EMC ControlCenter version 5.0.3 Troubleshooting Guidelines, P/N 300-000-302 Rev A01

Typographical ConventionsEMC uses the following type style conventions in this guide:

AVANT GARDE Keystrokes

Palatino,bold

◆ Dialog box, button, icon, and menu items in text◆ Selections you can make from the user interface,

including buttons, icons, options, and field names

Palatino,italic

◆ New terms or unique word usage in text ◆ Command line arguments when used in text◆ Book titles

Courier, italic

Arguments used in examples of command line syntax.

Courier System prompts and displays and specific filenames or complete paths. For example:

working root directory [/user/emc]:

c:\Program Files\EMC\Symapi\db

Courier, bold

◆ User entry. For example:symmpoll -p

◆ Options in command line syntax


System Overview

System OverviewControlCenter is a suite of applications used to manage the data storage and related devices in an EMC Enterprise Storage Network (ESN). A very brief overview of system components is provided here.

For a much more detailed overview on components and their context, consult EMC ControlCenter Version 5.0 Introduction (P/N 300-000-295).

Components ControlCenter is a suite of interworking software components. Each component is the subject of various tools that aid troubleshooting.

Figure 1 ControlCenter Software Tiers

Infrastructure Tier The infrastructure manages and maintains the driving components of ControlCenter, the ECC Server, Store, and Repository.

ECC Server This is the primary driving component of ControlCenter. It performs several management tasks among components.

RepositoryThis component is a relational database holding the current and historical metadata for the ControlCenter Console and infrastructure, as well as the extended storage environment.

Agent Tier

Console(s)

Infrastructure TierConsole Tier

ECC-000042

Repos

omer Storage Environment


6

System Overview

Store The Store populates the Repository with data collected from the storage environment by the agents.

Agent Tier Master AgentA Master Agent runs on each host to manage individual agents on that host.

Storage AgentsThese components handle communication with specific storage systems.

Includes agents for — Symmetrix®, Celerra™, CLARiiON®, Compaq StorageWorks, Hitachi Data System (HDS), RVA/SVA, and IBM ESS.

Host AgentsEach operating system class requires a host agent tailored to its environment.

Includes agents for — Windows, Solaris, HP-UX, AIX, MVS, and Novell.

Connectivity AgentsThese agents gather information about the interconnections between storage-system hosts, devices, and networks.

Includes agents for — SNMP, switches, and SDM.

Database AgentsTwo agents provide for communication with databases.

Includes agents for — Oracle and DB2.

Backup AgentsThese agents assist in backup operations.

Includes agents for — Tivoli Storage Manager for: AIX, HP-UX, Windows NT, Windows 2000, and Solaris.

Tape AgentProvides capability for reporting on and managing tape subsystems.

Includes agents for — OS/390 for StorageTek/STK, VTS/SMS, CA1/TMC, and RMM/DFSMS.

Other AgentsSome agents are not specific to hosts or other devices, but act on multiple devices and applications, or the system as a whole.


System Overview

Includes agents for — Integration Gateway and Workload Analyzer (WLA) Archiver.

Console and ClientTier

The Console is the consolidated GUI for ControlCenter. It provides a window on the objects managed by application, their attributes, and their relationships to each other. See Figure 2 on page 8.

Other client applications may be part of this tier.

Interfaces Command LineControlCenter processes produce command line output, which is written to a corresponding output file. See Table 34 on page 46.

Installation UtilityParts of the installation and configuration procedures are managed through the Installation utility. This software is installed from the ControlCenter CD set early in the installation procedure.

ConsoleAfter installation, normal ControlCenter operations are run through the GUI provided by the Console. (Examine detail in Figure 2 on page 8, or consult other ControlCenter manuals.)


8

System Overview

Figure 2 ControlCenter Console (Example View)


System Overview

Recognizing Problems in the Monitored Storage System

The following descriptions refer to features for dealing with problems detected by ControlCenter in the managed objects it is monitoring. Malfunctions in ControlCenter itself are the focus of this document, however; see the discussion starting with the following section, Troubleshooting Principles on page 12.

EMC ControlCenter Version 5.0 Introduction (P/N 300-000-295) and EMC ControlCenter Version 5.0 User Guide (P/N 300-000-299) provide more detailed descriptions of ControlCenter components and their operation.

GUI Devices The GUI has several devices that will indicate error conditions, when encountered, in the devices it monitors.

Window Panels

Tree Panel

This lower left-hand panel displays a structured list of storage environment physical and logical objects. The list may be collapsed or expanded in a manner similar to that in Windows Explorer.

A line-item object (for example, a physical device) may indicate a problem by using a warning or error version of its miniature icon. See Table 1 for descriptions of icon problem indicators.

Target Panel

As in the tree panel, warning and error icons indicate problems with devices. The icons may be presented in a graphical layout or as part of a table.

IconsThe Console uses small icons to indicate that particular devices are part of the interworking storage network. These icons provide:

Device Recognition - A device that may be physically present but is not recognized will not appear as a listed object in the tree or in configuration dialog boxes.

Warnings and Errors Indication - When there is a problem with a known device, the icon will change appearance to indicate that a problem exists. See Table 1 for descriptions.


10

System Overview

GUI Content AlertsControlCenter agents can be configured to monitor specific variables for warning and error conditions.

A condition occurs ordinarily when a value exceeds a preset threshold. When a warning or error condition occurs, an alert message is sent by the monitoring agent to the ECC Server. (In certain cases an autofix automatic recovery is performed.)

Activated alerts may be reviewed in the Console by monitoring alerts. An alert toolbar appears at the right side of the task bar. It shows number of critical errors as well as total number of alerts. Clicking the alert button displays an alert panel in the target panel.

Figure 3 Alert Toolbar

Table 1 Error Indicators in Console Icons

Icon Interpretation

Red circle with cross: Fatal or Critical alert exists in managed object

Yellow triangle with exclamation point: Warning or Minor alert exists in managed object

Red arrow at lower right of folder: Folder contains children (at any level below) that have alerts

Red circle (shown) or Yellow triangle:Named folder’s children (at any level below) have alerts

Red circle or Yellow triangle (shown), with red arrow:Managed object, which also acts as a folder, contains children (at any level below) that have alerts


System Overview

Activity Records Command History

From Console Monitoring view, Command History provides a table containing details about the sessions and commands associated with selected objects that were issued through the Console.

Trace and Log Files

System activity is recorded in some cases in trace (*.trc) files, and in some cases to log (*.log) files. For more information on these files, refer to Trace and Log Files on page 45.


12

Troubleshooting Principles

Troubleshooting PrinciplesControlCenter provides several reporting mechanisms and tools for recording application events and alerting the user when problems occur.

Recognizing ControlCenter Problems

ControlCenter is a configuration, maintenance, monitoring, and recovery system for a target storage environment. If ControlCenter has been configured correctly and is operating properly, then when its managed objects produce state errors, ControlCenter agents will send alarms as well as record passive indicators (such as error events) to log files.

Monitoring The user should establish a schedule for regular system monitoring. Through regular use, problem indicators are learned and problems may be recognized as soon as possible after they occur.

When ControlCenter Malfunctions

◆ Note the symptoms that prompted your attention. Record any information provided (pop-up dialog boxes, warning icons, and so on).

◆ Use Online Help from the Console (if it is running) to interpret error messages.

◆ Find symptoms in diagnostic tree, and review the information provided in the various paths.

If symptoms are not identified in diagnostic tree—

◆ Identify tools available from Powerlink (http://powerlink.emc.com/).

If symptoms and solutions are not identified in Powerlink, or you are otherwise unsure whether to take an action —

◆ Contact Customer Support.

• Before calling, have the following files available:

All server.trc.<n>, store.trc.<n>, consoleLaunch.log, console.trc, masteragent.log, <agent>.log files. Refer to the Reference section for information on locating these files.



Responding toProblems

The following is a general response sequence to symptom discovery. Use the itemization that follows as a checklist.

Symptom IdentificationTo identify the symptom of a ControlCenter problem:

1. Recognize the problem — Determine what specifically and visibly indicates a problem (such as warning icon appears, alert appears, expected icon is missing, pop-up error window). Note what is visible without (at this point) manipulating the GUI.

• In the diagnostic tree, see Symptoms (white box at top), as well as Procedure and Context (grey box below).

2. Interpret the Symptoms — Formulate an explanation (where, when, and how it is occurring).

• In the diagnostic tree, see Suspected Diagnosis column.

• If there is no obvious interpretation, follow diagnostic tree tests, in Item order.

PreparationTo prepare before acting:

3. Isolate the problem — Identify where (on which hosts, between which connections) the problem has occurred or is occurring.

4. Protect the environment — If relevant and practical, physically or logically isolate the problem from the rest of the symptom.

5. Gather information — At this point, look for additional symptoms.

• In the diagnostic tree, follow Diagnosis and Remedy paths in the order of numbered Suspected Diagnoses.

6. Decide if help is needed.

• In the diagnostic tree, Diagnosis and Remedy instructions may suggest you attempt a fix, or may ask you to call Customer Support.


14


DiagnosisWhen the diagnostic tree (see discussion beginning at page 15) does not address a problem:

7. Explore the symptoms — navigate through (drill down) the GUI tree and select views for further error indicators; open your logs.

8. Organize evidence gathered from exploration, and hypothesize diagnoses.

9. Formulate remedies based on your hypotheses.

RemedyFollowing confirmation of a Suspected Diagnosis, steps will be implied, or made explicit, in Diagnosis and Remedy Procedures to:

10. Perform remedies.

11. Test as indicated to confirm success.


Diagnosis and Remedy

.

Diagnosis and RemedyOnly a site system administrator or similarly qualified ControlCenter user should perform diagnostic procedures.

Using the Tree The diagnostic tree is a set of tables, generally one for a particular major symptom or group of symptoms indicating a malfunction. Each table has three basic parts:

◆ At the top of a table (in a white box), symptoms are stated.

◆ Below the symptoms (in a shaded box), operator context is identified.

◆ Below the symptoms and context, a set of numbered Suspected Diagnoses is given, with corresponding Diagnostic and Remedy Procedures.

Table 2 Diagnostic Tree Key

Symptoms

Procedure if applicableBook

ChapterSections

Context if applicableConsole

Tree panelMenu Items

Final Menu Item

Item Suspected Diagnosis Diagnostic and Remedy Procedures

Reference Number for each suspected diagnosis

Suspected Diagnosis is one (of possibly several) root causes for why the symptom occurred.

Diagnostic Procedure is the path the user should follow to confirm or deny a particular Suspected Diagnosis. Includes any work needed prior to actual fixing procedures—protect environment, gather visible information.Remedy Procedure is the (additional path) path (if any) the user should follow to addressed a confirmed Suspected Diagnosis.

Symptoms describes what the user saw to prompt his or her reaction.Procedure describes what the user was doing and/or the user’s objectiveContext tells where in the product interface the user was working.


16


Tree Outline Table 3 gives an outline of topics for which troubleshooting procedures are provided in the diagnostic tree (Tables 5 through 27) on the following pages.

Table 3 Diagnostic Tree Outline

Context Symptom Page

Infrastructure Installation

InstallInstallation Won’t Launch 18Error Dialog Boxes: “InstallerPane”, “error code 2” 19

Error Dialog Box: “ECC uninstall directory” 20Error Dialog Box: “Failed to launch”, “Access is denied” 21Error Dialog Box: “no disk in drive” 22

CD Fails to Eject 22Error: “Return code 255” 23Repository Installation Slow 23

Repository Installation Hang at 1% 24WLA PerformanceView License Key Not Valid 25

UninstallError Dialog Boxes: “InstallerPane”, “error code 2” 26

Processes and Connectivity

ECC ServerServer-Database Connection Fails 27Server-Database Connection Dropped 28

StoreStore-Database Connection Dropped 29

Console

Console Does Not Appear on Startup 1 30Console Does Not Appear on Startup 2 31

Hosts

Host Icon Does Not Appear in Console 35Agents

Agent Installation Cannot Be Performed on Console 32

Agent Startup Quits 33Host Agent Device Icon Does Not Appear in Console 36Console Error Dialog Box: “Agent not found”, “Communication failed”

39



Documentation References

Table 4 is intended to parallel the outline in Table 3. It identifies documentation to be consulted for further information about component installation and operation.

Symmetrix systems

Symmetrix Icon Does Not Appear in Console 37Symmetrix Icon Does Not Appear in Console; Agent Error Message

38

Table 3 Diagnostic Tree Outline (continued)

Context Symptom Page

Table 4 Topics Covered in Documentation

Scope Manual

Preparation

NetworksHostsEnvironments

EMC ControlCenter version 5.0.3 Installation and Configuration Guide, Chapter 2, Chapter 3.

InstallationECC ServerStoreRepository

EMC ControlCenter version 5.0.3 Installation and Configuration Guide, Chapter 2.

ConsoleOther ControlCenter clients

EMC ControlCenter version 5.0.3 Installation and Configuration Guide, Chapter 3.

Agents EMC ControlCenter version 5.0.3 Installation and Configuration Guide, Chapter 4, Chapter 6.

StartupMaintenance

Repository EMC ControlCenter version 5.0.3 Installation and Configuration Guide, Chapter 9.


18


Diagnostic Tree Refer to Table 3 on page 16 for a list of all symptoms.

Table 5 Installation Won’t Launch

Installation won’t launch

ProcedureInstallation and Configuration Guide

Context(Various)


1 CD-ROM drive in use. 1. Shut down any other running media players.2. Eject CD 1.3. Reinsert CD 1 and restart installation.

2 Autorun not enabled. You can launch the installation from CD 1 by command line: <CD_drive>:\ECC.exe

3 Dirty or scratched CD-ROM. 1. Clean CD: Wipe with clean cloth from inner toward outer edge.2. If CD still appears damaged, contact EMC for replacement.



Refer to Table 3 on page 16 for a list of all symptoms.

Table 6 Error Dialog Boxes: “InstallerPane”, “error code 2”

Error dialog boxes appear during installation


Context• Installation startup from CD 1


4 Installation target is nonwritable or nonexistent.

1. Close dialog boxes. Data entry screen “Choose installation directory” should be available.

Refer to EMC ControlCenter version 5.0.3 Installation and Configuration Guide, Chapter 2.

2. Enter valid, writable folder name for installation target.3. Continue installation.


20



Table 7 Error Dialog Box: “ECC uninstall directory”

Error dialog box appears during install


Context• After an uninstall• Installing infrastructure • Installation startup


5 Although registry keys have been removed, uninstall folder still exists.

1. Add registry key (REG_SZ):

HKEY_LOCAL_MACHINESOFTWARE

EMC CorporationEMC CONTROL CENTER

5.0INSTALL_ROOT

a. Open the Registry Editor with: Start, Run, regedit (or regedt32). b. Drill down (as shown above), and select folder (5.0). c. Create new key (right-click in right-hand pane, and select New, String

Value).

2. Set value of new key. To do so, right-click name, select Modify, under Value Data in text box,

enter absolute pathname of installation directory.

3. Restart installation.





Table 8 Error Dialog Box: “Failed to launch”, “Access is denied”

Error dialog box appears during installation


Context• After an uninstall• Installing infrastructure • Installation startup from desktop or Start

menu• After splash screen


6 One or more folders within the installation folder are missing or corrupt.

1. Remove the two registry keys shown:



5.0INSTALL_ROOTLAST_CDDRIVE_USED

a. Open the Registry Editor with: Start, Run, regedit (or regedt32). b. Drill down (as shown above), select, and delete.

2. Remove installation root folder.3. Restart installation from CD 1.



22



Table 9 Error Dialog Box: “no disk in drive”

Error dialog box appears during installation


Context• Installing infrastructure • Installation startup• CD-ROM tray is open


7 CD-ROM is expected in drive tray.

This error is harmless to software and installation procedure.

1. Insert next CD and close drive; dialog box should disappear.2. Continue installation.


Table 10 CD Fails to Eject

CD does not eject automatically


Context• Installation startup• Auto-eject is enabled


8 Windows Explorer CD-ROM folder is open. 1. Close Windows Explorer, or just the CD-ROM folder within it.

9 Demo32.exe process is still running. 1. Open Task Manager, select Processes tab, select Image Name list. 2. Locate and select Demo32.exe and click End Process.




Table 11 Error: “Return code 255”

CD 2 installation aborts at 90% complete, with error: “Return code 255”


Context• Installing infrastructure: Repository


10 The Repository backup directory is read-only or does not exist. Refer to EMC ControlCenter version 5.0.3 Installation and

Configuration Guide, Chapter 8.

1. Uninstall the Repository. 2. Create Repository backup directory.3. Reinstall Repository.

Table 12 Repository Installation Slow

Repository installation seems takes longer than 35 minutes




11 Host may not be equipped according to EMC specifications.

Check that system configuration meets current specifications:• Refer to EMC ControlCenter/Open Edition Support Matrix dated

4/24/02 or later, available on EMC Powerlink.• If unavailable, refer to EMC ControlCenter Performance and Scalability

Guidelines (P/N 300-000-431 Rev A04 or later), available on EMC Powerlink.

12 Repository host is down. 1. Leave ControlCenter installation as is—do not yet take recovery measures.

2. Check that Repository host is running. Reboot if necessary.3. Wait for ControlCenter installation to recognize the Repository; then

continue.


24



Table 13 Repository Installation Hang at 1%

Installation of Repository hangs at 1% completed


Installing and Configuring the InfrastructureInstalling the Repository



13 Slow host processor. Installation can take several minutes to advance past 1%; please wait. The process should not take longer than 35 minutes; else, refer to Table 12 on page 23.

14 Oracle silent installation failed. Look for Oracle installation error messages in:\\Program Files\Oracle\Inventory\logs\

InstallActions.log

and\\Program Files\Oracle\Inventory\logs\

oraInstall.err

15 Windows version is neither of the supported English or Japanese versions.

Replace host or host operating system with the EMC-supported English or Japanese language Windows 2000.




Table 14 WLA PerformanceView License Key Not Valid

WLA PerformanceView license key not valid


Installing and Configuring the InfrastructureInstalling Other Components

Context• Installing Workload Analyzer


16 The host had a previous installation of ControlCenter, but was not fully cleaned of previous software.

1. Delete any remaining previously installed folders.2. Reinsert CD 1.


26



Table 15 Error Dialog Boxes: “InstallerPane”, “error code 2”

Error dialog boxes appear during uninstall


Context• Launching uninstall from Add/Remove

Programs

Installation utility continues to function erroneously and will show the following dialog box if an uninstall is attempted:


17 Registry values REPOSITORY_HOSTNAME, REPOSITORY_SERVER, or both, are missing.

1. Exit the uninstall utility.2. Add registry keys (REG_SZ):



5.0REPOSITORY_HOSTNAMEREPOSITORY_PORTNUMBER

a. Open the Registry Editor with: Start, Run, regedit (or regedt32). b. Drill down (as shown above), and select folder (5.0). c. Create new keys (right-click in right-hand panel, New, String Value).3. Set values of new keys: a. Right-click REPOSITORY_HOSTNAME, select Modify; under Value Data

in text box, enter Repository host name. b. For REPOSITORY_PORTNUMBER, enter Oracle port (default = 1521).4. Restart installation.





Table 16 Server-Database Connection Fails

Server does not appear to connect to database

ProcedureInstallation and Configuration GuideUser Guide

Context(Various)


Procedures at right apply to Items 18 and 19.

Preliminary steps:Check for an error message to confirm the database is unavailable:1. Open server.trc 2. Locate the following error message: com.emc.ecc.common.EccuntimeException:Database connection failed

3. If above error message is not found, try any other Suspected Diagnoses available for the same symptoms. Else, if above error message is found, continue with Item 18 and/or 19.

18 Account eccadmin is missing. (User currently logged in does not have sufficient privileges to start the services.)

Perform “Preliminary steps” 1 through 3 above.4. Check that account eccadmin has been created on ECC Server

host. If not, create it.


5. Retry Console login.

19 Database failure. Perform “Preliminary steps” 1 through 3 above.4. Check Services to ensure that at least the following are running (Status

shows Started): OracleECCREP-HOMEAgentOracleECCREP-HOMEDataGathererOracleECCREP-HOMETNSListenerOracleServicerambdb

a. From the Windows desktop, select Start, Settings, Control Panel, Administrative Tools, Services.

b. For any service that is not running, right-click service name, and select Start.

5. Retry Console login.


28



Table 17 Server-Database Connection Dropped

Server connection to database appears to have dropped


Context(Various)


20 Database failure. 1. Check for an error message confirming the database is unavailable, and respond:a. Open server.trc b. Locate the following error message string: DatabaseMonitorPlugin: Database short term failure

2. If above error message is not found, try any other Suspected Diagnoses available for the same symptoms. Else, if the above error message is found:• Check Repository status using ControlCenter System Monitor:

a. Start System Monitor. (See ControlCenter System Monitor on page 47.)

b. Under the General Status (or Server Status) tab, check whether Repository is active or not.

• Alternatively, check status using Task Manager.

3. Check Services to ensure that at least the following are running (Status shows Started):

OracleECCREP-HOMEAgentOracleECCREP-HOMEDataGathererOracleECCREP-HOMETNSListenerOracleServicerambdb

a. From the Windows desktop, select Start, Settings, Control Panel, Administrative Tools, Services.

b. For any service that is not running, right-click service name, and select Start.




Table 18 Store-Database Connection Dropped

Store connection to database appears to have dropped


Context(Various)


21 Connection failure. 1. Check for a database connection error message:a. Open store.trcb. Locate error message: GotSQLException:Io exception:Connection reset by peer:socket write error

2. If above error message is not found, try any other Suspected Diagnoses available for the same symptoms. Else, if the above error message is found:• Check component status using ControlCenter System Monitor:


b. Under the General Status (or Server Status) tab, check whether Store and Repository are active or not.

• Alternatively, check status using Task Manager.

3. From the Windows desktop, select Start, Settings, Control Panel, Administrative Tools, Services.If Store is not running, skip to step 4. Else, if Store is currently running, prepare for a clean restart:a. Locate Service EMC ControlCenter Store.b. Stop Store: Right-click on service name, and select Stop.c. Wait 2 minutes.

4. Start (restart) Store: Right-click service name, and select Start.


30



Table 19 Console Does Not Appear on Startup 1

Console window does not appear


Installing the Console and Client ApplicationsInstalling the Console

ContextWindows desktop: Start ECC Console icon


22 ECC Server is not running. 1. Check that Console logging is enabled. (See Console Logging in EMC ControlCenter Version 5.0.3 Release Notes.)

2. Ping ECC Server host. If host does not respond, try other remedies for same symptoms.

3. Check for an error message:a. Open console.trc b. Locate the following error message:Could not connect to the ECC server; The EMC ControlCenter server on <hostname> may not be ready for logins

4. If above error message is not found, try any other Suspected Diagnoses available for the same symptoms. Else if message is found, check service status:

• Check component status using ControlCenter System Monitor:a. Start System Monitor. (See ControlCenter System Monitor on

page 47.)b. Under the General Status (or Server Status) tab, check

whether server is running (“active”) or not.• Alternatively, check component status using Task Manager.

5. If server is not running, skip to step 6. Else if server is running, prepare for a clean restart:a. From the Windows desktop, select Start, Settings, Control Panel,

Administrative Tools, Services.b. Locate service EMC ControlCenter Server.c. Stop service: Right-click on service name, and select Stop.

6. Restore system:a. Restart server: Right-click service name (see Step 5), and select

Start.b. Restart Console: Click Start ECC Console icon.




Table 20 Console Does Not Appear on Startup 2

Console does not start up


Context• Console startup from ECC Client Install

Configuration window


23 ECC Server is not running. 1. Check server status:• Check component status using ControlCenter System Monitor:


b. Under the General Status (or Server Status) tab, check whether ECC Server is running (“active”) or not.

• Alternatively, check component status using Task Manager.

2. If ECC Server is running, skip to step 3. Else, if ECC Server is not currently running, start it:a. From the Windows desktop, select Start, Settings, Control Panel,

Administrative Tools, Services.b. Locate service EMC ControlCenter Server.c. Start service: Right-click on service name, and select Start.

3. Verify that server.trc is being updated:Locate file detail for server.trc at:

<install_root>\ecc_inf\data\<hostname>\data\

4. If server.trc is being updated, skip to step 5. Else, if server.trc is not being updated:a. Stop ECC Server: From Services window, right-click service name,

and select Stop.b. Open a command window.c. To display any startup errors, start the ECC Server from the

command line, by executing: <install_root>\ecc_inf\data\StartServer.bat

5. Retry Console startup.


32



Table 21 Agent Installation Cannot Be Performed on Console

Console popup dialog box shows:Trouble communicating with master agent:

Cannot assign requested address: connect

ProcedureUser Guide

Context(Various)


24 Master Agent is down. 1. Check Master Agent status at its host.

2. If the Master Agent appears down, restart it:Windows:

Windows Services at the agent host.UNIX:<install_root>/exec/start_master.csh




Table 22 Agent Startup Quits

Agent process startup fails• Fails to establish server connection• Causes printing of occasional error messages• Hangs


Context(Various)


25 No server is running. Start (restart) server:a. From the Windows desktop, select Start, Settings, Control Panel,

Administrative Tools, Servicesb. Right-click Service EMC ControlCenter Server, and select Start.


34


26 No communication channel current between agent and ECC Server.

For items 26 and 27:1. Open <agent_abbrev>_sst.log 2. Locate:

Error caught in SST Function [init], Failed connecting to Host (IP) – Port

3. If error message found, reset port.

27 Wrong ECC Server host port.

28 No Store is running. 1. Open <agent_abbrev>_sst.log file. 2. Locate error message:

Error caught in SST Function [init], Received error from server: Cannot contact store: <store_name>

3. If error message is found, check services status:• Check component status using ControlCenter System Monitor:


b. Under the General Status (or Server Status) tab, check whether Store is running (“active”) or not.

• Alternatively, check component status using Task Manager.

4. If Store is not currently running, continue to step 5. Else if Store is currently running:Prepare for restart: a. From the Windows desktop, select Start, Settings, Control Panel,

Administrative Tools, Servicesb. Right-click service EMC ControlCenter Store, and select Stop.

5. Start (restart) Store: Right-click service name (see step 4), and select Start.

Table 22 Agent Startup Quits (continued)

Agent process startup fails• Fails to establish server connection• Causes printing of occasional error messages• Hangs


Context(Various)





Table 23 Host Icon Does Not Appear in Console

Host instance icon does not appear


Installing and Configuring Open Systems AgentsRemote Install via the Console

ContextConsole tree panel:

AdministrationInstall

Hosts


29 Master Agent is not running. Check that the Master Agent is running. Windows:1. Open Task Manager. 2. Under Processes tab, attempt to locate mstragent.exe UNIX:1. Run ps -ef 2. Attempt to locate mstragent in output.

3. If process name does not appear, Master Agent is not running. Do the following:

Windows:• Restart Master Agent through Windows Services.UNIX:• Run <install_root>/exec/start_master.csh

30 Network error occurred. Check to see if a network error is reaching the ECC Server.Windows, UNIX:1. Open masteragent.log2. Find any entries labeled network error3. Ping IP address of device indicated in error log entry.


36



Table 24 Host Agent Device Icon Does Not Appear in Console

Host agent device icons do not appear



ContextConsole tree panel


31 MSR Agent or MNR Agent is not running. Check whether MNR or MSR Agent is running:Windows:1. Open Task Manager.2. Under Processes tab, attempt to locate mnragent.exe,

msragent.exe

UNIX:1. Run ps -ef 2. Attempt to locate mnragent, msragent in output.

3. If the process does not appear, the agent is not running: Restart the agent.

32 Network error occurred. Check whether a network error is reaching the ECC Server.1. Open mnr_sst.log and msr_sst.log2. Find any entries labeled “network error.”3. Ping the IP address of device indicated in error item to see if it can be

reached.4. If necessary, restart host or check network.

33 Store errors occurred. Check Store log store.trc for any SQL errors.




Table 25 Symmetrix Icon Does Not Appear in Console

Symmetrix icon not found



ContextConsole tree panel


34 Symmetrix agent is not running.(installation problems)

Check that the Storage Agent for Symmetrix is running. Windows:1. Open Task Manager.2. Under Processes tab, attempt to locate egsagent.exe UNIX:1. Run ps -ef2. Attempt to locate egsagent in output.

3. If process does not appear, Symmetrix agent is not running. Restart Symmetrix agent through Console.

35 Network error occurred. Check to see if a network error is reaching the ECC Server.1. Open egs_sst.log2. Find any entries labeled network error.3. Ping IP address of error device to see if it can be reached.4. If necessary, restart host or check network.

36 Store errors. Check Store log store.trc for SQL errors.

37 Other errors. In general, check that logs show:• Symmetrix storage array(s) present and discovered.• AMPRIM message was transferred from server.• Configuration is being sent to STOR.


38



Table 26 Symmetrix Icon Does Not Appear in Console; Agent Error Message

Symmetrix icon does not appear andAgent continues printing error message


Context(Various)


38 1. Open <agent_abbrev>_sst.log2. Locate: EGSAgent::AmPrimary __|Current Symms list but without any Symmetrix storage arrays are listed.

3. If list is missing:a. Wait until connection is established.b. Break connection.




Table 27 Console Error Dialog Box: “Agent not found”, “Communication failed”

Console fails executing commands• Console popup shows:

Agent not found

or Communication failed

ProcedureUser Guide

Context(Various)


39 Master Agent is down. 1. Check Master Agent status using the Console.2. If the Master Agent appears down from the Console, check its status

locally, using the Task Manager at the agent host.3. If the Master Agent locally appears down, restart it from Windows

Services at the agent host.

40 Agent is down. 1. Check Master Agent status, and restart if necessary, according to Item 38.

2. Check agent status using the Console.3. If agent appears down, skip to step 6.4. Check agent logs on agent host for failure indicators.5. If no messages are found in agent logs, try a new Suspected Diagnosis,

or contact EMC.6. Restart agent from Console.


40

Reference

ReferenceThe following tables identify names and locations for specific objects in ControlCenter.

Contents

◆ Agent Names and Codes ................................................................... 41◆ Port Settings......................................................................................... 45◆ Trace and Log Files ............................................................................. 45◆ Error Messages .................................................................................... 47◆ Tools ...................................................................................................... 47


Reference

Agent Names and Codes Three-letter codes are assigned for each of the agents. They are used

as prefix names, and as labels of folders in the ControlCenter directory structure, as well as other reasons. The first and third letters refer to EMC-internal identification, but uniquely identify the agent. The second letter refers to the host operating system, as identified in Table 28.

These codes are listed alphabetically in Table 29 and in Table 30 (the latter ordered by the official “short name,” or a keyword or description in italics).

Agent code names do not always carry over to agent executable names—in some cases, generic agents are used. See Table 28 and Table 29 for full comparison in EMC ControlCenter version 5.0.3.

Table 28 Agent Operating System Indicator

Middle Characterof Agent Code Operating System

A AIX

G (multiple or “generic”)

H HP-UX

M MVS

N Windows NT, Windows 2000

S Solaris

V Novell


42

Reference

Table 29 Agent Codes

Code Agent NameExecutable (if applicable)

CNCCNGCSGEGAEGFEGNEGOEGSENFENNENOENSENWESFESOESSMARMAYMGAMGDMGLMGQMGYMHDMHRMHY

Storage Agent for CelerraIntegration Gateway (Windows)Integration Gateway (Solaris)Connectivity Agent for SNMPConnectivity Agent for SDM (generic)Connectivity Agent for Switches (generic)Database Agent for Oracle (generic)Storage Agent for Symmetrix (generic)Connectivity Agent for SDM (Windows)Connectivity Agent for Switches (Windows)Database Agent for Oracle (Windows)Storage Agent for Symmetrix (Windows)WLA Archiver (Windows)Connectivity Agent for SDM (Solaris)Database Agent for Oracle (Solaris)Storage Agent for Symmetrix (Solaris)Storage Agent for AIXTivoli Storage Manager Agent (AIX)Master Agent (generic)Storage Agent for HDS (generic)Storage Agent for CLARiiON (generic)Storage Agent for Compaq StorageWorks (generic)Backup Agent for TSM (Tivoli Storage Manager) (generic)Storage Agent for HDS (HP-UX)Host Agent for HP-UXTivoli Storage Manager Agent (HP-UX)

cngagent.exe

csgagent

egfagent.exe

egnagent.exe

egoagent.exe

egsagent.exe

enwagent.exe

egfagent

egoagent

egsagent

maragent

mayagent

mhdagent

mhragent

mhyagent


Reference

MM2MMCMMHMMIMMLMMNMMPMMSMMTMMYMNDMNLMNQMNRMNYMSRMSYMVR

Database Agent for DB2Storage Agent for IBM ESSHost Agent for MVS HSMStorage Agent for RVA/SVALogical Agent for MVSMaster Agent (MVS)Physical Agent for MVSHost Agent for MVS SMSTape Agent for MVSBackup Agent for TSM (Tivoli Storage Manager)Storage Agent for HDS (Windows)Storage Agent for CLARiiON (generic)Storage Agent for Compaq StorageWorks (Windows)Host Agent for Windows (Windows)Tivoli Storage Manager Agent (Windows)Host Agent for SolarisTivoli Storage Manager Agent (Solaris)Host Agent for Novell

mndagent.exe

mglagent.exe

mnqagent.exe

mnragent.exe

msragent

msyagent

mvragent.exe

Table 29 Agent Codes (continued)

Code Agent NameExecutable (if applicable)


44

Reference

Table 30 Agent Naming Index

Short Name orIndex-only item Code Full Name

AIXCelerraCLARiiONDB2GatewayHDSHostHostHostHostHP-UXIBM ESSLogicalMasterMasterMVS DB2MVS IBM ESSMVS (Logical)MVS (Physical)MVS (Tape Mgt)MVS HSMMVS SMSNovellOraclePhysicalRVA/SVASDMSNMP ConnectivitySolarisStorageWorksSwitchSYMAPI ServerSymmetrixTape ManagementTSMTSMWLA Archiver

MARCNCMGLMM2CNGMGDMARMHRMSRMNRMHRMMCMMLMGA MMNMM2MMCMMLMMPMMTMMHMMSMVREGOMMPMMIEGFEGAMSRMGQEGN—EGSMMTMGYMMYENW

Host Agent for AIXStorage Agent for CelerraStorage Agent for CLARiiONDatabase Agent for DB2Integration GatewayStorage Agent for HDSHost Agent for AIXHost Agent for HP-UXHost Agent for SolarisHost Agent for WindowsHost Agent for HP-UXStorage Agent for IBM ESSLogical Agent for MVSMaster Agent (Windows, UNIX)Master Agent (MVS)Database Agent for DB2Storage Agent for IBM ESSLogical Agent for MVS Physical Agent for MVS Tape Agent for MVSHost Agent for MVS HSMHost Agent for MVS SMSHost Agent for NovellDatabase Agent for OraclePhysical Agent for MVSStorage Agent for RVA/SVAConnectivity Agent for SDMConnectivity Agent for SNMPHost Agent for SolarisStorage Agent for Compaq StorageWorksConnectivity Agent for SwitchesSYMAPI ServerStorage Agent for Symmetrix Tape Agent for MVSBackup Agent for TSM (Tivoli Storage Manager)Backup Agent for TSM (Tivoli Storage Manager)WLA Archiver


Reference

Port Settings Port settings are requested by the installation wizard, but may be reset. Default listening port settings are given in Table 31.

Trace and Log Files Each component has at least one log file. Depending on its source code origin (see Error Levels), a log can capture up to five or six severity levels.

Error LevelsLog files for the Java-based ECC Server, Store, and Console can be set to capture messages at one of the threshold levels shown in Table 32; while log files for the C++ based agents can be set as in Table 33.

Logging can be adjusted from default levels by consulting with EMC Customer Support.

Table 31 Defalt Listening Port Settings

Component Port Number

Server 5799

Enhydra Server 30000

Repository 1521

Master Agent 5798

Table 32 Logging Error Levels: ECC Server, Store, Console

Level Name and Indication

7 ERROR — failure

6 WARN — warning

5 INFO — information

4 CONFIG — static configuration message

3 FINE — tracing information

2 FINER — detailed tracing information

1 DEBUG — most detailed tracing information


46

Reference

Trace and LogFiles and Locations

Tables 34 and 35 show trace log files available and their locations.

Table 33 Logging Error Levels: Agents

Level Name and Indication

0 Fatal error

1 Warning

2 Information

3 Detailed information

4 Debugging (primarily for development and testing)

5 Debugging (primarily for development and testing)

Table 34 Infrastructure Trace and Log Files on Infrastructure Host

Name Location Description

server.trc <install_root>\ecc_inf\data\<hostname>\data\ Server trace

store.trc <install_root>\ecc_inf\data\<hostname>\data\ Store trace

console.trc %USERPROFILE%\ecc\<infrastructure_host>\ Console trace

consoleLaunch.log %USERPROFILE%\ecc\<infrastructure_host>\data\ Console launch log

snmpcollector.log <install_root>\ecc_inf\data\<hostname>\data\ SNMP collector trace

Table 35 Trace and Log File Paths

Name Location Description

On infrastructure host(s):

masteragent.log <destination_directory>\exec Master Agent log

mga_console <destination_directory>\exec

Example logs:On infrastructure host(s), in agent-specific directories:

egs_sst <destination_directory>\exec\EGS500

EGS500_<year-month-day-time> <destination_directory>\exec\EGS500

mnr_sst <destination_directory>\exec\MNR500

MNR500_<year-month-day-time> <destination_directory>\exec\MNR500


Reference

Error Messages Consult the online error Help for comprehensive error message information.

Online Error Help Each Help message contains the message label, description, reason, and action to take (if any). See Figure 4 on page 47.

Figure 4 Online Help Message Entry

ConsoleTwo types of error messages are presented:

◆ Responses to unauthorized requests to delete a managed object

◆ Symmetrix error alert messages

Tools Software tools to help with ControlCenter troubleshooting are identified below. These packages are available only from EMC Powerlink.

ControlCenter SystemMonitor

This utility, which is installed on an ECC Server host, provides status information for ControlCenter components.

ControlCenter LogViewer

This utility, which runs on Windows platforms, is used to search logs for patterns to assist log analysis, and to zip log files for delivery to Customer Service.


48

Reference

Copyright © 2002 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

EMC and Symmetrix are registered trademarks, and Celerra, ControlCenter and Connectrix are trademarks of EMC Corporation. All other trademarks used herein are the property of their respective owner.


Documents

EMC Control Center Troubleshooting