Admin Aspirin — Reduce Performance Headaches Through Proper Domain Monitoring

© 2010 Wellesley Information Services. All rights reserved.

Admin Aspirin — Reduce Performance Headaches Through Proper Domain MonitoringAndy PedisichTechnotics

2

In This Session ...

• Let’s talk about your servers Are they struggling to keep up or not even breaking a sweat? Do you have too many users on a server or could you easily

consolidate the ones you have? Are they having problems, but you’re just waiting for a help

desk ticket to come in before you find out what’s wrong?• This session takes you behind the scenes to show you how you

can be a more powerful administrator using data that’s always there, but a little hard to find

3

What We’ll Cover …

• Understanding the fundamentals of statistic collection• Exploring the native Notes monitoring system• Taking advantage of server activity logging and trending• Exposing message tracking’s capabilities• Using the Domino Configuration Tuner• Dominating your domain with DDM• Customizing STATREP.NSF to pull out hidden data• Analyzing statistics for proactive problem solving• 10 statistics to monitor for top performance and stability• Wrap-up

4

The Two Things Needed

• Two things are required for statistics collection: The Collect task must be running on any server that is

designated to collect the statistics Not all servers should run the Collect task

The EVENTS4 database must have at least one Statistics Collection document Statistics should be collected centrally on one or two servers

so that the data is easy to get to Stats should be collected every hour to be effective

EVENTS4 should be the same replica on all servers in the domain

5

We Know What the Replica ID Should Be for EVENTS4

• Replica ID of system databases, such as EVENTS4.NSF, are derived from the replica ID of the address book

Database Replica ID

NAMES.NSF 852564AC:004EBCCF

CATALOG.NSF 852564AC:014EBCCF

EVENTS4.NSF 852564AC:024EBCCF

ADMIN4.NSF 852564AC:034EBCCF Notice that the first two numbers after the colon for the

EVENTS4.NSF replica are 02 Make sure that EVENTS4.NSF is the same replica ID

throughout the domain by opening it and putting it on your desktop

6

Want to Add Every EVENTS4.NSF to Your Desktop?

• Add this code to a button on your toolbar This is courtesy of Thomas Bahn

www.assono.de/blog• This code will prompt you to pick the servers that have the

database you want on your desktop Then it will prompt for the name of the database

And open it on all the servers you’ve selected

_names := @Subset(@MailDbName; 1) : "names.nsf"; _servers := @PickList([Custom]; _names; "Servers"; "Select servers"; "Select servers to add database from"; 3); _db := @Prompt([OkCancelEdit]; "Enter database"; "Enter the file name and path of the database to add."; "log.nsf");@For( n := 1; n <= @Elements(_servers); n := n + 1;@Command([AddDatabase]; _servers[n] : _db) )

7

A Required Design, but No Required Name

• There has to be a STATREP.NSF on every server It is used by the server to store monitoring data

It must be designed using the Statrep5.ntf Monitoring Results template Its default title is Monitoring Results

• But you don’t have to use one of those for your statistic collection repository Create your own collection points and give the database a

unique name

City Collecting Server Monitoring Results DB New York USNYAdmin1 USStatrep.nsf

Amsterdam EUNeHub01 EUStatrep.nsf

8



9

Event Monitoring Details

• Event monitors are set in the EVENTS4 database• Event generators

Specify what built-in event shouldbe monitored

• Event handlers Specify the action that Domino

takes when an event generator occurs• Here’s the high-level concept of monitoring:

Setup an event generator to capture an important moment Use an event handler to be notified about it

You can also use event handlers to send you a notification of anything that appears on the console or in a LOG.NSF

10

Event Generators Are Your Friends

• Event generators are pre-set groups of categories to monitor• Some of the more useful are:

Database generators Triggers when the ACL in the address book is changed

Statistical generators Trigger for low disk space, too many messages waiting

Domino server response generator Triggers when servers are not reachable

Task status generator Triggers when HTTP task is down

11

An Example: Database Event Generator

• The ACL of NAMES.NSF should be monitored for changes in every Notes domain Once properly set, the ACL of NAMES.NSF should rarely

change! All kinds of bells and whistles should go off when it does

We’ll talk about notification in a moment Here’s how to set up the monitoring of the ACL

Select New Database Event Generator

12

Setting Up the Database Generator

• Select Names.nsf You can choose either a single

server, such as the administration server for the address book, OR

All servers in the domain• I like to pick all servers in the domain

Admins won’t get away with anything!

But I do get a storm of messages when an ACL change occurs Every server tells me about

the change

13

Another Event Generator Example

• Domino Server Response Event Generator Checks connectivity/port status of server’s network

• One server checks others by sending a probe It’s a good idea to try opening Names.nsf

If you can’t open Names.nsf, then something is wrong!

14

Event Handlers Set Up Notification

• When configuring an event generator, you can click the button on the final tab for the Event Handler Wizard Simple to do and guarantees it will work the way you want it to The wizard will walk you through the process of creating an

event notification method for the event generator you just made

15



16

Turning on Activity Logging

• Activity logging is switched on in server configuration documents Select all the server tasks you would like to use to produce

activity logging data If you want to see activity trends, enable all tasks except

Domino.MAIL At a minimum, you must enable Domino.Notes.Session

and Domino.Notes.Database

17

Check Points and Prime Shift

• Leave the default checkpoint interval at 15 minutes It seems to be sufficient

• We’re surprised some of these are checkboxes since you really need them all for activity logging Log checkpoint at midnight is required for both activity logging

and activity trends Log checkpoint for prime shift is required for activity trends

Enter whatever you consider as your prime shift• And voila! Activity logging is started!

18

Activity Logging

• Activity logging causes Domino to write information to the LOG.NSF about the tasks you are interested in Expect the log to grow to about 2 GB The data is kept in a few hidden views in a form that is not

meant for humans to see IT is waiting there for us to access in two ways:

Activity analysis Activity trending

19

Activity Trends Are Very Useful

• You can enable the activity trends collector in the same configuration document that you used to set up activity logging If you activate it in the default config document it creates an

ACTIVITY.NSF database on every server in the domain Then it collects data from the log every night at 3:23 AM

You can run the task manually at the console using Load trends

20

Activity Data Rocks!

• Activity trending gives you an unbelievable amount of data about your environment such as: How your protocols are used How your servers are utilized during prime time Growth rates for databases

21

Inactivity Is Important Too

• Inactivity is also tracked Lets you rid your systems of dead mail files and applications

that are truly no longer being used A little bit of investigation using Activity.nsf will probably

save you gigabytes of disk space And you won’t be wasting time backing up obsolete

databases

22

But Wait, There’s More …

• Activity trending also provides information about user activity It’s probably more information than you need for everyday

administration But it’s great for capacity planning or when you have a server

consolidation to accomplish

23

Catalog Must Be Enabled

• Make sure there are no errors when the activity trends are rolled out Check the Run Log for details about the collection process

• The catalog task must run on each server you want to include in activity trending You’ll see an error message in the Run Log if there is no

catalog on the server

24

Demo

25



26

Message Tracking

• Allows administrators to track all messages Users can track their own messages

• Shows delivery status and current disposition Read/Unread, Deleted

• Configurable Track message subject Don’t track messages from certain users

• Great tool to use when users complain of “lost” email!

27

Enabling Message Tracking

• From the Server Configuration document Configuration tab in the Administration client Go to the Router/SMTP – Message Tracking tab Complete fields as needed

• Tracking Interval controls how often data is written to the message tracking database The default interval is 15 minutes, which is a great place to start

28

Access to Message Tracking

• The Access Settings fields are used to limit who can track other users’ messages Include administrators in this field

Maybe Help Desk?• Make sure that LocalDomainServers is included if you need to

track across servers

Message Tracking Reports

• Daily/weekly/monthly options for all reports: Message volume Largest messages Top 25 next hops Top senders and receivers

By count By size

Message status Delivered/not delivered/in

process

29

30

Caution Using MTC on iSeries

• Servers have seen the MTC task take up large amounts of CPU Up to >50% on iSeries systems Appears to be related to full text indexing of the MTSTORE.NSF

• To remediate this problem Stop MTC task Delete and recreate FTI on mtdata/mtstore.nsf Start MTC task

This can be done on all platforms I’ve seen the problem on Wintel and UNIX

31



32

Domino Configuration Tuner

• Domino Configuration Tuner (DCT) produces a list of items scanned and provides complete explanations of each item It’s a fast and easy way to take stock of your domain

33

Domino Configuration Tuner (cont.)

• Domino Configuration Tuner (DCT) compares your server settings to an IBM catalog of Best Practices DCT looks at settings in the Domino Server documents, the

NOTES.INI file, and advanced database properties It also looks for worst practices

All servers in your domain can be evaluated at once

34

Batteries Not Included

• The DCT is not included with the Domino install files• You can download the latest and greatest version here:

http://www-01.ibm.com/support/docview.wss?rs=463&uid=swg24019358

• IBM updates the design and rules that the application uses on a regular basis Make sure you use the Check for Updates before you scan

your systems

35

Run Scan and Wait for the Results

• Clicking on Run New Scan will start the tuner which will access your domain’s address book to pull out a list of servers Select the servers you want the tool to scan

You’ll also have the opportunity to manually add servers and optionally create a name for the scanning

This is a very valuable free asset from IBM

36



37

DDM Details

• DDM provides a single location where administrators can access issues that are affecting multiple servers and databases Reduces the risk of unplanned downtime

• The DDM database is the central repository of all monitoring data This can be data collected by probes that you can configure Result messages from event generators that you configured in

previous releases of Notes Results for routine checks that run as part of specific server

tasks, such as the router or replicator

38

Configure DDM for Centralized Data Collection

• DDM.NSF has the most value when it is the central repository for all issues It will contain all of the issues that come from all of the servers

• There is no collection hierarchy set up by default If your collection hierarchy looks like the screen shot below, the

collection hierarchy has not been configured yet

39

Aggregate Data Centrally

• A DDM server collection hierarchy lets you aggregate the data onto a key server or servers This must be configured in the EVENTS4.NSF

• The simplest hierarchy is to configure one server to collect from all servers in the domain

40

Rolling Up the Data

• DDM data roll-up propagates the probe results up the DDM server collection hierarchy Data roll-up is accomplished using Domino’s selective

replication to transport the data The replication formulas are created automatically when you

define your DDM server collection hierarchy• Collection replication occurs about every five minutes

It’s hard coded and can’t be changed

41

Address Issues by Severity Level

• Looking at issues by severity gives you the chance to deal with the most important issues first They are broken out by severity category

42

Looking at a Failure

• This simple event in DDM tells us what the problem was and when it occurred And it offers a possible solution

This example shows the basics of DDM

43

Analysis Probes

• Configure DDM using probe documents in EVENTS4.NSF Otherwise known as the Monitoring Configuration database

• A probe is a check, or set of checks, configured to run against one or more servers, databases, and services A probe returns status and results to the Domino Domain

Monitoring Database – DDM.NSF The DDM.NSF is created automatically when an R7/R8 server

starts You can create multiple probes for each feature area

• Just remember that you don’t need probes to use DDM!

44

What’s in These Probe Documents?

• Probe type and probe subtype For example, Security is a probe type

One of its probe subtypes is Best Practices This combination of probe type and probe subtype creates

a Security probe

45

Extra Information About the Probe Is Provided

• These probe documents also contain a general description of the probe, its purpose, and its intended use

46

Demo of Domino Domain Monitoring

47

Plenty of Cool Probes

• R8 gives us 58 default DDM probes to work with R7 gives us 48 – still plenty to get us started

• You can get probing as soon as R7/R8 is up• Just plug in your server info to get DDM started

You can also create new probe documents Define and customize your own probes

48

Zeroing in on Probes

• We’re going to focus on three probes that have a high value in almost every Domino domain: Application probes Web probes Security probes

49

Agents Are Tracked by Application Probes

• Application probes monitor agents None of these probes can be scheduled

They are all real-time monitors• Application probes have the following subtypes:

Agents behind schedule Agents ranked by CPU usage Agents ranked by memory usage Long-running agents

50

Web Probes

• Web probes provide results as a Domino Event document in the DDM database

• There are two subtypes: Configuration probe

Verifies the accuracy of a Web server’s configuration Best Practices probe

Compares servers’ configurations against a known good server

• The default schedule for these probes is once a month

51

The Best Practice DDM Report

• A DDM document details the values that don’t match the Best Practice values recommended by Lotus And provides information on how to resolve any issues

Could be corrections to server functionality, server performance, or server security

This helps improve overall security and Web performance

52

The Five Security Check Probes

• Best Practices Compares your settings to Lotus recommended settings

• Configuration Compares settings to those in a specified “good” Server doc

• Database ACL Monitors access that groups and individuals have

• Database Review Reviews the security properties for a specified database

• Security Review Generates a report on the security settings specified in the

Specifics tab of the probe document

53

The DDM Database Has More Than What’s on the Surface

• The DDM DB lets you review probable causes and possible solutions for reported DDM events And it lets you assign them to an administrator for resolution You can open a link to the appropriate database from which

you can resolve a reported event

54

Explaining the Issue

• In some cases, the event will give details on the problem and provide a link to help you get to the place where the problem occurred One document might represent several occurrences of the

same event

55

Resolving Problems

• In some cases a Choose Solution button is available Use it to help you automatically implement a particular solution

to resolve an event

56

Using the Assign Button

• You can assign the event to a team member and add comments about the task using the Assign button Or you can simply assign the event to yourself

57



58

Why Would the STATREP.NSF Need Customization?

• STATREP.NSF database is surprisingly lacking in value• There are eight views of statistical data

Calendaring & Scheduling and Clusters views lack any substantial information

The Communications view strictly shows dial-up stats Fortunately we can add views to help us significantly

59

Everything Is Everywhere

• Keep in mind that every single statistic that is generated is contained in every document in the Monitoring Results database You just can’t see it all because it’s not in views or documents

And views are the most important place to have it because that’s where it gives you the ability to compare samples And analyze trends

60

Agents No Longer Have Secrets

• Our modified AD2K10STATREP.NTF template has lots of new views to keep you informed

• Are you wondering how many minutes the agent manager used running resource-hungry agents? Now you have a view to help you see it hour by hour

No more guesswork about server conditions

61

Caveats About the Admin 2010 version of STATREP.NSF

• Some views expose the following statistics Agent.Daily.UsedRunTime and Agent.Hourly.UsedRunTime

This stat generated the agent runs in seconds• Some versions of Domino produce this stat as a text field, others

as a numeric field A formula converts it to a numeric field

This might not be necessary in your domain @If(@IsAvailable(Agent.Hourly.UsedRunTime);

(@TextToNumber(@LeftBack(Agent.Hourly.UsedRunTime;8))/60);“N/A”)

The formula also converts the statistic from seconds to minutes

62

One More Caveat

• A few views display disk utilization statistics such as: Platform.LogicalDisk.2.AvgQueueLen.Avg

• Disk statistic names vary from platform to platform AIX and iSeries systems can have much longer device names Even in the Wintel platform they can be listed as:

Logical disks Physical disks

• Be sure to check to see how it is represented in your domain You might find it necessary to customize all disk views for your

own environment

63



64

Transactions Per Hour and Concurrent Users Per Hour

• These can be a very important statistics if you are thinking about consolidation Our customized Monitoring Results database lets you export

this and other data for analysis in your favorite spreadsheet• The idea is to compare your servers statistically

To find out: How many users are hitting these systems concurrently? How many transactions are these servers turning over?

• We want to want these things on a PER HOUR basis IBM’s STATREP.NSF doesn’t have this info, it must be extracted

from the cumulative statistics

65

Start by Going to the Export Transactions/Users View

• Analysis starts with Export Transactions/Users view Do not hesitate to add new views to Monitoring Results

I don’t change the old ones, I just add new ones• Note that Server.Trans.Total is a cumulative stat

And the Trans/Hour column is blank We have a custom formula to apply to this column after the

data is exported into the spreadsheet you use

66

Next, Export View to CSV File

• Export the contents of the view to a CSV file Call the file is C:\delme.csv so you can find it quickly

It’s overwritten each time I do an export Make sure you include the View titles

The import is fast

67

Next, Open the Special Spreadsheet

• Start your spreadsheet (in this case it’s Microsoft Excel) and open the spreadsheet on the CD Admin2010MasterFormulaStatExports.xls

• The spreadsheet contains the formulas that will help to break down server activity into per hour averages Don’t worry about the #value errors

Then open the C:\delme.csv file

68

We’re into MS Excel for the Analysis

• Next, we open the C:\delme.csv in Excel Excel knows we want to import it because it’s a CSV file

It opens quickly with no further prompts

69

The Data Is Now in Excel

• The view brought it in sorted by Server and Collection Time Remember, we’d like to see the number of transactions

per hour With the way this spreadsheet is set up, it’s pretty easy to

construct a formula where we simply: Subtract the last hour’s number of transactions from this

hour’s transactions to get the number per hour

70

Tricky Calculations – Server Restarts and Stuff

• Except sometimes when servers are restarted Then the cumulative stats start over

• Or when the next server starts being listed in the statistics You have to be careful not to subtract without paying attention

to these things

71

Special Formulas to the Rescue

• To cope with the anomalies in the way the data is listed, I built a few fairly straightforward formulas you can use on your spreadsheets They are in the master formula spreadsheet

Just copy it from the cell

72

Insert the Copied Cells

• Move to the delme.csv spreadsheet• Then use the Insert menu to insert the copied cells into your

spreadsheet Move the cells to the right or down to get them out of the way

You’ll be copying the proper formula into your spreadsheet• Copy that formula down your entire column of data

Save your spreadsheet as an XLS

73

Create a Pivot Table or Its Equivalent

• We’re going to make a Pivot Table with our data The Pivot Table will take our data and let us easily manipulate it

and graph it Select all the data, including the column titles, and use the

menu to select PivotTable and PivotChart Report

74

Take Defaults

• If you’re new at this, just take the default answers for the questions Excel asks

75

Drag Server to the Column Tops

• It drops you into the Pivot Table function where you have a field list to drag and drop into the table Drag Server to the column top and Hour to the row

names column

76

Drag the Data to the Center of the Table

• Drag the transaction per hour data to the table It defaults to the “Count of Trans/Hour”

But you’ll want to change it to Average, and format it to look nice too

77

There You Have It

• You now have a nice breakdown of the average number of transactions per hour, per server

78

Graphing Your Results

• This is where it really gets cool Just click on the Chart Wizard

And …

79

Bingo, You Have an Instant Chart

• Stacked bar isn’t what we want, but that was quick!

80

Line Graph Coming

• Use the icon on the right to change graph types A line graph is quite effective, most of the time

81

Here’s the Line Graph You Ordered

• Simple, fast, and straightforward This is an average of transactions per hour

82

Average Number of Concurrent Users/Hour

• This is an extremely valuable stat, especially when consolidating However, when a user connects, they stay connected

And are not dropped until they are inactive for four hours This makes it impossible to track actual concurrency

• Use this NOTES.INI server parameter to prevent idle sessions from taking up valuable resources Server_Session_Timeout = 30

• Sets number of minutes of inactivity after which a server automatically terminates network and mobile connections

Users will not have to re-enter a password if they become active after the time limit

• The minimum recommended setting is 30-45 minutes

83

Change the Field List Easily

• It’s easy to remove the field Trans/Hour off the chart and replace it with the Average of Users

84

Demonstration: Exporting Data and Creating Pivot Table

85



86

Disk Statistics

• Disk.D.Free Track whatever drive holds your data Monitor based on a percentage or an actual value

Depends on the size of the drive Larger drives are sometimes better to track by actual

space left rather than a percentage because the percentage you’d want free is very small

87

Critical Mail Statistics

• Mail.dead Mail that cannot be delivered and cannot be returned to

the sender Watch for greater than 100

• Mail.Waiting Number of outgoing mail messages waiting to be either

delivered locally or transferred in MAIL.BOX Useful for detecting problems with the mail Router

Depends on the server SMTP server might watch for > 200 NRPC mail server might watch for > 50

88

Critical Mail Statistics (cont.)

• Mail.TotalPending May be more reliable than Mail.Waiting which depends on the

router task If the router is not running, Mail.Waiting is not updated

The Mail.TotalPending count includes both dead and held mail The Mail.Waiting figure does not include these items

Updated by the server task to reflect the current number of messages that are pending delivery

It is updated by the server task every five minutes and its value therefore becomes “stale” Threshold depends on type of server

SMTP threshold will again be higher than NRPC server

89

Critical Mail.Box Statistics

• Mail.Mailbox.Accesses Total number of times that threads accessed any mailbox on

the server• Mail.Mailbox.AccessConflicts

Total number of times a thread attempting to access a mailbox had to wait because the number of concurrent threads exceeded the number of mailboxes configured

(Mail.Mailbox.AccessConflicts / Mail.Mailbox.Accesses) x 100 Should be less than 2

90

Platform Disk Statistics

• The disk specification will vary by server• Platform.LogicalDisk.1.AvgQueueLen

AvgQueueLen the average number of both read and write requests that were queued for all logical disks on all physical disks during the sample interval Should not consistently rise above 2

• Platform.LogicalDisk.1.PctUtil PctUtil percent of time the drives are busy reading or writing

Watch for disks constantly hitting above 80% • Track both of these statistics in Notes

Follow up with performance monitoring on the OS level

91

Important Cluster Statistics

• Replica.Cluster.SecondsOnQueue The number of seconds that the last database spent waiting in

the replication queue before being replicated Generally this should be under 20

• Replica.Cluster.WorkQueueDepth Number of items in the replication queue

Should be under 10• We’ll talk more about this during the session on Domino

clustering

92

Bonus 11th Statistic for Transaction Logging

• Database.RM.SinceStartup.Critical.Log.Times Shows how many times your system ran critically log full

Should be zero If non-zero, consider increasing log size or number of times

that Domino filled the allocated log space If greater than zero, consider increasing log size If circular and >0, the system has a long-running transaction

If this number creeps into the hundreds, your server is probably going to crash soon

93



94

Resources

• Description of HTTP statistics for a Lotus Domino server www-1.ibm.com/support/docview.wss?uid=swg21207314

• How is the mail delivery latency formula calculated? http://www-01.ibm.com/support/docview.wss?uid=swg21279936

• Download location for Domino Configuration Tuner http://www-01.ibm.com/support/docview.wss?rs=463&

uid=swg24019358• Lotus Education on Demand: Domino Domain Monitoring (DDM)

http://www-01.ibm.com/support/docview.wss?uid=swg27007060

95

7 Key Points to Take Home

• Make sure all EVENTS4.NSF are the same replica ID• Use activity trending to build ACTIVITY.NSF and discover

databases that aren’t being used• Use the Domino Configuration Tuner to discover settings that

improve performance and security• Create a collection hierarchy for DDM in the EVENTS4.NSF

Monitoring Configuration database• You don’t have to run probes to take advantage of Domino

Domain Monitoring• Add views to your Monitoring Results database to pull out hidden

information• Use SERVER_SESSION_TIMEOUT=30 to ensure idle users don’t

take up valuable server resources

96

Your Turn!

How to contact me:Andy Pedisich

[email protected]