Logging Wars: A Cross-Product Tech Clash Between Experts

MakeEvery

MomentCount

2016ConnectThe Premier Social Business and Digital Experience Conference

#ibmconnect

Logging Wars: A Cross-Product Tech Clash Between ExpertsBen MenesiChris Miller


Ma

ke Every Mom

ent Coun

t

Acknowledgements and DisclaimersCopyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

IBM, the IBM logo, ibm.com, BrassRing®, Connections™, Domino®, Global Business Services®, Global Technology Services®, SmartCloud®, Social Business®, Kenexa®, Notes®, PartnerWorld®, Prove It!®, PureSystems®, Sametime®, Verse™, Watson™, WebSphere®, Worklight®, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.


Ma

ke Every Mom

ent Coun

t

Speaker• Ben Menesi

§ Head of Product at Ytria§ IBM Notes Administration & Development§ Certified Domino Admin, Dev and Adv.

Security Professional v7+ § Speaker at ICS / ESS tech conferences

around the globe § IBM Champion ‘14 – ‘16


Ma

ke Every Mom

ent Coun

t

Speaker• Chris Miller

§ Director, Collaboration at Connectria§ @IdoNotes on everything§ IBM Champion ’11 – ‘16


Ma

ke Every Mom

ent Coun

t

Questions to ask ourselves

Does Domino always display what is logged?

Does Domino always log what it displays?


Ma

ke Every Mom

ent Coun

t

Agenda

Mail Logging

Usage Activity

Replication Logs

Web Server Logs

DDM vs. Others

Logging Misc.

1.

2.

3.

4.

5.

6.


Ma

ke Every Mom

ent Coun

t

Mail Logging• Daily logging is different than troubleshooting logging

§ The goal is reduce logged data and utilize built in tools• MTC can be your friend across a domain for daily logging

issues


Ma

ke Every Mom

ent Coun

t

Mail Logging• Other alternatives to MTC:

§ Move4Ideas Audit4Mail: Server task, intercepts all mails via HTTP, Router, SMTP or AMGR

• Can show custom items, keeps light headers for longer time and accessible by helpdesk, without the Domino Administrator client

§ GSX Mail Monitor


Ma

ke Every Mom

ent Coun

t

Mail Tools• Mail File Digest - summary report for mailfiles and mail-in

databases from OpenNTF§ http://idonot.es/mailfiledigesttool

• Server add-on, requires downtime to get done, plus only supports 8.0.1, 8.5.2 and 9

• Implement Converter_Log_Level=10§ Removes tons of unnecessary log entries on mail conversion

• Log_MailRouting=0§ Yes this is an actual value when used with one we discuss later


Ma

ke Every Mom

ent Coun

t

Bonus Tip: One remedy for all mail rule issues• Mail rules are frequent troublemakers for Administrators

§ Let’s understand their architecture a bit better§ Tip: If you ever struggle with Mail Rules, just create a new one.

Mail Rule 1 CalendarProfile

$Filterformula

$Actions

$Conditions

Ordernum

$Filterformula_1

§ More info: http://techlab.ytria.com/5243/ibm-notes-secrets/mail-rules-architecture-sort-scanez


Ma

ke Every Mom

ent Coun

t

User Activity• Who’s accessing what?

§ Increasingly important question whether you’re:• Upgrading / consolidating your environment

• Considering going to the cloud • In the process of migrating

§ What data to use and HOW to use it• Frequently used databases and apps that aren’t used at all?


Ma

ke Every Mom

ent Coun

t

User Activity• Possible Data Sources

Activity Trends

Database User Activity

Log.nsfUser Activity

3rd Party Solutions

§ Valuable high level reports

§ Not very customizable§ Lack of customization

leads to inaccuracies

§ Very detailed logs § Really crucial info

buried inside one text field

§ Unless through 3rd

party tools: NONO

§ Great amount of detail § Only available per

database§ To be discussed later

§ Amazing products out there

§ Some installed on server side

§ Some client side§ To be discussed later


Ma

ke Every Mom

ent Coun

t

User Activity• Activity Trends – How does it work?

Log.nsf Catalog.nsf Names.nsf Statistics

Activity Trend Collector

Activity.nsf


Ma

ke Every Mom

ent Coun

t

User Activity• Activity Trends – Why is it good?

§ Incorporates valuable data and provides high level reports on: • Databases usage (Active / Inactive, space, growth,

• Users (Active users, connection times, databases used, R/W counts, Data sent and received from server)

• Server tasks (High level reports on HTTP, IMAP, LDAP, POP3, Replication etc…) According to configuration

• Server Disk Space (Usage, growth, etc...) • Server Statistics

§ Tip: Use the Domino Administrator client to visualize data:


Ma

ke Every Mom

ent Coun

t

User Activity• Activity Trends – Why is it good?

§ Interactive Analysis in the Domino Administrator client


Ma

ke Every Mom

ent Coun

t

User Activity• Activity Trends – Cons

§ Summary data reports: you can’t really dig deeper § Reports aren’t customizable

– E.g. Database marked ‘Active’ – yet only a Server and an Administrator has used it within the past 3 months

§ Does not properly calculate Space Used Percentages


Ma

ke Every Mom

ent Coun

t

User Activity• Log.nsf

§ There is a TON of useful, raw data captured in User Activity Documents

§ Yet, using log.nsf for User Activity Analysis is the common Enemy § What is actually recorded in a User Activity Event?

Date & Time Reads Writes User

Bytes Read Bytes Written DB Open Time


Ma

ke Every Mom

ent Coun

t

User Activity• Log.nsf – What is recorded?


Bytes Read Bytes Written DB Open Time


Ma

ke Every Mom

ent Coun

t

User Activity• Recorded User Activity

§ User Activity can be a *VERY* rich source of data (incl. HTTP!)§ Tip: No_Force_Activity_Logging=0 (Default value) allows on ALL

dbs§ Tip: Set confidential to prevent users with lower than Designer

access


Ma

ke Every Mom

ent Coun

t

User Activity• Prior to ODS 48

§ Entry size allocated 44 bytes, object size 61600 bytes (max 1400)

• ODS 48+ (undocumented improvement) § Entry size 92 bytes. Object size 128800 bytes (same max 1400)

§ Curious: only two additional counters added, yet ODS48+ entry size is over 2x the size of ODS48 – why?


Date & Time Reads Writes Updates Deletes User


Ma

ke Every Mom

ent Coun

t

User Activity• Does Notes only capture what it displays?

§ Nope

§ Read more: http://techlab.ytria.com/5448/lotus-notes-articles/interpret-user-activity-traces-portrait-global-usage/

Reads Adds Updates Deletes

Data

NonData

Σ

Data

NonData

Σ

Data

NonData

Σ

Data

NonData

Σ


Ma

ke Every Mom

ent Coun

t

User Activity • Cons

§ Only available per database • You have free tools available to facilitate working with this data

across multiple databases: http://www.agecom.com.au/useractivity– Or implement via W32_NSFDbGetUserActivity

– Or get Ytria’s databaseEZ

§ Makes no difference between users and servers • Filtering this info is easy in tools like databaseEZ

§ Only maintains 1400 entries• Funny you should say that…


Ma

ke Every Mom

ent Coun

t

User Activity • Deriving usage frequency from User Activity

§ Database A is more frequently used than Database B

Database A

Entry Date

Entry Date#1400

#1

75 days

Database B

Entry Date

Entry Date

120 days

#1400

#1


Ma

ke Every Mom

ent Coun

t

User Activity• 3rd Party products for usage analysis

§ Domino Doublecheck• http://goo.gl/maL0fo

§ Panagenda iDNA• https://www.panagenda.com/products/idna

§ Ytria databaseEZ & consoleEZ• http://ytria.com/databaseez• http://ytria.com/consoleEZ


Ma

ke Every Mom

ent Coun

t

Replication Logging• Replication is a subject that raises various questions such as

§ How do I make sure replication runs smoothly? § What do I do when problems occur (and problems do occur)§ The log file and log_replication is pretty useless


Ma

ke Every Mom

ent Coun

t

Replication Logging• Various Replication diagnostic tools are available such as the

Log_Replication notes.ini parameter. § What if we want to know what exactly happens during replication?

• Set Log_Replication=3 to get information about each replicated note:


Ma

ke Every Mom

ent Coun

t

Replication Logging• Can we go even higher in Log_Replication?

§ Why not go to 4• This will show detailed logging information about each replicated

field

§ Or how about 5• Adds some additional logging summary

• http://www-01.ibm.com/support/docview.wss?uid=swg21214739


Ma

ke Every Mom

ent Coun

t

Replication Logging• What about Cluster Replication?

§ Cluster Replication isn’t always logged (Check initiatedby item on replication logs)

• Use RTR_Logging (manage information written into log.nsf) – 1 – Default

– 2 – All context structure changes

– 4 – Log Attempted & Performed replications

– 8 – Log iterations through main polling loop

– 16 – Verbose debug logging

– 32 – Log all lock operations


Ma

ke Every Mom

ent Coun

t

Replication Logging• Replication Logs in your Log.nsf

§ There’s more data in there than you might see • Categorize your Replication Logs view by “Initiatedby” to differentiate

between replication events carried out by the Replicator and Cluster Replicator.

• Use a formula @IsAvailable(“Eventlist”) to display exclamation mark next to events, to instantly see replication events that need attention

• Use @Elements(Pathname) to get the number of databases (and directions) replicated per session.

– Makes it easy to map replication events to connection documents!


Ma

ke Every Mom

ent Coun

t

Replication Logging• Improved Replication Logs view

§ Let’s take a look


Ma

ke Every Mom

ent Coun

t

Web Server Logging• Filtering and excluding is a key to reducing log clutter

§ Server document -> Internet Protocols -> HTTP -> Exclude

§ By listing file extensions (*.gif) n URL request they are removed§ Use full MIME type listings to remove image logging

• Content-Type: image/gif


Ma

ke Every Mom

ent Coun

t

Web Server Logging• Consolidating HTTP & Miscellaneous Logs

§ Use the HTTPLogUnauthorized=1 Notes.ini Parameter • Logs 401 HTTP Errors in console AND Misc. log events

• HTTP Users attempting to access resources they aren’t entitled to access + user authentication requests.


Ma

ke Every Mom

ent Coun

t

DDM vs. Other methods• DDM is about building proper triggers for logged and non-

logged data§ DDM takes effort to configure properly§ Domino exposes all errors but does not technically log them


Ma

ke Every Mom

ent Coun

t

DDM vs. Other methods• Customer Example

§ DDM constantly communicates disk space low § Customer used to running low on storage § 90% of databases with <25%§ Discovered Program doc.

turned off in 2009


Ma

ke Every Mom

ent Coun

t

DDM vs. Other methods• How to extract more from Log.nsf?

§ There is MUCH more data in your logs than you know• Each event comes with a status code, severity, server add-in

§ Extract more data from your log documents using formulas such as:

• # of events, unique events

• Highest severity event encountered • # of various severity events• Total # of messages transferred per each Mail Routing document


Ma

ke Every Mom

ent Coun

t

DDM vs. Other methods• How to extract more from Log.nsf?

§ More Info: More info: http://www.slideshare.net/BenedekMenesi/ibm-connected-2015


Ma

ke Every Mom

ent Coun

t

DDM and Stat Tools• Andy Pedisich statrep.nsf customization

§ http://www.andypedisich.com/blogs/andysblog.nsf/dx/resources.htm


Ma

ke Every Mom

ent Coun

t

Agent Logging• Debugging Agents through Print statements is BAD

§ How can we log and control our agents better? • OpenLog on OpenNTF

§ https://www.openntf.org/main.nsf/project.xsp?r=project/OpenLog• Thomas Lindberg Agent Controller (OpenNTF)

§ https://www.openntf.org/main.nsf/project.xsp?r=project/AgentController

• AgentEZ§ http://www.ytria.com/agentEZ


Ma

ke Every Mom

ent Coun

t

Logging Misc. • Clean up your logs even more • Log_disable_session_info=1

§ This will remove when sessions are closed§ It is on by default on all servers

• Log_sessions=0§ Do we really need to know when a user has a session?§ This works in conjunction with Log_MailRouting=0 or alone


Ma

ke Every Mom

ent Coun

t

Logging Misc. • Log filtering

§ This is built into all your Domino servers in events4

§ See Chris' article http://idonot.es/filterlognsf


Ma

ke Every Mom

ent Coun

t

Logging Misc. • Log File Retention

§ Default Deletion time specified via “Log=“ Notes.ini Parameter• Don’t use this!

§ Use “Remove Documents not modified in this file” § Use Archive Profile(s) to split into multiple databases § Use console logs to retain old log data

• Log Mirroring, Mirror retention

§ Conclusion


Ma

ke Every Mom

ent Coun

t

Thank you• Thank you for attending!

§ We’d love to hear from you

ca.linkedin.com/in/benedekmenesi

Twitter.com/BenMenesiTwitter.com/Idonotes

Ben Menesi

Linkedin.com/in/idonotes

Chris Miller

Software

Logging Wars: A Cross-Product Tech Clash Between Experts