Unicenter Automatic Duplicate Suppression EvtADS -EvtAds r11 -Field Developed Utility -January 17 th 2006

Unicenter Automatic Duplicate Suppression

EvtADS- EvtAds r11

- Field Developed Utility- January 17th 2006

2 © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Disclaimer

This is a field developed utility targeted to be included in the future release for NSM. It is not supported by the CA Support organisation.


Acknowledgments

- Kevin Higgins - Cap Gemini UK plc

Introduction


What is EvtAds?

- A simple but function rich utility to intelligently manage duplicate events to aid in auto-ticketing and eliminate opening multiple problem tickets for the same problem.

- Automatic Duplicate Suppression

- Tool for Windows Event Management


Problem Summary

- Unicenter message processing repeats messages on the operator console unless explicitly discarded.

- Such duplication may prevent use of auto-ticketing or occasionally cause acute message storms, (where too many messages per second are generated).

- Message storm can result into denial of service if extensive instrumentation is carried out to address duplicates.


Automatic Duplication Suppression- EvtAds is Unicenter Event Management Anti-Duplication Suppression driven by Event Message Actions.

- It is specifically designed for scalability to handle large message storms without any performance degradation.

- In a lab environment, it has successfully processed message storm of over a million events


Automatic Duplication Suppression - ADS is called using EXTERNAL Message action and determines if the event has been previously seen within the reset interval. It returns with an &rc indicating the number of times an event / alert has been previously seen.

- Analysing &RC value assists in auto-ticketing and prevents multiple tickets been raised for the same problem.

- Provides means of escalation based on the number of alerts received for the same problem. For example, if 15 failure events are received within reset interval period for the same problem, it would make sense to escalate the problem to emphasis the impact of the problem. Without ADS, this may result in 15 problem tickets being raised


Why EvtAds?

- Unicenter Event Management provides frequency based message rules. This is based on Counter and Interval.

- Frequency based event rules are driven by message records and may require complex rules to integrate with Help desk.

- Interval computation for Frequency event rule is based on time the first event occurred. EvtAds works on interval based on last update time – this is key for effective duplicate suppression.


Help Desk

- To provide integrity, it is important that duplicate problem tickets are not raised. Multiple problem tickets for the same problem can easily distract the main focus of resolving the problem.

- For this reason, duplicate Alerts should be identified and discarded (or in some cases escalated).

- If duplicated alerts exceed an acceptable limit, the problem should be escalated to minimise the impact. It is important to identify the number of times an alert has occurred within a clear down interval.

Usage


EvtAds Functions

- CAds

- Checks for duplicate Events

- CVersion

- Displays Version details

- CStats

- Displays how many events are cached

- CList

- Generate list of cached entries for review


EvtAds FunctionsCOptions

Funtion Name Option Description

COptions -Purge On Demand Purge expired entries

-Debug On Demand start Debugging

-StopDebug On Demand stop Debugging

-Reinit

Reinitialise options by reading the EvtAds.dat file again. This is normally useful if Search and Replace entries have been changed


ClearDownInterval

- This is used for computing if the cached event entry is current and not expired. It is based on Last Update time of the event.

- If (Last Update Time + ClearDownInterval) < Current Time, the cached event entry is expired and the duplication count of the cached event is reset to zero.

- This enables identification of new occurrences of problems outside the ClearDownInterval


define msgact

name=(*,10)

action="EXTERNAL"

attrib="DEFAULT"

color="DEFAULT"

condop=" "

evaluate='Y'

quiet='N'

status="ACTIVE"

sim='N'

text="EvtAds.DLL CAds /T:120 /M:""&text"""

EvtAds Message ActionCAds Function

Function Name

ClearDownInterval

Message Text

Action


define msgact

name=(*,10)

action="EXTERNAL"

attrib="DEFAULT"

color="DEFAULT"

condop=" "

evaluate='Y'

quiet='N'

status="ACTIVE"

sim='N'

text="EvtAds.DLL CAds /Q /T:120 /M:""&text"""

EvtAds Message ActionCAds Function Query Option

Function Name

ClearDownInterval

Message Text

Action

Query Option


Query Option

- Query option function works in the same as CAds except the cache count is not updated. The clear down rules still applies.

- This function is extremely useful when conditional logic is required such as “Stop forward action if more than 10 forwards have failed due to destination down”.


Sample Application of Query Option

There is another MRA for %CAOP_E_504 with EXTERNAL action. This issues CAds Function call with /M”&1,&6”. This updates the cache counter to identify the number of times, the forward has failed within cleardown interval.

Prior to forward action, Query option is requested to determine the number of times %CAOP_E_504 message has been generated. The query option will not update the counter. In this example if more than 5 FORWARD to node BREER01PACO have failed, forward action is suspended with appropriate messages. This would be good candidate for EVALUATE function


Sample Application of Query Option


define msgact

name=(*,10)

action="EXTERNAL"

attrib="DEFAULT"

color="DEFAULT"

condop=" "

evaluate='Y'

quiet='N'

status="ACTIVE"

sim='N'

text="EvtAds.DLL CAds /T:600 /F:2 /M:""&(2:9)"""

EvtAds Message ActionSearch & Replace Override

Function Name

ClearDownInterval

Message Text

Search & Replace Item


CVersion

- Display EvtAds Version details

define msgrec

msgid="EVTADS CVersion"

type="MSG"

msgnode="*"

desc="What version of EvtAds are you running"

cont='N'

msgact='Y'

wcsingle='?'

wcmany='*'

case="y"

regexp="n"

define msgact

name=(*,10)

action="EXTERNAL"

attrib="DEFAULT"

color="DEFAULT"

condop=" "

evaluate='Y'

quiet='N'

status="ACTIVE"

sim='N'

text="EvtAds.DLL CVersion"


CVersion


define msgrec

msgid="EVTADS CStats"

type="MSG"

msgnode="*"

desc="Tell me how many entries have been cached so far"

cont='N'

msgact='Y'

wcsingle='?'

wcmany='*'

case="y"

regexp="n"

define msgact

name=(*,10)

action="EXTERNAL"

attrib="DEFAULT"

color="DEFAULT"

condop=" "

evaluate='Y'

quiet='N'

status="ACTIVE"

sim='N'

text="EvtAds.DLL CStats"

CStats

- Display number of events cached


CStats

CStats - 2000 entries cached


define msgrec

msgid="EVTADS CList"

type="MSG"

msgnode="*"

desc="Dump Current Cached entries to a ascii file"

cont='N'

msgact='Y'

wcsingle='?'

wcmany='*'

case="y"

regexp="n"

define msgact

name=(*,10)

action="EXTERNAL"

attrib="DEFAULT"

color="DEFAULT"

condop=" "

evaluate='Y'

quiet='N'

status="ACTIVE"

sim='N'

text="EvtAds.DLL CList"

CList

- Dumps cached entries for review

- List file placed in EvtAds directory.

- File name

- yyyymmdd.ads


CList


CList

Clear Down Interval


Clear Down Interval

- Clear Down Interval is Reset Interval. Can be different for different message rules.

- Cached entry is expired if the last update of the event is older than clear down interval.

- If no Clear Down Interval specified on the message rule, it defaults to Initialisation Option default


Clear Down Interval


Hierarchy

- /T: override on message action

- Initialisation Default from Ini File


Clear Down IntervalOverride

Purge


What is Purge?

- Remove matured cached event entries from the queue

- Cache entries eligible for Purge if Current Time > (LastUpdateTime + (ClearDownInterval * PurgeLag)

- PurgeLag is customisable option via configuration GUI.


PurgeLag


Why Purge?

- This is a simple housekeeping function to keep the Queue to a manageable size.

- In the lab environment, Queue Size of 300,000+ has been tested successfully.

- This allows removal of single event entries with no probability of reoccurrence


How is Purge Initiated

- Purge can be initiated on demand or automatically performed at the specified Check Interval

- CheckInterval = nn - where nn = interval value in minutes. - INI File option - configured via Configuration GUI - Check Interval value should be based on the Event Activity. In

most cases Check Interval of 2hours should suffice. However, in lab environment, this has worked fine with value of 1 minute.


CheckInterval


Purge On Demand

- To run purge on demand, simply call COptions –Purge function.

- Utility is built with threshold monitoring of Queue Size. If it exceeds Warning or Critical thresholds, it generates Warning or Critical Messages.

- This messages should be reviewed with the view to execute Purge On Demand request on receipt threshold violation messages


Queue Size Monitoring


Automatic Purge

This shows Purge is executed every 15 minutes. CheckInterval value set to 15


How to disable automatic Purge

- If you wish to stop the automatic purge process and certainly not recommended, then this can be accomplished by manually updating EvtAds.dat file.

- Set CheckInterval=0

- This will disable to automatic purge process

- The Configuration Wizard, will not permit this value to be set to 0 as it is not recommended


Purge

12,000+ Entries purged in no time

Search and Replace


Why?

- Some events may include fields such as Time which would make the event unique and thus potentially not suitable for ADS.

- If fields such as Time are not relevant to determine the duplication count, then search and replace function should be reviewed.


How?

- Using Configuration GUI, specify POSIX regular expression to search the message text passed to EvtAds and replace with the supplied text.

- GUI provides a TEST function to verify your search and replace function.

- Each Search & Replace item is allocated a sequential number. This number should be specified on the CAds function as /F:n option, where n is the item number


Replace Time Field with hh.mm.ss

- Define Search and Replace Item.

- Item number for this field is 2

- ReformatEventSearch<ItemNum>

Replace Item

Search RegExp


Search & ReplaceOverride Option

Item Number


Time Field Replaced with hh:mm:ss

cawto EVTADS Jobxxxx failed at 10:11:38





Generate Time Based Events

Cached Entry


Wizard

Message Prefix


Ignore Message Prefix


Message Prefix

- Some Message prefix are not suitable for EvtADS suppression. For example you may want to see all %CAOP messages or messages generated by EvtADS itself.

- This function provides assurances in case catch all message record rule is used for ADS, such as *

Cache Statistics


Release r11

- r11 release has been enhanced to maintain some degree of statistics of the cache events. This enhances the CList function if MaintainStats option is set to 1.

- By default, it will work as prior releases i.e no history of cached events will be maintained


MaintainStats

- This option is not currently configurable from the wizard. If the option is set to 1, it will enable Stats option


MaintainStats – Automatic Mode

- If MaintainStats option is enabled, it will maintain cache count (duplicate count) of each cached entry.

- CheckInterval, is a configurable option from the configuration wizard. It specifies how often to perform housekeeping on the expired cache entries i.e entries that gone past (CleardownInterval * PurgeLag).

- If MaintainStats is enabled, at each CheckInterval, it will automatically generate statistics and append to the file YYYYMMDD_ads.csv.


MaintainStats – Manual Mode

- If MaintainStats option is enabled, stats information can also be manually requested by calling function CList.


define msgrec

msgid="EVTADS CList"

type="MSG"

msgnode="*"

desc="Dump Current Cached entries to csv file if MaintainStats enabled"

cont='N'

msgact='Y'

wcsingle='?'

wcmany='*'

case="y"

regexp="n"

define msgact

name=(*,10)

action="EXTERNAL"

attrib="DEFAULT"

color="DEFAULT"

condop=" "

evaluate='Y'

quiet='N'

status="ACTIVE"

sim='N'

text="EvtAds.DLL CList"

MaintainStats - CList

- Dumps cached entries for review

- List file placed in EvtAds directory.

- File name

- yyyymmdd_ads.csv


Stats File

This shows EvtAds was processing 1700 events per seconds


MaintainStats

- Id- Groups all entries for a specific stats request

- Cache Rate- Number of Events cached per second for the respective message text

- If the entry was purged due to no activity, the event rate will be calculated from the time the entry was re-cached till the lastupdate time. If the entry has not been previously purged, then the event rate is calculated from the time the entry was first cached till last update time.


MaintainStats

- Total Cached

- Total number of Events cached for the respective message text

- If the entry was purged due to no activity, the “Total Cached” count is from the time the entry was re-cached. If the entry has not been previously purged, then the “Total Cached” count is from the time the entry was first cached.


MaintainStats

- DuplicateCount- The total number duplicate entries within the current

cleardownInterval period. - ElapsedSeconds

- The number of seconds from the time the cache entry was created or recreated if previously purged

- Request- Request Type. The entries written to the csv file was as a

result of CheckInterval, then this is set to Auto. If the CList function is called by the user, this is set to Manual

- FirstTimeStamp- The timestamp of the first occurrence of the cached

message text. If the entry was previously purged, it is timestamp when the entry was re-created


MaintainStats

- LastUpdateTimestamp

- The timestamp of the last cached entry update

Scalability


Test Plan

- Generate Message Storm of quarter million (250,000) events

- Analyze the latency with EvtAds caching the entries from this message storm

- Display the first entry and last few entries

- Re-run the tests without caching entries.

- Analyze the latency without EvtAds caching the entries


Message Actions

1. Cache Entries based on message token 2 and 3.

2. Check if first entry. If so, display the text

3. Check if the last few entries. If so, display the text

4. Otherwise do not display the message on the Unicenter Console


Generate Message storm

- Message Storm Ended at 08:20:29


Message Storm 250,000 Events

- 250,000 Events message storm cached by EvtADS

- Latency on the last cached event 71 seconds


EvtAds

This shows EvtADS processing 2121 events per second


Re-run test without EvtADS

- Re-run the tests without executing the 4 MRA processed by the message storm

The latency without EvtADS caching 250,000 events and not processing 4 MRA is 50 seconds. That is 21 seconds less than EvtADS


Purge

60,000+ Entries purged under a second


Cache Table Search Time

- If large number of entries are cached (over 50,000) , degradation in response time should be expected as the time to search for a specific cache entry in the cache table will increase as well

- EvtADS monitors its cache table and generates alert messages if it exceeds Warning or Critical threshold.

- If you are planning to trigger EVTAds with MRA such as “*” i.e for all messages, then please review purge housekeeping options such as PurgeLag and CheckInterval to maintain the cache table to a reasonable size

Install


Install


Install

Configuration Wizard


Configuration Wizard


Options Configuration


Message Prefix Exclusion


Search & Replace


Search & Replace


ADS Management Options

Verification


Verification

- Execute EvtAds_Demo.cmd few times and verify

- Duplicates Message Discarded Test- Only one message should displayed evtAds_001 New Entry Cached - EVTADS DupTest This should only show the first message

- Search and Replace Function- Only one message displayed for duplicate Time field messagesevtAds_001 New Entry Cached - EVTADS Jobxxxx failed at 10:11:38

- Escalation- 3 Message displayed

- Normal when duplication 0

- Warning with 10th occurrence of message

- Critical with 15th occurrence of message


Alert based on 10th and 15th occurrences

Count based Escalation verification. Warning EVTADS ServiceDesk Process displayed on 10th occurrence of the duplicate message within 2 minutes interval. Critical message on 15th occurrence

EvtADS Initialize


Message Storm at Startup

- If the message storm is generated and EvtADS is not initialized, then there may be a small delay at start up. This delay is mainly due to CA OPR daemon has to load the EvtADS external library.

- For release r11, it is recommended to initialize EvtADS at startup of CA OPR daemon, so that external library is pre-loaded.


How to initialize EvtADS at Startup?- Define Message Records Actions for the Message Id

- %CAOP_I_DAEMONINIT, Console Daemon initialized on node

- Define Message Actions as defined in the file EvtADS_init.cautil


Initialize EvtADS at Startup


Initialize EvtADS at Startup

- If EvtADS is not initialized at startup, CA OPR daemon will initialize the first execution or EXTERNAL message action for EvtADS library


Questions and Answers

Any questions?Any questions?

Documents

Unicenter Automatic Duplicate Suppression EvtADS -EvtAds r11 -Field Developed Utility -January 17 th 2006