Performance SeNTry: Development Status
ReportDemand Technology SoftwareA DataCore Software company
1020 Eighth Avenue South, Suite 6, Naples, FL 34102phone: (941) 261-8945 fax: (941) 261-5456
e-mail:[email protected]://www.demandtech.com
2Demand Technology Software.-- Confidential.
Agenda
Performance SeNTry version 2.4.2 highlights NTSMF Collector SeNTry Administration
Version 3 roadmap Alerts and Traps File System Collector Improved Systems Console for real-time
notification, management, and control
3Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2Topics
Windows 2000 ‘features’ NTSMF features SeNTry Administration features Event log use New Enhancements – SQL Server 2000 New Enhancements - Exchange 2000 support New collection service architecture –
Threaded collector Improved Exception handling
4Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2Windows 2000 ‘features’
System level objects now in visible performance libraries
System object ‘missing’ some NT4 counters: %Total Processor Time, % Total User Time, % Total Privileged Time, %
Total DPC Time, % Total Interrupt Time, Total Interrupts/sec
Delay in some objects reporting data at system start going through RegQueryValueEx()
New values in Performance Registry keys Timeouts DisablePerformanceCounters
5Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2 New fields in Registry: HKLM\…\Services\PerfDisk\Performance
6Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2 Additional NTSMF features
Install option to run under a user account -install -account <account name> -password
<password>
Program defaults with filters active - NT4 and Win2K have specific sets
Memory size field in header fixed for Win 2000 systems with 4GB or more of physical memory No fix for Windows NT4 systems with 4GB or more of
physical memory since the GlobalMemoryStatus api call returns only a 32 bit number
7Demand Technology Software.-- Confidential.
SeNTry Administration enhancements
New default DCS for Win2K and Performance Gallery Gold templates
SQL Server 2000 object support MS Exchange 2000 objects support Tool to compact the Performance SeNTry
Database ('Tools' menu item in the 'Network Browser' window)
More filters More options
8Demand Technology Software.-- Confidential.
Process New Process Filter on Process Working Set Size Always write named processes – availability
reporting Threads
Add 'Ready Threads' counter in the 'Process' object No longer need to write Thread objects into .smf
Terminal Server suppress Idle User record write User and Session records (automatic) only
when activity occurs
SeNTry Administration enhancementsCollection Filters - Process
9Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2Collection filters, Thread
10Demand Technology Software.-- Confidential.
SeNTry Administration enhancementsParameters Administration, Collection Cycle
On Error Command capability for launching commands when errors are encountered Same rules as Cycle
End commands ‘poor man’s’ alert –
something as simple as ‘Net Send user-name message’
11Demand Technology Software.-- Confidential.
SeNTry Administration enhancementsParameters Administration, Scheduling
Flexible DCS chaining specifying Next DCS in chain(e.g., prime shift.DCS chains to nighttimeDCS, which chains back)
Collection Period to specify the period that the collector is active and reporting data
Provide basic scheduling options without requiring a 3rd party package
12Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2Parameters Administration, File Contents
Support for summarization program
Collect Inventory data from Registry keys
13Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2Parameters Administration, File Contents
New option – “Write denominator counters” Used by .smf Summarization (DMSumSMF.exe) uses
in calculations to compute correct average values of compound Counters
e.g., Copy Read Hits % One more option to manage the volume of data that
can be collected effectively “Base” appended to denominator counter name
strings for identification in .smf file
14Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2Parameters Administration, File Contents
Summarization utility (based on Counter Type) Requires Type 6 records! Designed to run at Cycle End, but can be run
standalone Creates sum.smf files
Identical in format to detail records aged like any other .smf file in Current/Previous/Archive
folders Calculates the arithmetic mean for most Counters Calculates interval peak (min, max) where
appropriate Some counters simply save the last observed value
(e.g. elapsed time)
15Demand Technology Software.-- Confidential.
SeNTry Administration enhancementsParameters Administration, File Contents
‘Include Registry Key Values’ to get registry key value information into configuration records in the SMF data file. Used to collect hardware and software inventory, versioning,
etc. No restrictions: inventory as many values as you need
Must specify the full Registry key name and field name Use in conjunction with regedit Copy Key Name function Key Not Found earns a Warning message, but no other side
effect Invalid spec earns a Warning message, but no other side effect
16Demand Technology Software.-- Confidential.
SeNTry Administration enhancementsParameters Administration, File Management
PKZIP compression option at Cycle End Another way to
manage data volume
17Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2Event Log messages
Informational, Warning and Error messages are placed in the Application Event log
Also in <system>.ntsmf.log (plain text) Date, time User, System, source filled in by
event facility Type, category, event id, description defined
by developers I and W will not
18Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2Event Log messages - Warning
19Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2Event Log messages – ntsmf.log example
10/24/01-00:00:01 - Event ID: 210, Category: Discovery, Severity: Warning
The Performance Library, "C:\Program Files\Exchsrvr\bin\exprfdll.dll", for service "ExIPC" (ExIPC), failed to load.
The system error was "The specified module could not be found (126)".
Performance data for this service will not be available.
The Event Viewer (source Perflib) may have more details
20Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2Event Log messages – unhandled exception
10/13/01-00:00:02 - Event ID: 96, Category: N/A, Severity: Error
Unhandled exception encountered
10/13/01-00:00:02 - Ended Performance SeNTry service
10/13/01-00:00:02 - Event ID: 104, Category: Initialization, Severity: Info
The Performance SeNTry service stopped with error code, -1073741728.
Version=2.3.6
21Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2New Enhancements – SQL Server 2000
Lots of new SQL Server 2000 records A single machine can be run multiple instances of SQL
Server No instance names for separate SQL Server processes, as
delivered by MS MSSQL$db-name:object-name e.g. MSSQL$SUP:Buffer Manager,… MSSQL$SUP1:Buffer Manager,…)
Anticipate object selection and reporting problems We write a MSSQL:Buffer Manager Object instance
where the db-name token becomes an Instance name2.2.2,0,5,2001,4,17,20,36,45,859,0,MSSQL:Buffer Manager,1,18,22,1,SQL Server DB,Buffer cache hit ratio,…
22Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2New Enhancements - Exchange 2000 support
New objects visible when optional components installed (e.g. Instant Messaging)
Cases where a perflib loads another .dll exprfdll.dll loads exchmem.dll
Objects have many counters: MSExchange Web Mail … 75 counters MSExchangeIMAP4 … 96 counters MSExchangeIS ……… 107 counters (actual) Database …………….. 127 Counters (many of ‘no
data’ type)
23Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2New Architecture - Threaded collector
Separate thread for each performance library found in the Registry 3rd party Perflib DLLs perform the actual performance
data gathering Windows 2000, XP expose the ‘system’ perflibs,
including perfos.dll, perfdisk.dll, perfproc.dll, perfnet.dll
Improved error handling, especially in 3rd party code Additional collection service threads manage
dispatching, file writes, other tasks
24Demand Technology Software.-- Confidential.
NT Performance monitoring API
Perflib Dlls are loaded into the dmperfss process address space when opened
3rd party Perflib Dlls might need to synchronize with a separate monitored address space
Init ializat ion:Enumerate Performance subkeys
Loop Forever:Call Collect
Perfos.dll
sqlctr80.dllPerfnet .dll
Performance Monitoringapplicat ion address space
Monitored applicat ion address space
SQLSERVR.exe
25Demand Technology Software.-- Confidential.
NT Performance monitoring API
Open and Collect Timeout values exceeded trigger DisablePerformanceCounters setting, which must be disabled manually!
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib ExtCounterTestLevel can be used to verify integrity of 3rd party Perflib dlls.
26Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2New Architecture - Exception handling
Try to catch exceptions that occur in a thread, i.e., inside code in a Perflib DLL
The failing thread and perflib (and the performance Objects it supplies) are deactivated, while other Objects continue to be collected
Problem perflib is automatically retried every new cycle or on demand (Notify Command) No manual intervention required
Unless the Object is on the critical list: System, Memory, Cache, Process, Thread, Processor Critical missing means log and stop
27Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2New Architecture - Exception handling
On exception: system error text message the loaded modules with addresses and the call stack
Helps to identify ‘rogue Perflib dlls’ Very compact and accurate
28Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2Log message and module list
10/26/01-10:20:05 - Event ID: 1300, Category: PerfData, Severity: WarningCould not collect performance data for object, "Database ==> Instances (L28698, G3370), counter, Pages Converted/sec (L28700, G3372)".The system error was "Win32 exception, 0xC0000005, encountered at location 0x0043FBB1Read attempt at location 0x00000020Module List:(0x00400000-0x00480000) D:\NTSMFv24224\DMPerfss.exe(0x01000000-0x01017000) F:\WINNT\System32\BEPerfDll.dll(0x01140000-0x0114E000) D:\Program Files\Exchsrvr\bin\davexpc.dll(0x01630000-0x0163B000) D:\Program Files\Exchsrvr\bin\eseperf.dll(0x02000000-0x02010000) F:\WINNT\system32\dssdata.dll(0x02C40000-0x02C46000) D:\Program Files\Exchsrvr\bin\srsperf.dll….(lots more of these)
29Demand Technology Software.-- Confidential.
Stack Trace (ThreadId 4540):
DMPerfss.exe: PSSPerfLocalCounter::RealizeCache + 111 bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfLocalCounter.cpp, Line 372 + 15 bytes
DMPerfss.exe: PSSPerfObjectCache::RealizeCache + 821 bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfObjectCache.cpp, Line 247 + 11 bytes
DMPerfss.exe: PSSPerfObjectCache::OnCollectionComplete + 12 bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfObjectCache.cpp, Line 408 + 9 bytes
DMPerfss.exe: PSSPerfDataSink::Collect + 597 bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfDataSink.cpp, Line 395 + 14 bytes
DMPerfss.exe: PSSPerfDataSink::Collect + 15 bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfDataSink.cpp, Line 463 + 15 bytes
DMPerfss.exe: PSSPerfObject::_Collect + 75 bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfObject.cpp, Line 1688 + 26 bytes
DMPerfss.exe: PSSPerfObject::Collect + 79 bytes
E:\DmPerfss 2.4.2\Collector\PSSPerfObject.cpp, Line 1748 + 20 bytes
Performance SeNTry version 2.4.2Stack Trace
30Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2Stack Trace
DMPerfss.exe: PSSCollectionCycle::Start + 1483 bytes
E:\DmPerfss 2.4.2\Collector\PSSCollectionCycle.cpp, Line 1500 + 45 bytes
DMPerfss.exe: PSSPerformanceSeNTry::StartCollection + 248 bytes
E:\DmPerfss 2.4.2\Collector\PSSPerformanceSeNTry.cpp, Line 1571 + 11 bytes
DMPerfss.exe: PSSPerformanceSeNTry::Run + 109 bytes
E:\DmPerfss 2.4.2\Collector\PSSPerformanceSeNTry.cpp, Line 1254 + 22 bytes"
31Demand Technology Software.-- Confidential.
Performance SeNTry version 2.4.2 New Architecture - Exception handling
New dmperfss.exe ships with debug symbols included for extended error diagnostics report Size of the image file is now approximately 5.8 MB The larger image file installs by default
A dmperfss.exe without symbols is also on the CD E.g., a floppy disk distribution/install
dmperfss.exe with symbols is preferred, especially if new applications or new versions are discovered
32Demand Technology Software.-- Confidential.
Performance SeNTry Version 3
BREAK
33Demand Technology Software.-- Confidential.
Performance SeNTry Version 3 Roadmap
Real-time Alerts Event log & SNMP support
File System Collector Revamped Systems Console
34Demand Technology Software.-- Confidential.
Performance SeNTry Version 3Alerts
Alert is written into the Application event log
Triggered by system condition (e.g. cpu busy threshold)
Frequency of alerts is also thresholded; won’t fill up the Event Log unnecessarily Trigger, Release mechanism
Alerts are cleared when condition is absent (based on the ‘lower threshold’)
35Demand Technology Software.-- Confidential.
Performance SeNTry Version 3SNMP - Traps
Like Alerts, defined by same rules SNMP extension agent notified of the ‘alert’ by
dmperfss.exe SNMP extension agent generates a Trap
message Trap is sent to management entities by SNMP
facility Management entities would be OpenView,
Tivoli, CA-Unicenter, others
36Demand Technology Software.-- Confidential.
Performance SeNTry Version 3File System Collector
At prototype stage now… Probably another thread in the collector Designed to run infrequently, say once/day
(don’t know how long it takes yet)
Report files that are above threshold (e.g. file size > 2% Logical disk size)
Report folders that are above threshold when files in them are totaled(don’t also report parents, just siblings)
37Demand Technology Software.-- Confidential.
Performance SeNTry Version 3File System Collector
Summary records – size of files remaining NTSMF header record format Pseudo object id Records are instances where the UNC path is
the instance name Probably optioned - allow include in .SMF file
or write to a file of its own
38Demand Technology Software.-- Confidential.
Performance SeNTry Version 3System Console
Processes real-time alerts Can examine objects and counters
related to the alerts Can change the alert definition, collection
interval, etc., as needed for the display Display results graphically Capture the data and graphs for later
review
39Demand Technology Software.-- Confidential.
Performance SeNTry Version 3Exchange 2000 references
http://www.exchange-mail.org/whitepapers.html
http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/exchange/reskit/ex00res/resguide/c29monit.asp(Microsoft's online version of the Exchange Resource Kit Documentation)
http://www.netiq.com/Downloads/Library/White_Papers/NetIQ_Monitoring_and_Managing_Exchange.pdf
40Demand Technology Software.-- Confidential.
Performance SeNTryreach us at:
Demand Technology SoftwareA DataCore Software company
Address: 1020 Eighth Avenue South, Suite 6, Naples, FL 34102 Phone: (941) 261-8945 fax: (941) 261-5456
E-mail: [email protected] Web site: http://www.demandtech.com
41Demand Technology Software.-- Confidential.
Performance SeNTry Version 3 Roadmap
Questions?