9/3/2006

BMC Configuration Management (Marimba) Best Practices and Troubleshooting

Andy SantosaSenior Technical Support Analyst

9/3/2006 ©2006 BMC Software2

Agenda

› CM Infrastructure› CM Inventory› CM Subscription› CM Software Distribution (App. Packager and Content Replicator)› CM Deployment Manager

9/3/2006 ©2006 BMC Software3

CM Infrastructure

› What’s New in Tuner 7.0?› What’s New in Transmitter 7.0?

9/3/2006 ©2006 BMC Software4

What’s New in Tuner 7.0?

› Status Report– Returns general information (for example: tuner version, release date, VM version,

heap-size, arguments, and operating system information) and checks if the tuner is running

• http://hostname:7717/workspace?status

› Debug Report– Returns specific information and checks for problems with the tuner (for example:

tuner properties, thread dump information, license information, RPC configuration and connections)

• http://hostname:7717/workspace?debug&<option>=t includes the option&<option>=f excludes the option

• http://hostname:7717/?debug&config=t• http://hostname:7717/?debug&threaddump=t

› Log Report– Returns the tuner and channel history logs

• http://hostname:7717/workspace?log• http://hostname:7717/workspace/http://hostname:5282/Marimba/Current/SubscriptionService?log

9/3/2006 ©2006 BMC Software5

Status Report

9/3/2006 ©2006 BMC Software6

What’s New in Tuner 7.0? (Cont’d)

› Thread Dump– You can get thread dump information remotely from both client endpoints and

servers.• runchannel http://hostname:5282/Version7/TunerAdministrator -getthreaddump -tuner hostname:7717

-username admin -password "" -output C:\Temp\threaddump.txt

– If you want to generate thread dumps remotely, the Java VM must be launched with the –Xdebug Option. • On servers (transmitters, proxies, etc.), profiles do not include the -Xdebug

option for performance reasons.

› Minimal Mode– Minimal mode is now a separate process known as minituner.exe and is no longer

part of the tuner. This makes minimal mode more robust and the tuner more reliable.

9/3/2006 ©2006 BMC Software7

What’s New in Tuner 7.0? (Cont’d)

› Improved Robustness and Stability– IPC service switched from DDE to named pipe.

• Comparing to DDE, named pipe provides a more efficient and generic bidirectional communication mechanism. It does not require applications to be GUI-based, and data exchanged can be in arbitrary format.

› Receipt Service– BMC® Remedy® Change Management could initiate a change request/task on a

Marimba Tuner endpoint through Deployment Manager and Policy Manager.Tuner is to provide a receipt service so as to leave a meaningful “audit trail” for the work performed on an endpoint.

– Receipts are stored in the tuner workspace.

9/3/2006 ©2006 BMC Software8

What’s New in Transmitter 7.0?

› More robust Transmitter storage› Detection of ungraceful Transmitter shutdowns› Tx Verify improvements› New debug flags:

– Tuner• TUNER/RECEIPT

– Transmitter• TX/GC • TX/DIFF• TX/DSL• TX/HTTP

9/3/2006 ©2006 BMC Software9

Infrastructure - Recommendations

› Separate CMS, Transmitters and Windows Patch Source

› Java Launch Arguments:– marimba.launch.javaArgs=-Xms128m -Xmx512m -XX:PermSize=32m -

XX:MaxPermSize=128m

› Using Profile Property– If the tuner profile property doesn’t exist -> ONLY Kernel Upgrade– If the tuner profile property exists, but Transmitter doesn’t have the profile segment -

> No Kernel Upgrade– If the tuner profile property exists, but Transmitter has the profile segment -> Kernel

upgrade + Profile applied

› Health Check Tools– Replication Monitor– Tx Verify– Transmitter Guardian

9/3/2006 ©2006 BMC Software10

Troubleshooting - Scheduler

› Update schedule vs. start schedule– Subscription/Policy Service -> update.schedule– Patch Service -> update.schedule– Inventory/Scanner Service -> start.schedule– Infrastructure Service -> start.schedule

› Missed update/start schedule while the machine was off the network/turned offExample:– Subscription update schedule at 2 AM– Inventory start schedule at 3 AM– Marimba schedule start delay=10000

› Schedule dumper to help debugging scheduler issue

9/3/2006 ©2006 BMC Software11

CM Inventory

› What’s New in Inventory 7.0? › Performance› Debugging

9/3/2006 ©2006 BMC Software12

What’s New in Inventory 7.0?

› In 6.x, compliance will ALWAYS re-send all data (even when no data changes between scans). This results in an expensive delete of all compliance data followed by a re-insert of all data. This was fixed in 7.0.

› Using AMT Data As mac_id– Intel AMT data is also now part of Intel inventory scans. Identify if machine is AMT-enabled

and fetch persistent Tuner ID from AMT chip.

› New Diffing Logic– Maintain a full report cache on the endpoint. – Diff against the current inventory data in the DB if there is a checksum mismatch.– Purge full report cache based on size and/or time interval.

› Double-Byte Storage

› Mirror Forwarding– Mirrors on WAN can forward Inventory reports to LAN Mirrors/Master – Reduce the chances of database blocking or deadlocks

› Task Receipts (CCM)– Scan and report task receipts

9/3/2006 ©2006 BMC Software13

What’s New in Inventory 7.0? (Cont’d)

› Old 6.X Diff Over-Write That Resulted in Full Scan Report› [15/May/2006:22:23:33 -0400] - warning - 6672 Over-writing older report for machine: wuscxdw221(win32:3888154505819947868)

Diff 1

Inventory Service Plugin

DBDiff 1

Diff 2

Inventory Service Plugin

DBDiff 1

Disk Queue:Diff 2

Diff 3

Inventory Service Plugin

DBDiff 1

Disk Queue:Diff 2Diff 3`

Endpoint

1

2

3

Diff Checksum Mismatch: Diff Scan Old Checksum is

for Diff 2, Plugin Local Checksum Cache has Diff 1.

Request Full Scan report from Endpoint

4

Endpoint sends Full Scan Report Full

Scan Report

Inventory Service Plugin

DBDelete AllInsert All

Disk Queue:

5

9/3/2006 ©2006 BMC Software14

What’s New in Inventory 7.0? (Cont’d)

› 7.0 Special Diff Scan Report (Instead of 6.x Full Report)– [15/May/2006:22:23:33 -0400] - warning - 6672 Over-writing older report for machine: wuscxdw221(win32:3888154505819947868)

Diff 1

Inventory Service Plugin

DBDiff 1

Diff 2

Inventory Service Plugin

DBDiff 1

Disk Queue:Diff 2

Diff 3

Inventory Service Plugin

DBDiff 1

Disk Queue:Diff 2Diff 3`

Endpoint

1

2

3

Diff Checksum Mismatch: Diff Scan Old Checksum is

for Diff 2, Plugin Local Checksum Cache has Diff 1. Request Special Diff report

from Endpoint

4

Endpoint generates new special diff scan report

between the full report at Diff 1 state and the full

report at the Diff 3 state and resends back to the plugin

Diff 3'

Inventory Service Plugin

DBDiff 3'

Disk Queue:

5

9/3/2006 ©2006 BMC Software15

Performance

› SQL Server – Report Center queries are basically unusable while inventory insertions are running in the background.

› Highly recommend enabling “dirty reads” if inventory insertions need to be run while Report Center queries take place. Reason:

• Inserts/Updates/Deletes issue an exclusive lock• Select statements issue a shared lock• A shared lock cannot be issued when an exclusive lock is in place and an

exclusive lock cannot be issued when a shared lock is in place• You can change the behavior of locking via a lock hint:

- Nolock - Enables dirty reads where a Select does NOT issue a shared lockand Select does NOT honor exclusive locks

- 6.x – Only Report Center interactive queries can use dirty reads (enabled through property)

- 7.0 – Report Center interactive queries, email reports, collections, and processing queries that another application requests can use dirty reads (enabled through UI)

9/3/2006 ©2006 BMC Software16

Performance (Cont’d)

› Inventory Plugin Tuning– Set the scan schedule as far apart as possible (24 hours, if possible).– Use the scheduling “vary” option to reduce the load all insert at the same time load

(due to time sync’d machines).– 3 to 5 inventory plugin Oracle database connections appears to be optimal. More

connections just result in more resources and waiting instead of increasing throughput.

– Disk I/O is low.– CPU utilization is low. – Memory is low.– Java GC tuning is lower priority since currently GC’s / heap size growth does not

appear to be causing issues.

9/3/2006 ©2006 BMC Software17

Debugging Customer Performance Issues

› Guide to Debugging Customer Performance Issues– Get inventory plugin logs from all masters/mirrors/repeaters.– Gather a few sample Endpoint Inventory Service channel workspaces.– Oracle redo logs and/or archive logs (if archive log mode is enabled).

• If gathering archive logs, also request character sets, dictionary file, version/platform Oracle is running on.

– Tuner/History logs where inventory plugins running.– System Architecture – Scan schedule, # endpoints, # repeaters, # mirrors, load

balancer, which plugins insert directly into database, etc. – Gather any queued-up scan reports in the inventory plugin disk queue.– Find out if the scan schedule is too aggressive and check if it is possible to reduce

the scan schedules.– In more complex environments, see if client IP Load Balancer persistence might

help reducing checksum mismatches.

9/3/2006 ©2006 BMC Software18

CM Subscription

› What’s New in Subscription 7.0?

9/3/2006 ©2006 BMC Software19

What’s New in Subscription 7.0?

› Install Priority • You can now set install priority values by typing numeric values per package on the

Edit Policy page.• In M6, you can only scroll arrow up and down to change the install priority.

› User-Based/Machine-Based Policies• Policy will be assigned based on either user-based, machine-based, or both. In M6, the

only option was both.• Property: ‘marimba.subscriptionplugin.resolvetype’ Value: user or machine, otherwise

both.

› Directory Service Hierarchy Structure

• Policy Manager now uses an updated LDAP container structure.

• Enables you to more effectively extend the schema of the Active Directory

infrastructure, and store and secure policies.

9/3/2006 ©2006 BMC Software20

What’s New in Subscription 7.0? (Cont’d)

• Supports install order • Supports staging new url before doing an autoupdatefrom• Will not updatefrom if marimba.subscription.nodelete=true

• Will revert url if channel fails to install

9/3/2006 ©2006 BMC Software21

Compliance – Architectural Overview

Inventory Plug-In• Implements differencing for compliance. It only inserts compliance data that

has been modified from the previous inventory scan on the endpoint.

Subscription Service• Invoked as a custom scanner by Inventory Service to calculate compliance

on the endpoint.

Query-Based ComplianceTwo types of queries in compliance engine:

1. Inventory-only query; query based solely on inventory scan data2. Latest-data query; query based on inventory, LDAP-synced policy and

Transmitter

9/3/2006 ©2006 BMC Software22

Compliance (Cont’d)

› Scheduling LDAP Sync• The LDAP synchronization process is new in this release.• LDAP synchronization optimizes the accuracy of compliance queries and reports by

updating the configuration database with the latest available policy data.

› To optimize policy management workflow, schedule the following processes in sequence:• Update policy service• Perform inventory scans• Schedule LDAP Sync• Typically, you schedule the processes on a daily basis • You can schedule LDAP Sync using the LDAP-to-Database Synchronization Service link

on the Data Source tab in CMS System Settings

› Compliance Reporting• In addition to viewing policy compliance information based on targets, you can now view

compliance based on a specified package.

9/3/2006 ©2006 BMC Software23

New Features: Compliance (Cont’d)

9/3/2006 ©2006 BMC Software24

CM Application Packager/Content Replicator

› What’s New in Application Packager 7.0?› What’s New in Content Replicator 7.0?

9/3/2006 ©2006 BMC Software25

What’s New in Application Packager 7.0?

› Support for MSI 3.0– No need for ‘User Elevation’ when installing MSI packages

› Delaying the download of MSI package until pre-scripts are run› Repair of MSI now shows progress bar› Return code mapping› App-friendly name› DSL support

9/3/2006 ©2006 BMC Software26

What’s New in Content Replicator 7.0?

› Packaging and Installation of Content From– Existing

• 1 source to 1 target folder

– N source to N target folders› Installing Unix Packages

– Using XML Command file to specify packaging commands• {5.1 screen shot}

› Self-Installing Channels– New “package” command that will facilitate installation of self-installing channels

• {5.2 Screen shot with syntax}› Channel Signing (Credit Suisse)

9/3/2006 ©2006 BMC Software27

What’s New in Content Replicator 7.0? (Cont’d)

› No More Auxiliary Channels; Now Segments» (Segments also, only for n source - n target use case)

9/3/2006 ©2006 BMC Software28

CM Deployment Manager

› What’s New in DM 7.0?

9/3/2006 ©2006 BMC Software29

What’s New in DM 7.0?

› Main Features (Architecture Changes)– Database-backed workspace– Improved DS->DM communication– Improved handling of log data– Report Center integration

9/3/2006 ©2006 BMC Software30

Previous Workspace Architecture

› File System-Based– Object-tree is stored on disk in File System hierarchy– Each object consists of a directory with several files– Entire object-tree is loaded up and kept in memory at startup

› Problems– Keeping all nodes in memory doesn’t scale

• Start-up time grows linear to workspace size• Workspace size is restricted by available memory

– Hard to: • Query data• Share data with other applications• Maintain referential integrity• Guarantee safety of the data

9/3/2006 ©2006 BMC Software31

New Workspace Architecture

› Database-Backed– Object-tree is stored in a database under DBTree

• Simple– Similar to what DM uses today (PTree). Easy to learn and replace.

• Yet powerful– Lazy loading: In-memory caching and batching of object-updates, etc.

– An object is a DBTree dir-node• All common files are merged into a set of node-properties.• Other data stored as child-nodes.• Exception: Log data is stored in the logging schema.

9/3/2006 ©2006 BMC Software32

DB-Backed Workspace

› How Does This Solve The Problems?– Workspace-size scalability

• Tree no longer needs to be traversed at start up; start up time will be minimal• Number of objects no longer limited by amount of memory

– Querying the data• Logging data is stored in logging schema; querying will be easy• Object data is stored in DBTree-schema; querying is possible but somewhat

complicated– Sharing data

• All data is in the database– Referential integrity

• Yes, we can guarantee it. – Safety of data

• Increases; databases are built to be safe datastores.

9/3/2006 ©2006 BMC Software33

DS->DM Communication

› Problems– DS communicates too frequently– Protocol is inefficient and simplistic– DS queue logs in memory

› Solutions– Tighter control of DS communication

• Intelligent flushing of queue– New, efficient protocol

• Binary • Support for compression• Support for new client commands (re-try later, etc.)

– Better use of HTTP• Be efficient about keep-alives (partly done in 603S releases)

9/3/2006 ©2006 BMC Software34

Handling of Log Data

› Problems– DM does “on-the-fly” data processing (CPU-intensive)– Incoming data is immediately processed– UI becomes unusable when many clients upload

› Solutions– Incoming data is placed in disk-based queue– Limited set of processor threads read off of queue– We get in control of CPU resources used for log-processing vs. UI– May introduce latency in respect to reporting under high load

• This should be acceptable since the alternative is an unusable UI

9/3/2006 ©2006 BMC Software35

What’s New in Deployment Reports?

› Leverage The Fact That Data Is In The DB – Use Report Center for building reports

• Separate schema for reporting• Near real-time reports (for deployment status)• After-the-fact reports

– Reports to be available in DM UI• RC query-tree is accessed remotely• Report can be selected from DM UI

9/3/2006 ©2006 BMC Software36

Q & A