Monitoring Large Data Centers With OEM

8/18/2019 Monitoring Large Data Centers With OEM

1/57


2/57

Strategies for Monitoring Large Data Centers with Oracle Enterprise

Manager

Ana McCollum

Consulting Product Manager


3/57

The following is intended to outline our general

product direction. It is intended for information

purposes only, and may not be incorporated into any

contract. It is not a commitment to deliver any

material, code, or functionality, and should not berelied upon in making purchasing decisions.

The development, release, and timing of any

features or functionality described for Oracle‟s

products remains at the sole discretion of Oracle.


4/57

Agenda

Overview of Oracle Enterprise Manager

Monitoring Best Practices

Q&A


5/57

Business-Driven IT Management


6/57

Business-Driven IT Management


7/57

Enterprise Manager MonitoringFundamental part of Integrated Application-to-Disk solution

Lights-out data center monitoringManage by Exception: continuous monitoring oftargets, generation of alerts when exceptions aredetected

Metric: mechanism used to monitor targetconditions (availability, performance, etc.)

Alert: generated when metric crosses its

thresholds (warning, critical)Notifications: sending of alert information (email,etc.)

Complete and integrated across stackEntire Oracle stack

Heterogeneous infrastructure monitoring by plug-ins

Extensible for custom needs

Manage Many as OneFeatures to setup and monitor many targets as one

Integrates with third party systemsHelpdesks and other management systems

E x t en d e d

I nf r a s t r u c t ur e

…

Monitoring scope


8/57

Common Monitoring Questions

What‟s the best way to...

Set Up MonitoringDeploy monitoring settings on

targets

Set up notifications for administrators Assign right level of target privileges

to administrators

Manage AlertsControlling volume of alerts

Removing unwanted alerts Automating fix for common alerts

Administrator

Data center


9/57

Best Practices for Monitoring the Data Center

Goals:

Meet monitoring requirements

Deploy your monitoring standards

Alert notifications sent to the appropriate

persons

Comply with security practices

Follow Principle of least privilege when

granting target privileges

Easy to manage

Infrastructure does not become

administrative task„Manage many as one‟ for managed

targets

Scalable as enterprise grows

Strategy:

Set up monitoring for

„economies of scale‟ by laying

the groundwork for monitoring

As enterprise grows, minimaleffort to monitor new targets or

add new Enterprise Manager

administrators

• Leverage “Manage Many as

One” features


10/57

Best Practices: Setting Up Monitoring4 Step Methodology

Step 1: Organize targets into groups

Step 2: Use Roles to segregate responsibilities

Step 3: Define and enforce monitoring standards

Step 4: Set up notifications using groups


11/57

Setting Up MonitoringSTEP 1: Organize targets into Groups

Plan your group structure

Considerations:

Group together targets monitored in

the same way

Same monitoring settings due to:

Supporting same application

Same Deployment type

(Production, Development,

Test)

Visually monitor them together in

a dashboard

Can have group hierarchies

Sample group hierarchy:

By Line of Business

By Deployment (Prod vs Devt)

By Ownership

By Line of BusinessFINANCE

…… ……

By Deployment

(Prod vs Devt)PRODUCTION DEVT

By Owner A B C D E


12/57

Create the group inEnterprise Manager

You can search targets byoperational criteria (akatarget properties)

Deployment Type, Line ofBusiness, Location…

Additional tips:Can add new targetproperties via EMCLIadd_target_property

Can bulk update targetproperties via EMCLIset_target_property_value


“Create group based onProduction database and host

targets in the Finance department ”


13/57


Make the group “privilege-propagating” A privilege on the group that isgranted to a user automaticallyextends to all members of thegroup

Includes subgroups

Requires:Create Privilege-PropagatingGroup privilege

Full privilege on all targets to beadded to the group

EMCLI modify_group verb to

convert group to privilegepropagating group

If a group is privilege propagating,all its parent groups must beprivilege propagating.

Administrator

ProductionFinance Group

ProductionGroup

ProductionSales Group

Operator

Priv i lege


14/57

Setting Up Monitoring4 Step Methodology






15/57

Setting Up MonitoringSTEP 2: Use Roles to segregate responsibilities

Who should do what on the targets in the group?Map operations on the groups to job responsibilities (SeniorLead, DBA owner, First Line Support, etc.)

Planning Considerations:For the groups created, who can do these operations on them?

Change group membership

Grant privileges on the group to other users

Who can do these operations on the targets in the group?

Add / delete the target from Enterprise Manager

Define monitoring settings

Define notification settings

View / receive notifications for alerts

Acknowledge an alert Act on target to resolve alert

Blackout target for planned or unplanned downtime


16/57


Operations Enterprise Manager Privilege

...On the Group:

•Change group membership

•Grant privileges on group to users

Group Administration

…On the member targets:

•Delete target from Enterprise Manager •Full on group

•Set blackout for planned downtime

•Change monitoring settings

•Change monitoring configuration

•View and acknowledge alerts, Clear alerts

•Operator on group

•Blackout Target

•Manage Target Metrics

•Configure Target

•Manage Target Alerts

•View target, receive alerts on target View on group

Mapping of operations to Enterprise Manager privileges


17/57


Group Administrator

• Adds / Deletes target from Enterprise Manager

• Manages group membership

• Grants privilege on group to other users

Senior Administrator

• Adds / Deletes target from Enterprise Manager

• Sets up monitoring for targets

• Sets up notification rules for targets

First Line Support

• Receives notifications for alerts

• Responds to alerts

Target Owner

• Receives alerts and responds to alerts

• Changes monitoring settings for targets

• Perform target maintenance

Examples of common job responsibilities:


18/57


Group Administrator ROLE

• “Add Any Target” system privilege

• “Group Administrator” on the group

Senior Administrator ROLE• “Add Any Target” system privilege

• “Full” on the (privilege propagating) group

First Line Support ROLE

• “Manage Target Alerts” on the (privilege propagating) group

Target Owner ROLE

• “Operator” on the (privilege propagating) group

Create roles for each job responsibility


19/57

Setting Up MonitoringSTEP 2: Use roles to segregate responsibilities

Create roles containing theappropriate privileges on theprivilege propagating group

Examples:

SeniorAdmin Role =

Full on „Production SalesGroup‟

FirstLineSupport Role=Manage Target Alerts on„Production Sales Group‟

Grant roles to administrators

who manage the groupDon’t grant privileges onindividual member targets

Harder to maintain as group grows

Production Sales Group

(Privilege-Propagating)

SeniorAdminRole

Full

FirstLineSupport Role

Manage

Target

Alerts

Senior Administrator

Junior Administrator


20/57







21/57

Setting Up MonitoringSTEP 3: Define and enforce monitoring standards

Simplify management of many targets by defining standards formonitoring (set of metrics and thresholds)

Monitoring standard for production databases

Monitoring standard for test databases

etc.

Create Monitoring Templates to encapsulate monitoring

standardsMonitoring standard for “Production‟ systems put this in onetemplate

Monitoring standard for “Test” systems put in a separate template

Monitoring Template typically contains complete set of metricsettings

Specific to a target typeExamples:

Monitoring Template for production databases

Monitoring Template for test databases


22/57


Create monitoring templateSenior Administrator creates thetemplate on behalf of the LOB / Team

Grant „View 'on template to otherconsumers of the template

Grant „Full ‟ on template only to othersenior administrators (or role) who areentitled to edit the template

Apply monitoring templates to groups Apply to highest level in grouphierarchyWill apply to the applicable targets in thegroup / subgroups

Example: Database template will only beapplied to database targets in group

Usage Notes:

Apply requires at least “Manage TargetMetrics”

Multiple templates can be applied on atarget, potentially overriding metricsettings

Target can have specific metric settingsby setting „ prevent template override‟ flag

Metric Warn CritCorr

Action

Table-

space

75 90 xxx

Archive

Area %

70 80 yyy

Monitoring

Templates

Apply

Templates

Production

Group


23/57


What if I have one monitoringstandard for all my targets?

Define Monitoring Template

Specify it as the „Default” forthe target type

For targets added in the future,will be used instead of Oracle‟sout-of-box monitoring settings

Enterprise Manager willautomatically applytemplate upon targetdiscovery

Usage notes:

Manually apply template forexisting targets

Requires Super Administratorprivilege to specify template as„Default‟

Enterprise Manager

Default Template for

database


24/57


What if I have enterprise-

wide settings and

application-specific

settings for my targets?

Use Default Template tospecify enterprise-wide

settings common to all

targets

Use another template

containing settings specificto application

Apply to target after

discovery

Default Template for

database (all)

Enterprise Manager

Database Template for

Finance Production DBs

Target applied

with Default

Template

then with

FinanceTemplate


25/57







26/57

Setting Up MonitoringSTEP 4: Set up notifications using groups

Notification Method : means of

sending notifications (e.g. email)

You can extend Notification

Methods to accommodate custom

alert handling: OS Script,

PL/SQL, SNMP traps

Notification Rule: when alert

occurs, who gets notified and how

Use groups as the target for the

notification rule

If a target is added to the group,

the notification rule willautomatically apply to the target



Notification Rule:

Target: Production Sales Group

Critical alerts

Action: Send email to DBA team


27/57

Leveraging your monitoring setup As your enterprise grows, minimal effort required for target setup

When new targets are added

to Enterprise Manager:



Apply template

Do nothing (if using Default

Templates) or apply Template

Add to appropriate group

… That‟s it!

Results:Targets are monitored accordingto your standards

Notifications for alerts on thetargets go to the rightadministrators

Administrators have the rightprivileges to manage the targets

Enterprise Manager


28/57

Common Monitoring Questions

What‟s the best way to...

Set Up MonitoringDeploy monitoring settings on

targets

Set up notifications for administrators Assign right level of target privileges

to administrators

Manage AlertsControlling volume of alerts

Removing unwanted alerts

Automating fix for common alerts

Administrator

Data center


29/57

Managing Alerts

1. Control alerts at the sourceWas the alert raised prematurely?

Are the thresholds too high/low?

Review metric trend

Adjust thresholds, set number of occurrencesDo I care about this condition? If NOT, then:

Disable metric collection schedule

Note: Other metrics may be impacted

For database alert log metric, use alert log filters

To disable alerting for database TEMP, UNDOtablespaces, see Support Note 816920.1

To deploy changed metric settings across targets, useMonitoring Templates


30/57

Managing Alerts

2. Use Correct ive Act ions toauto-resolve alerts

Tasks that automatically run inresponse to an alert

Is the resolution of the alert arepeatable process that can bescripted?

Usage Notes:

Defined for a metric Can be same or different forWarning vs Critical severity

Can have different tasks basedon monitored object

Ex: Filesystem Space

Available(%) can havedifferent corrective actionsfor /u1 and /u2

Set up notifications forcorrective action failure

Critical

Alert

Corrective

Action

Automated

AlertResolution


31/57

Managing Alerts

3. Clear old log-based alertsregularly

Think about operationalpractice for regularly clearingold, resolved alerts

Automate using duration-based notification rules

Tip: Create a separatenotification rule for this

Do not combine with rulesfor sending notifications

Manually via EMCLIclear_stateless_alerts

Bulk clears statelessalerts for a target

Alert

Log

Auto-Cleared

Alerts After N Days

DBA Fixes

Issues

Alert

raised

Automated

Manual / On Demand


32/57

Managing Alerts

Auto-clearing log-based alerts using notification rules

First sp eci fy the durat ion-based

cond i t ion …

(“Alert Log”aler ts op enedfor at least 30days)


33/57

Managing Alerts

Auto-clearing log-based alerts using notification rules

Then choose „Clear Alert‟ action


34/57

Managing Alerts

4. Perform proactivemonitoring using the

System Dashboard

Shows overall health of

your group using universal

colors of alarm

Use System Dashboard at

any level in group

hierarchy:

Highest level – visibilityinto status/performance

of ALL targets

Lower level – details of

alerts for specific group

FINANCE

…… ……

PRODUCTION DEVT

A B C D E

System

Dashboard(overview)

System

Dashboard

(details)


35/57

Managing Alerts

System Dashboard

Include metrics showing

overall health

To help triage/assign the

alert, add operational datausing target properties (e.g.

Contact, Application

Supported, etc.)

Latest comment for alert is

shown. Use this to:Display ticket ID

Show alert ownership

System Dashboard

Alert Details

Target

Status Add operationsdata e.g.Owner


36/57

More Monitoring Tips in the Appendix

Refer to Appendix for information on additional topics:

Choosing metrics and thresholds for alerting

Customizing email to add more operational context

Practices for setting up notification rules

Sending alert reminders using repeat notificationsEscalations through email

Did my fix resolve the alert?

Is this old alert still valid?

Are my targets following my monitoring standards?Do I have correct notification coverage for my target's alerts?

Changing dbsnmp credentials across many databases


37/57

Benefits: Enterprise Manager MonitoringEnabling value through best practices

Minimal effort to scale as

enterprise grows

Reduce manual tasks

Improve administrator

productivity

Flexible to meet monitoringand security requirements

Enables IT to meet servicegoals

Standardized approach to

monitoring Manage More with Less


38/57

Oracle Enterprise Manager 11g Resource Center

Access Videos, Webcasts, White Papers, and More

Oracle.com/enterprisemanager11g


39/57

Additional Oracle Enterprise Manager Sessions

Thursday, Sept. 23 Location

9:00 a.m. - Oracle WebLogic Server Management for Oracle

DBAs

Marriott Marquis,

Salon 9

9:00 a.m. - Enabling Database as a Service Through Agile Self-

Service Provisioning

• Moscone S. Room

102

9:00 a.m. - Reduce TCO with Oracle Application Management

Pack for Oracle E-Business Suite

• Moscone W L2, Rm

2024

10:30 a.m. - Best Practices for Managing Your PeopleSoft

Applications

Marriott Hotel, Golden

Gate A

10:30 a.m. - Oracle Enterprise Manager Grid Control

Deployment Best Practices

Moscone S. Room

102

10:30 a.m. - Managing Sun SPARC Servers with Oracle

Enterprise Manager Ops Center

Moscone S. Room

252

10:30 a.m. - Heterogeneous Data Masking: Oracle, SQL Server,

and DB2 Database Best Practices

Moscone S. Room

306

12:00 p.m. - Scalable Enterprise Data Processing for the Cloud

with Oracle Grid Engine

Moscone S. Room

310

12:00 p.m. - Spot Problems Before Your Users Call: User

Experience Monitoring for Oracle Apps

Marriott Hotel, Golden

Gate A

12:00 p.m. - Reduce Problem Resolution Time with Oracle

Database 11g Diagnostic Framework

Moscone S. Room

102


40/57

Additional Oracle Enterprise Manager Sessions

Thursday, Sept. 23 Location

1:30 p.m. - Patching Enterprisewide Databases: Automation

Techniques and Real-World Insights

Moscone S. Room

310

• 1:30 p.m. - Managing User Experience: Lessons from eBayMarriott Hotel, Golden

Gate A

1:30 p.m. - Deep Java Diagnostics and Performance Tuning:

Expert Tips and Techniques

Marriott Marquis,

Salon 9

1:30 p.m. - Oracle Enterprise Manager Configuration

Management Unleashed: Top 10 Expert Tips

Marriott Marquis,

Salon 6

1:30 p.m. - Oracle Enterprise Manager Security Best PracticesMoscone S. Room

102

3:00 p.m - The X-Files: Managing the Oracle Exadata and

Highly Available Oracle Databases

Moscone S. Room

102

3:00 p.m. - Monitoring and Diagnosing Oracle RAC

Performance with Oracle Enterprise Manager

Moscone S. Room

310


41/57

Oracle Enterprise Manager Hands On Labs

Thursday September 23, 2010

9:00 - 10:00 a.m. Database Performance Diagnostics and Tuning Marriott Hotel,Salon 12/13, YB

Level

10:30 - 11:30 a.m. Oracle Fusion Middleware Management Marriott Hotel,

Salon 12/13, YB

Level


42/57

Oracle Enterprise Manager DemogroundsD E M O T I T L E L O C A T I O N

Oracle Real Application Testing: Database Replay Moscone West

Oracle Real Application Testing: SQL Performance Analyzer Moscone West

Self-Managing Database: Automatic Performance Diagnostics Moscone West

Self-Managing Database: Automatic Fault Diagnostics Moscone West

Self-Managing Database: Automatic Application and SQL Tuning Moscone West

Application Quality Management: Application Testing Suite Moscone South - S022

Real User Monitoring with Oracle Enterprise Manager Moscone South - S021

Siebel CRM Application Management Moscone South - S024

Real User Monitoring with Oracle Enterprise Manager Moscone WestOracle WebLogic Server Management and Java Diagnostics Moscone West

SOA Management with Oracle Enterprise Manager Moscone West

Oracle Business Transaction Management Moscone West

Push Button Provisioning and Patch Automation Moscone West

Smart Configuration Management Moscone West

Oracle Enterprise Manager Ops Center Moscone West

Managing the Enterprise Private Cloud Moscone West

System Management, My Oracle Support, and Oracle Enterprise Manager Moscone West

Self Managing Database: Change Management for DBAs Moscone West

Oracle Enterprise Manager: Complete Datacenter Management Moscone West

Self-Managing Database: Data Masking for DBAs Moscone West


43/57

Appendix


44/57

Setting up Monitoring: Tips and TrapsChoosing metrics and thresholds for alerting

Default thresholds may over-alert

Some defaults designed for PRODUCTION use cases

Use template with adjusted thresholds to apply to DEV and TEST

Choose metrics for alerting carefully:

Time-based metrics superior for performance

Base resource consumption (CPU, I/O, Memory)

Workload or application-specific metrics or health checks

Threshold valuesUse metric history to analyze value ranges

Be conservative with critical thresholds:

Reserve CRITICAL for high signal of serious problem


45/57

Database Performance Metrics

#1 Metric: Average Active Sessions

Measures active load on database instance

Sudden high spikes usually mean severe performance issue

Use Adaptive Thresholds:

Sets thresholds automatically and adjusts for workload cycles

Warn at 0.99 significance (measured sample statistic)

Critical at 0.9999 significance (estimated high significance value)

11g: New Adaptive Thresholds user interfaceSupports threshold “what-if” analysis over recent history

Organizes metrics into Classes

Located under “Baseline Metric Thresholds” in Grid/Database Control


46/57

Setting Up Monitoring: Beyond the Basics

Additional tips for setting up notifications

Customize email format to add more operationalcontext

Add target properties (Line of Business, Owner,Contact..) in the email to provide additionaloperational information

Practices for setting up Notification RulesDesignate users (Senior Administrators) to createrules on behalf of the team

Common use cases

Rules for production targets different than rules fornon-production targets

Separate rules by Line-of-Business / team

Use naming convention ( e.g. include team namein rule)

Facilitates searching for rules


47/57



Send alert reminders using Repeat NotificationsSet the global defaults to the least frequent interval used anduse this in most rules

Example: Global setting: Repeat every 30 minutes up toa max of 3 repeats

For rules that include important targets or critical alerts, set tohigher repeat frequency

Example: For target down rule(s): repeat every 5 minutesup to a max of 10 repeats

Usage Note: Users need to „acknowledge‟ the alert in the

Enterprise Manager console to stop repeat notificationsRemember to provide “Manage Target Alerts” privilege toyour operators


48/57



Escalate unattended, important alerts via emailSend email to different person (e.g. manager level) if alert is

open too long

To set up this rule:

Create new notification rule and put a „duration condition‟associated with the alert

Rule action: Send email to the manager


49/57

Notification Rule for Alert Escalation via E-mail

In Metrics tab, define

duration condition for

the alert…

(Apply rule if alert

opened > 12 hoursand not

acknowledged)

.. then in Actions tab,

send email to the

manager


50/57

Managing Alerts: Other Tips

How do I know if my fixresolved the alert?

Use „Reevaluate Alert‟

feature

Alternative to waiting fornext metric evaluation

Causes the agent to

reevaluate the metric alert

Current severity will be

providedUsage: requires 10.2.0.5

agent or higher


51/57

Managing Alerts: Other Tips

Is this alert still valid?Enterprise Manager does

not change the alert

triggered date

Validation shown in the

Console

Last Collected Value

Last Collected

Timestamp

Alert triggered: June 18, 2010

Last Collected

Value

Last Collected

June 29, 2010


52/57

Monitoring: Ongoing Maintenance

Are my targets still following my monitoringstandards?

Generate report using Monitoring Template Comparison

reporting element and/or

Use “Compare Settings” feature in Monitoring Templatespage

Shows differences between monitoring template and target‟s

settings


53/57


Report using the Monitoring Template Comparison element

Differences arehighlighted


54/57


Do I still have correctnotification coverage for mytarget?

Notification Rule Coveragereport (per target)

For each metric contained ina rule:

Alert severities covered

Rule(s), if any

Type of notification

Shows alert-able metrics not covered in any rule

Potential missednotification


55/57


What‟s the easiest way to change monitoringcredentials (e.g. dbsnmp) across many databases?

EMCLI update_db_password

Changes password associated with the user in Enterprise

Manager and database targetChanges the password across all features that use it:

Preferred credentials, Corrective Actions, Jobs, User-

defined metrics, target monitoring credentials

Usage tip: Blackout the target during this operation to avoid

metric collection errors due to invalid password


56/57

AQ &


57/57

Documents

Monitoring Large Data Centers With OEM