Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential
Optimizing the Enterprise Manager 12c EnvironmentEnterprise Manager 12c, Release 4
Kellyn Pot’Vin-GormanConsulting Member of Enterprise Manager Technical TeamStrategic Customer ProgramMay, 2015
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Oracle Confidential 4
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 5
Program Agenda
1
2
3
4
5
Oracle Cloud Control User Interface, (Console)
Oracle Management Service, (OMS)
Oracle Management Repository, (OMR)
Metric Collections and Jobs
Weblogic and Agent Tuning
Summary6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 6
The Challenge•Slow Login Page Response
•Slow EM Page Response
•No Data Results to Screen
•Slow Refresh of Data Upon Request
•Job Delays
•Upload or Collection Delays/Not Uploading
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 7
Architecture
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 8
Program Agenda
1
2
3
4
5
Oracle Cloud Control User Interface, (Console)
6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 9
Health OverviewSetup Manage Cloud Control Health Overview
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 10
Console Performance Page
Setup Manage Cloud Control Health Overview
OMS and Repository Monitoring Page Performance
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 11
Page Performance OMS and Repository Monitoring Page Performance
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 12
Page Correlation vs. Network/OMR/OMS
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 13
Which Pages Are Highest Impacting
OMS Repository Monitoring Page Performance
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 14
Java or Agent?
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 15
SQL and Plan
SQL from Page
Plans
Compare to awrsqrpt.sql
(aka sql_id specific awr)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 16
Job Run
Not to be confused with user sessions
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 17
Load Balancers
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 18
Why a Load Balancer?
Load Balancer setup and configuration
– Users require session affinity to a particular OMS
This allows for new connections to perform Round-Robin vs. Most Requested which is less efficient.
Allows for simpler scaling for environments to support future growth.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 19
LOAD
ER STATISTICS
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 20
Distributing Work Load to Multiple Oracle Mgmt Servers
Setup Manage Cloud Control Repository
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 21
Using the Access Log
$OMS_HOME/EMGC_OMS1/access.log
•Tracks each login, along with app/module logged in from:
Use NSLookup to locate workstation name.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 22
Using EM CLI to Track UI Page
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 23
Controlling Login and Login Time
•Automatically controls logout of inactive sessions
•Single configuration setting for ALL OMS in environment-
•default is 45 minutes, but can be changed with EMCTL command:
$emctl set property -name oracle.sysman.eml.maxInactiveTime -value 30
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 24
Have You Succeeded? Bottom of Page Performance Page:
OMS and Repository Monitoring Page Performance
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 25
Program Agenda with Highlight
1
2
3
4
5
Oracle Management Service, (OMS)
6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 26
What We Think Of When We Think of EM12c
FAILED
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 27
Enterprise Level- Enterprise Manager Environment
Maximum Availability Architecture
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 28
EM12c Project Consideration
•Although it requires licensing, consider RAC-One Node for any production repository to allow for easy scalability in future.
•8Gb for a small OMR, 12Gb+ for medium and 16Gb+ for large.
•Consider ASM to assist with tablespace maintenance and enhanced management. Remember, everything is in the MGMT_TABLESPACE.
•Consider installing the OMS and OMR on separate hardware. Each requires different resources, but can compete with each other if constrained.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 29
Initialization Parameters for the Repository Database
Note the Compliance Checks
Setup Manage Cloud Control Repository
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 30
Why ODL Format Log Files Are ImportantOracle Diagnostic Log Files provide a lot of data in an aggregated format that allows you to locate what you need if you know what you are looking for!
Located in following directories for EM12c:$OMS_HOME/EMGC_OMS1$OMS_HOME/EMCG_ADMINSERVER$OMS_HOME/emoms$OMS_HOME/nodemanager$OMS_HOME/opmn_logs, etc., etc.*Agent has similar directory structures
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 31
EMGC AdminServer Diagnostics Log File
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 32
Format Translations
TimestampSource and Type
Weblogic Server Mgmt Product
Component ID
Error Message
Execution Context ID
Transaction ID
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 33
Google is our Friend
Search using the error message first, then add in filters with the context or type information to assist.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 34
Tuning Java Heap Size
JAVA heap size for the OMS is a crucial setting. This value is used to accommodate memory for each user logged in.
$OMS_HOME/user_projects/domains/GCDomain/bin/startEMServer.sh
– Use OMS_HEAP_MAX, OMS_HEAP_MIN, OMS_PERMGEN_MAX, OMS_PERMGEN_MIN parameters to change the heap settings
$emctl set property -name OMS_HEAP_MAX -value 2560
– Stay between JAVA Heap size between 2Gb and 4Gb
Default values: OMS_HEAP_MAX: 1740M OMS_HEAP_MIN: 56M OMS_PERMGEN_MAX: 768M OMS_PERMGEN_MIN: 128M
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 35
Using EM Diagnostics
Repvfy(Repository Verify)
•Run level 9 –details
•Review Jmap output file.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 36
EMGC_AdminServer.out
Includes Java Heap Settings in first five lines.
$OMS_HOME/EMGC_ADMINSERVER/EMGC_AdminServer.out
JAVA Memory arguments: -Xms256m -Xmx512m -XX:MaxPermSize=512m
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 37
Program Agenda with Highlight
1
3
2
4
5
Oracle Management Repository, (OMR)
6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 38
Repository Support
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 39
The OMR SchemaSetup Manage Cloud Control Repository
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 40
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 41
Repository Space Usage in Table View
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 42
Partition Retention
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 43
Retention Killing Your Space?
Change at Table/Partition Level
> execute gc_interval_partition_mgr.set_retention('SYSMAN', <table name>, <number of partitions to retain>);
Modify retention or view impact of changing.
> repvfy send set_purge_retention -name {ALERTS|ERROR} -id <num days>
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 44
Management Server Incidents
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 45
OMR High Redo
Very common, consider at least 500M-1G Redo logs with at least 5 groups for high demand EM environments and consider retention data
>repvfy send set_purge_retention -name CONFIGURATION -id <num days>
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 46
The “Weight” of Different Tasks
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 47
Most Important Graphs for OMR
Setup Manage Cloud Control Repository Repository Tab
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 48
Upload Performance
•Uploads are collections of data from agents to the OMS.•If backlogs occur, experiences of-• Delayed notifications• Unresponsive agents• Inaccurate information to console• Delayed response to incidents from
jobs
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 49
SQL Baselines
For the SYSMAN schema, no SQL plan baselines should get captures, and no baselines should be used. The use of SQL plan baselines can severely impact the performance of the OMS and the repository (in a very negative way).SQL> ALTER SYSTEM SET optimizer_capture_sql_plan_baselines=false SCOPE=both;
SQL> ALTER SYSTEM SET optimizer_use_sql_plan_baselines=false SCOPE=both;
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 50
Repository Performance•Average Collection Duration in Seconds
•Number of Collections in Backlog**
•Throughput per Second
•Metric Data Rollup Performance, (metric extensions included)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 51
Task Worker ThreadsThese workers process tasks and messages from the internal subsystems. These internal housekeeping jobs are executed by 2 main DBMS_SCHEDULER jobs in the repository database:
One to execute small jobs (SHORT worker pool), and one to execute larger, more time consuming jobs (LONG worker pool).
The number of 'workers' for each of these pools is configurable, and can be changed. To set the minimum of 2 workers for both the short and long running pool, use this command:
>repvfy send set_workers
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 52
Program Agenda with Highlight
1
5
2
3
4 Metric Collections and Jobs
6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 53
All Target Metrics Are Not Created Equal- Uploads by Day
Exadata Storage Server Target: 27K rows, 9.5M
Cluster Target: 23K rows, 29M
Infiniband Switch: 27K rows, 11M
Database Instance: 122K rows, 7M
Oracle Home: 8K rows, 3M
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 54
How Much Metric Data Am I Receiving?
Data can be gathered by running an EMDiag Report:
>repvfy dump metric_stats
Or….
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 55
Top Data Loading MetricsSetup Manage Cloud Control Repository Metrics Tab
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 56
Setup Manage Cloud Control Repository Metrics Tab
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 57
Demands and Consequences
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 58
Cluster Detail Bubble Map
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 59
Metric Collection ErrorsSetup Manage Cloud Control Repository Metrics Tab
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 60
Collection Performance ReconfigureUpdate from Short Running to Long Running Graphed
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 61
Metric Raw Data
Metric data represents all the metric collections that are enabled against a target (3 main settings).
Raw Data, 7 days by default. Largest data, take care in changing retention.
Once per day, raw data is aggregated by day, month and 366 days. To change this retention:
>repvfy send set_data_retention -name {RAW|HOUR|DAY} -id <num days>
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 62
Advanced Troubleshooting•Highlight a metric alert type•Hover cursor to view options to:
•Problem Analysis•Metric Details•Target Home
OMS and Repository Monitoring Metrics Collections
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 63
Problem Analysis
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 64
Files Processed Vs. Backoff
Outage
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 65
Collection Performance- Improvement?Setup Manage Cloud Control Repository
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 66
Repository Job Activity
>repvfy send set_job_retention -name <job module> -id <num days>
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 67
Optimizing OMS Jobs
•Backload of Jobs, consider updating the parameter job_queue_processes up to 10.
•Edit the schedule for the daily jobs to run in off-peak time or more often to lessen impact.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 68
Job Pool Tuning
In any environment relying on EM to execute jobs and deployment procedures, the size of the job connection pool from the OMS to the repository will have to get tuned.
To change the site of the job pool connection pool, run this command on each OMS in the environment:
>emctl set property -name oracle.sysman.core.conn.maxConnForJobWorkers-value <recommended> -module emoms
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 69
Program Agenda with Highlight
1
5
2
3
4
Weblogic and Agent Tuning
6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 70
Weblogic Server- Stuck Threads
Stuck Thread Timeout (BEA-337 incidents in the WebLogic log/trc files)
From OMS in Admin Console:
Stuck Thread Timeout = 1800
Stuck Thread Timer Interval = 120rom OMS in Admin Console:
Admin Console → Environment → Servers → EMGC_OMSx→ Tuning
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 71
Weblogic: LDAP Configuration Flag
For LDAP and SSO setups, always make sure the local LDAP can be used.
Without this set, users are unable to login, (even as SYSMAN) remotely. The console will report remote authorization and authentication as down or unavailable.To correct, set in the Admin Console:
LDAP Provider Control Flag = Sufficient
Admin Console → Security Realm -> myrealm -> Providers → LDAP Provider
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 72
Agents
•Separate Management Area from all OMS/OMR Performance Screens.
•How many targets are monitored by the agent.
•Incidents involving the agent or dependent targets.
•Last successful upload to the OMS.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 73
Agents
Setup Manage Cloud Control Agents
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 74
Agent Down Detection
To eliminate staleness of agents after downtime and allow OMS's to catch up, force the agents to send heartbeats with a more effective grace period by defining in the system. This grace period serves as a time period right after an outage during which the OMS’s will NOT report any Agent as 'down'.
SQL> exec gc_diag2_ext.SetPingGracePeriod;
To set the grace period to a specific value, use this command:
SQL> exec gc_diag2_ext.SetPingGracePeriod(90);
The value for the grace period should be between 60 and 300 seconds. Higher values should only be used if specifically instructed by Oracle.
Setup Manage Cloud Control Agents
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 75
View by Name Default
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 76
Agent HeartbeatIncoming heartbeats from agents are “batched” to allow for bulk updates in the repository. For optimal performance, the size of such a bulk operation performed by a single thread in the OMS should never exceed 1000 Agents.
The default number of threads configured per OMS is two, which should be sufficient to handle up to 2000 Agents per OMS. If the number of Agents is above this, the number of threads to do the Agent heartbeat processing should be increased:
>emctl set property -name oracle.sysman.core.omsAgentComm.ping.heartbeatPingRecorderThreads –value "3"
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 77
Auto Tuning Set in Agents
To make sure the auto tuning is enabled check the value of the 'enableAutoTuning' parameter:
>emctl getproperty agent -name enableAutoTuning
If the value is not set to true, change it, and restart the Agent.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 78
Agents with Variety of Target Types
If the Agent is monitoring a various number of different target types, the overall timeout value can be changed, to avoid having to specify the timeout for each individual target type one by one:
>emctl setproperty agent -allow_new -name dynamicPropsComputeTimeout -value 180
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 79
Agent Tuning on Incoming to OMS
Similarly, if there are a lot of targets (+50) of a given type discovered on the Agent, and those targets are administered in bulk (by applying templates, submitting jobs, or performing repetitive tasks), the number of incoming connections from the OMS to the Agent can exceed the default of 25. For those cases, the default value can be increased. Values above 50 should be extreme cases, and need to be validated and verified by Oracle.
>emctl setproperty agent -allow_new -name MaxInComingConnections –value 40
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 80
Agent Patching TipsPre-stage patches
To apply a set of patches automatically to an Agent while deploying it via the console, stage the patches in the
>$OMS_HOME/install/oneoffs/<version>/Generic/ (generic platform independent patches)
and >$OMS_HOME/install/oneoffs/<version>/<platform>/ (platform specific
patches)Every Agent path to apply will have to get copied on each of the OMS’s, to make sure that regardless which OMS does the Agent install, the pre-staged patches will get applied.
Download Agent Software and OpatchThe Agent software for the managed platforms can be downloaded using the Self Update pages in the console.Before installing additional agents for other platforms, ensure that the OPatch download job has completed. This job runs on a 24hr cycle, and will get the latest OPatch from My Oracle Support.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 81
Program Agenda with Highlight
1
5
2
3
4
Summary6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 82
Summary•Best Performance requires different skills for optimizing Enterprise Manager Cloud Control than for a database or engineered system.
•There are impressive enhancements and bug fixes in the PSU patches and yes, they exist for EM12c- the OMS, OMR and Agent. Address this FIRST!
•Don’t get overwhelmed- use the tools provided in the Manage Cloud Control and rep/agt/oms vfy to inspect the environment on a quarterly basis.
•Don’t just settle with the default values for the environment- many times these are set by development and you most often use your EM12c differently than they imagined or plugins, etc. can change usage.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 83
Staying on Top of PSU Patching
96% of issues experienced are addressed by applying OMS and Agent patches. Please remember to provide tender loving care to your Enterprise Manager…
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 84
Tip Working With MOS and Service Requests
omsvfy “Diagnostic dump” contains a file called “env”.
File contains all lib files, all environment information, ports, etc. that will be required for an service request. Upload this one file and save a LOT of time..
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 85
Want to Learn More?
DBAKevlar Blog Posts: http://dbakevlar.com/2014/10/em12c-release-4-health-overview/http://dbakevlar.com/2014/11/em12c-rel-4-oms-and-omr-health-part-ii/http://dbakevlar.com/2014/12/oms-and-omr-performance-part-iii-metrics-page/http://dbakevlar.com/2014/11/agents-management-via-em12c-release-4-console/Change Retention: http://docs.oracle.com/cd/E24628_01/doc.121/e24473/repository.htm#EMADM12785
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 86
Connect with me via Social Media: