66

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | EXADATA HEALTH AND RESOURCE USAGE MONITORING Including the Benefits of Administration

Embed Size (px)

Citation preview

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

EXADATA HEALTH AND RESOURCE USAGE MONITORINGIncluding the Benefits of Administration Tuning Methods

Kellyn Pot’VinConsulting Member of Enterprise Manager Technical TeamStrategic Customer ProgramMike ChafinPrincipal Member of Technical StaffMaximum Availability Architecture

Oracle Open World 2014

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Program Agenda

1

2

3

4

5

Oracle Tuning Methodologies

Preparing for Tuning

Working through Tuning Exercise

Digging Deeper

Miscellaneous Tips

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Program Agenda with Highlight

1

2

3

4

5

Oracle Tuning Methodologies

Preparing for Tuning

Working through Tuning Exercise

Digging Deeper

Miscellaneous Tips

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Evolution of Methodologies

Documented in the Oracle Documentation Set:

Database 2 Day + Performance Tuning Guide

Oracle Database Performance Tuning Guide

Exadata Performance Tuning White Paper

Each are similar, in that they offer a structural approach to problem solving using the latest tools and technologies.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Tuning Level Legend

What level is the Issue?

• Hardware

• Database

• Schema/Design

• SQL

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Important Tuning Concepts

Tuning a database by “DBTime”

DBTime is Oracle’s term for total time used by active CPU, IO and other foreground processes from the total time elapsed to calculate wait events that are performed within the database.

“Tune for Time or You’re Wasting Time…”

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Program Agenda with Highlight

1

2

3

4

5

Oracle Tuning Methodologies

Preparing for Tuning

Working through Tuning Exercise

Digging Deeper

Miscellaneous Tips

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Tuning Exercise Steps

• Prepare

• Compare

• Review (research, investigate)

• Rule out, Test

• Implement/Resolve

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Prepare

Collect hardware and software healthchecks • exachk, cluvfy, emdiag • Healthchecks• AWR Baseline • Verify and/or set AWR retention period

Copy configuration files• Automate collection and base-lining (EM jobs)

Create AWR, ASH and ADDM reports for research.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Stopping the Insanity Flowchart

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

The Value of Quick Check Lists

Old School, but VALUABLE.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Program Agenda with Highlight

1

3

2

4

5

Oracle Tuning Methodologies

Preparing for Tuning

Working through Tuning Exercise

Digging Deeper

Miscellaneous Tips

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Review

• Errors in the system?

• EM Incident Manager

• Alert Log(s)

• Grid Infrastructure

• ASM

• Database

• OS logs / system logs

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

AND Rule Out- Hardware

• Compute Node CPU

• I/O Bound / at known limits

• Exadata – Flash cache

• Mixed Workloads creating contention?

• ALL Components up!

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Exachk- See Oracle Support Document 1070954.1: “Oracle Exadata Database Machine Exachk or HealthCheck”

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Exachk and Healthcheck Utility Checks…

• Incorrect configurations

• Parameter Issues

• Recommendations, steps to correct, along with current value.

See Oracle Support Document 1070954.1: “Oracle Exadata Database Machine Exachk or HealthCheck” and Exadata best practices in Oracle Support Document 757552.1: “Oracle Exadata Best Practices”

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Automate Exachk via Enterprise ManagerAutomating will populate healthchecks to UI!

Need to insert screenshot

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Storage Grid OverviewStorage Cell monitoring and administration support

–Cell Home page and performance pages

–Actions supported: Start/stop Cell, verify connectivity, setup SSH

Automatic discovery of Cells

Management by Cell Group

–All cells used by a database automatically placed in a group

–Cell Group level administration operations (batch job monitoring)

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

More Storage Grid Overview

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Max Cell Disk Limits

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Hardware- Cell Server View

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Hardware- DB Node View

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

A Review of Hardware/Network troubleshooting steps• HA can mask problems, reduced capacity even while

meeting service levels

• Hardware Utilization Issues

• CPU is not what it may seem

• I/O Subsystem (Cell screens)

• Network Congestion (IB screens)

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Comparison Tools

• Incredible Power in Knowing What’s Changed.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

CompareIf there is a problem what has changed? And who might know?

Considerations• Patch levels (everywhere!)• Schema• Tunable OS parameters • Resource Management Plans• Code Changes• ADDM Comparison Report

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Compare Configurations- Exadata Level

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Compare Configuration- Database Level

• EM Job to compare one ‘reference’ database against one or more other databases

• Job can be scheduled on a repetitive basis, or run ad-hoc

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Compare Configurations- Schema Level

Capture Schema baselines

Compare schema’s

• With a baseline

• Between different databases

Synchronize schema’s

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Compare ADDM

• Full ADDM analysis across two AWR snapshot periods• Detects causes, measure effects, then correlates them

Causes : Workload changes, Configuration changes Effects : Regressed SQL, Reach resource limits (CPU, I/O, memory, interconnect)

• Makes actionable recommendations along with quantified impact

AWR Snapshot Period 1

AWR Snapshot Period 2

Analysis ReportCompare Period ADDM

SQL Commonality

Regressed SQL

I/O Bound

Undersized SGA

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

IO Issues

What type of IO Issues are addressed with Exadata features?How can temp tablespace usage be an impact to performance in an Exadata?

• Identify IO Issues- ASH Analytics, ADDM Comparison Report

• Isolate IO Issues- Search SQL, ASH/ADDM Reports,

• Address IO Issues- IO Resource Manager, SQL Tuning Advisor,

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Exadata and SQL Monitor

Offload Percent Efficiency Displayed in Report and UI.• Automatically monitors long running SQL• Enabled out-of-the-box with no performance overhead• Monitors each SQL and PL/SQL execution • Shows global PL/SQL and SQL level statistics

• Drill-down to slow SQL for diagnosing unexpected PL/SQL behavior

• Guides tuning efforts

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

IO Bound Dashboard in ADDM Comparison• Base vs. comparison period• Temp reads/writes specified• Single block read latency

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Setting Thresholds

Thresholds with incident rule sets to notify when growth in resource occurs and impacts the environment.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

IO Resource Manager

• IO Resource Manager, (IORM) offers ability to maintain consistent performance levels across an engineered system.

• Allocates and limits IO per settings configured

• Simple configuration access via Enterprise Manager

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

IO Resource Management

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Enterprise Manager Exadata Resource Management

• Exadata plugin provides access to manage IO Resources for db and across cell nodes.

What IO resources on Exadata are shared vs. on a non-Exadata RAC environment?

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Monitoring IORM

• Allows for throttling by service and IO type.

• Ensures that resources that require the IO, get the IO.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Monitoring Disk Utilization

• Ability to monitor all disk usage.

• Broken up by type

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Why Setting Limits are Important

• No Hard Limits equate to IO being freely redistributed

• Setting limits ensures that resources are available for critical databases

• Less high priority databases aren’t able to allocate more than necessary

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Program Agenda with Highlight

1

4

2

3

5

Oracle Tuning Methodologies

Preparing for Tuning

Working through Tuning Exercise

Digging Deeper

Miscellaneous Tips

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

AWR Baselines

• Collection of snapshots used for performance comparisons.

• Baselines are retained within the AWR even after the retention time for the data has been reached.

• Exadata should have a moving and a static baseline in place to capture different workloads.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

AWR Baselines

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Details of Baseline Creation in EM

Highlight a snapshot range to use for your static baseline or use a time range to base it from.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Completing Baseline Creation in EM

Schedule the job to run in a non-impacting schedule and choose not to purge.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

SQL Issues

SQL Tuning – what kind of problem?• Reactive• Transient• Long Running SQL• Proactive• Persistent• Comparative

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Proactive SQL Tuning Choices

• SQL Performance Analyzer• SQL Access advisor• ASH and ADDM to prioritize further work• Group operations (DB12)• Create an SLA / baseline• Use SQL Plan Baselines (SPM)• Validate with SQL Performance Analyzer

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Reactive SQL Tuning Choices

After all changes, including any environments are ruled out…

• Start with AWR, ADDM and ASH

• Identify outliers / worst performing (Pareto)

• Review top wait events

• SQL Tuning Advisor

• If resource constrained consider resource limiting strategies

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Database Performance Page in EM12c

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

ASH Analytics

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

ASH Analytics

Requirements for 10g and 11g databases:• Additional PL/SQL packages to be installed into the DBSNMP schema• CONNECT, CREATE TYPE and CREATE PROCEDURE privileges• Submit the job to install as SYS from the UI

View information in real-time in the database

Save a snapshot as HTML (or email the report) for offline analysis• Active report, user can visualize data on multiple dimensions, (or run from command line for text version)

Help videos available: Use Active Session History (ASH) Analyticshttps://apex.oracle.com/pls/apex/f?p=44785%3A24%3A0%3A%3ANO%3A24%3AP24_CONTENT_ID%2CP24_PREV_PAGE%3A5451%2C1

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Advisor CentralAll database advisors grouped together on one launch page:

• Advisors, focused on recommendations

• Checkers, focused on diagnostics

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

SQL Advisors• SQL Tuning Set

– Capture of SQL in the database

– Ideally run after a representative workload has been run

• Access Advisor– Analyses access patterns of SQL in the cache, or from a defined workload, and gives

recommendations on how to (re)structure the database objects

• Performance Analyzer– Analyses before and after images of SQL in a SQL Tuning Set (Testing scenarios)

– Compares results

• Tuning Advisor– Requires Tuning Sets (Collection of SQL from the database)

– Analyses the SQL from a SQL Tuning Set and gives recommendations

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

AWR Warehouse and Exadata

AWR Warehouse

Scheduler jobs with Agent to Agent push to AWR Warehouse Target, scheduled by EM12c EM Job Service.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Query Across Entire Exadata to Show True Consolidation Value

Adapt Valuable Exadata Specific Queries to AWR Warehouse-

from dba_hist_sys_time_model stm, dba_hist_snapshot s, gv$parameter p, dbsnmp.caw_dbid_mapping m

where stm.stat_name in ('DB CPU','background cpu time')

and LOWER(m.target_name)= '&dbname'

and s.dbid= m.new_dbid and s.snap_id = stm.snap_id

and s.dbid = stm.dbid and s.instance_number = stm.instance_number

and p.name = 'cpu_count' and p.inst_id = s.instance_number)

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

IO Information Across Databases on Exadata?

16:5916:5917:2917:2917:5917:5918:3018:3019:0019:0019:3019:3020:0020:000

50000

100000

150000

200000

250000

300000

350000

400000

June 10, 2013

l_reads_s read_iops_maxread_iops_direct read_iops_direct_max

20:3020:30

20:5920:59

21:3021:30

22:0022:00

22:3022:30

23:0023:00

23:3023:30

23:5923:59

0:290:29

0

100000

200000

300000

400000

500000

600000

June 10, 2014

l_reads_s read_iops_maxread_iops_direct read_iops_direct_max

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Answer Specific IT Questions

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Updating Existing AWR Queries to Go Across Hosts

select * from (selectm.target_name,sum(CPU_TIME_DELTA),sum(DISK_READS_DELTA),count(*)fromDBA_HIST_SQLSTAT a, dba_hist_snapshot s, dba_hist_database_instance di, dbsnmp.caw_dbid_mapping mwhere di.host_name='&host'and di.dbid in m.new_dbidand m.new_dbid = a.dbidand a.snap_id = s.snap_idand s.begin_interval_time > sysdate -120group by m.target_nameorder bysum(CPU_TIME_DELTA) desc)

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Graphing CPU Usage Per DB for One Host

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Program Agenda with Highlight

1

5

2

3

4

Oracle Tuning Methodologies

Preparing for Tuning

Working through Tuning Exercise

Digging Deeper

Miscellaneous Tips

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Making Copies of Configuration Files

Copy and automate copying of essential database files via Enterprise Manager-

• All instance/database init and spfiles• Database password files• ASM init files• ASM password files

Create an Enterprise Manager job to copy these files on a regular basis.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Achieving the tao of Exadata

Exadata is an Engineered System- Powerful!

• Perform Exachk reviews quarterly, (preferably monthly.)

• Install Enterprise Manager Plug-in for advanced monitoring and enhanced performance impact detection.

• Take advantage of the great performance and tuning features included as part of the diagnostic and tuning packs

• Use IO Resource Manager to deter from impact to critical database systems.

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Connect with me via Social Media: