Oracle AWR Data mining

AWR Data mining

Yury Velikanov Senior Oracle DBA

2 © 2011 Pythian

Why Companies Trust Pythian

• Recognized Leader:• Global industry-leader in remote database administration services and consulting for

Oracle, Oracle Applications, MySQL and SQL Server

• Work with over 150 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments

• Expertise:• One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ

6 Oracle ACEs/ACE Directors.

• Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC.

• Global Reach & Scalability:• 24/7/365 global remote support for DBA and consulting, systems administration, special

projects or emergency response

http://www.upenn.edu/

http://www.mdsinc.com/

© 2009/2010 Pythian

Mission

Let you remember/consider AWR next time you troubleshoot

Performance issue!

3

© 2009/2010 Pythian

NOTE

AWR = STATSPACK = Performance Repository

4

Excerpt from11GR2:$OH/rdbms/admin/awrrpt.sql

“Rem This report is based on the Statspack report.”

© 2009/2010 Pythian

AWR Agenda

• Introduction & Background

• Examples, Examples, Examples

• Concept & Approach

• More examples

• Q & A

5

Google: Oracle YuryBlog, Twitter, Linkedin, ACE … email, phone

number

© 2009/2010 Pythian

• Google Yury Oracle [LinkedIn, twitter, blog, email, mobile, …]

- Email me to get the presentation

• Sr. Oracle DBA at Pythian, Oracle ACE and OCM• Started as Oracle DBA

- with 7.2 (in 1997, 14+)• First international appearance

- 2005 - Hotsos Symposium 2005• Professional Education

- Jonathan Lewis (2003 – 3 days), Tom Kyte (2004 – 3 days), Tanel Põder (2008 – 2 days ), Cary Millsap (2011 – 3 days) …

• Education (Master Degree in Computer science)- OCP 7/8/8i/9/10/11 + OCM 9i/10g/11g

• Oracle DBA consultant experience (14+ years)• Pythian Oracle Clients support (2+ years)

- 140+ Clients around the world- Different products, different load, different problems

6

Few words about Yury

© 2009/2010 Pythian

Background

• AWR is one of many RDBMS performance data sourcesJonathan Lewis / Tom Kyte / Tanel Põder / Cary Millsap• Sometimes it isn’t the best source (aggregation)

• SQL Extended trace (event 10046)• RAW trace• tkprof• TRCAnlzr [ID 224270.1]• Method-R state of art tools

• PL/SQL Profiler• LTOM (Session Trace Collector)• others

• Sometimes it is the best source!• Sometimes it is the only one available!

7

© 2009/2010 Pythian

Background

• Once I was called to troubleshoot high load• Connected to the database I saw 8 active processes

running for 6 hours in average at the time I connected• I switched 10046 event for all 8 collected 15 minutes

data and analyzed it one by one• Found several SQLs returning 1 row million times• Passed the results to development asking to fix the logic• Spent ~2 hours to find where the issue was

• Next day a workmate asked me• Why did you use 10046 and spent 2 hours?• He used AWR report and came up with the same

answer in less than 5 minutes

• Lesson learned: Right tool for the right (job - no) case !8

orawski

colleague

© 2009/2010 Pythian

When should you consider AWR mining?

• General resource tuning (high CPU, IO utilization)• You are asked to reduce server load X times

• You would like to analyze load patterns/trends

• You need to go back in time and see how things progressed

• You don’t have any other source of information

• Existing official AWR interface doesn’t provide you information at the right angle/dimension or are not available (Grid Control, awrrpt.sql)

• AWR SQL Execution Plans historical information analysis

9

© 2009/2010 Pythian

TOP CPU/IO Consuming SQLs ?select

s.SQL_ID, sum(CPU_TIME_DELTA), sum(DISK_READS_DELTA),count(*)

from DBA_HIST_SQLSTAT

group by SQL_ID

order by sum(CPU_TIME_DELTA) desc

/

SQL_ID SUM(CPU_TIME_DELTA) SUM(DISK_READS_DELTA) COUNT(*)------------- ------------------- --------------------- ----------05s9358mm6vrr 27687500 2940 1f6cz4n8y72xdc 7828125 4695 25dfmd823r8dsp 6421875 8 153h1rjtcff3wy1 5640625 113 192mb1kvurwn8h 5296875 0 1bunssq950snhf 3937500 18 157xa8wfych4mad 2859375 0 2...

10

© 2009/2010 Pythian

TOP CPU Consuming SQLs ? select

s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA),count(*)

from DBA_HIST_SQLSTAT s

group by

s.SQL_IDorder by

sum(s.CPU_TIME_DELTA) desc

11

© 2009/2010 Pythian

TOP CPU Consuming SQLs ?select * from (select


from DBA_HIST_SQLSTAT s

group by

s.SQL_IDorder by

sum(s.CPU_TIME_DELTA) desc)where rownum < 11/

12

© 2009/2010 Pythian



from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p

where 1=1and s.SNAP_ID = p.SNAP_ID

and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16

group by

s.SQL_IDorder by

sum(s.CPU_TIME_DELTA) desc)where rownum < 11/

13

© 2009/2010 Pythian



from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p

where 1=1and s.SNAP_ID = p.SNAP_ID

and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16

and p.END_INTERVAL_TIME between SYSDATE-7 and SYSDATE

group bys.SQL_ID

order by sum(s.CPU_TIME_DELTA) desc

)where rownum < 11/

14

© 2009/2010 Pythian



from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p, DBA_HIST_SQLTEXT t

where 1=1and s.SNAP_ID = p.SNAP_IDand s.SQL_ID = t.SQL_IDand EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16and t.COMMAND_TYPE != 47 –- Exclude PL/SQL blocks from outputand p.END_INTERVAL_TIME between SYSDATE-7 and SYSDATE

group bys.SQL_ID

order by sum(s.CPU_TIME_DELTA) desc

)where rownum < 11/

15

© 2009/2010 Pythian

TOP CPU Consuming SQLs ?

16

52.8 %

1.2. 3.

4.

5.

© 2009/2010 Pythian

TOP CPU Consuming SQLs ?select

SQL_ID, sum(CPU_TIME_DELTA), sum(DISK_READS_DELTA),count(*)

from DBA_HIST_SQLSTAT

group by SQL_ID

order by sum(CPU_TIME_DELTA) desc

/

SQL_ID SUM(CPU_TIME_DELTA) SUM(DISK_READS_DELTA) COUNT(*)------------- ------------------- --------------------- ----------05s9358mm6vrr 27687500 2940 1f6cz4n8y72xdc 7828125 4695 25dfmd823r8dsp 6421875 8 153h1rjtcff3wy1 5640625 113 192mb1kvurwn8h 5296875 0 1bunssq950snhf 3937500 18 157xa8wfych4mad 2859375 0 2...

17

© 2009/2010 Pythian

5 Slides ofConcept & Approach

18

© 2009/2010 Pythian

AWR = DBA_HIST_% Views +

• 111 Views in 11.2.0.2.0

• I use just few on a regular basis• DBA_HIST_ACTIVE_SESS_HISTORY• DBA_HIST_SEG_STAT• DBA_HIST_SQLSTAT• DBA_HIST_SQL_PLAN• DBA_HIST_SYSSTAT• DBA_HIST_SYSTEM_EVENT

• Most of the views contain data snapshots from V$___ views

• DELTA columns (e.g. DISK_READS_DELTA)• DBA_HIST_SEG_STAT• DBA_HIST_SQLSTAT

19

- V$ACTIVE_SESSION_HISTORY- V$SEGMENT_STATISTICS- V$SQL- V$SQL_PLAN- V$SYSSTAT ( ~SES~ )- V$SYSTEM_EVENT ( ~SESSION~ )

© 2009/2010 Pythian

AWR Things to keep in mind …

• The data are just snapshots of V$ views

• Data collected based on thresholds• Some data is excluded based on thresholds

• Some data may not be in SGA at the time of snapshot• Longer time difference between snapshots more

data got excluded• For data mining use ALL snapshots available

20

Begin

Endt

© 2009/2010 Pythian


• Forget about AWR if there are constants in the code• Indicator is high parse count (hard) (10-50 per/sec)• It isn’t just hard parsing! (related bugs)• cursor_sharing = FORCE

• In RAC configuration do not forget INST_ID column in joins

• Most of the V$ (DBA_HIST) performance views have incremental counters. END - BEGIN values

• You may get wrong results (sometimes negative)• Sometimes counters reach max value and get reset• Counters got reset at instance restart time

• Time between snapshots may be different• Suggestion (ENDv - BEGINv)/(ENDs - BEGINs)=value/sec

21

© 2009/2010 Pythian


22

2011.10.22 10:39:24

2011.10.22 10:40:24

2011.10.22 10:41:24

2011.10.22 10:42:24

2011.10.22 10:43:24

2011.10.22 10:44:24

2011.10.22 10:45:24

2011.10.22 10:46:24

2011.10.22 10:47:24

2011.10.22 10:48:25

2011.10.22 10:49:25

2011.10.22 10:50:25

2011.10.22 10:51:25

2011.10.22 10:52:25

2011.10.22 10:53:25

2011.10.22 10:54:25

2011.10.22 10:55:25

2011.10.22 10:56:25

2011.10.22 10:57:25

2011.10.22 10:58:25

2011.10.22 10:59:25

2011.10.22 11:00:25

2011.10.22 11:01:25

2011.10.22 11:02:25

2011.10.22 11:04:26

2011.10.22 11:05:26

2011.10.22 11:06:26

2011.10.22 11:07:26

-800000

-600000

-400000

-200000

0

200000

400000

600000

800000

2011.10.22 10:39:24

2011.10.22 10:40:24

2011.10.22 10:41:24

2011.10.22 10:42:24

2011.10.22 10:43:24

2011.10.22 10:44:24

2011.10.22 10:49:25

2011.10.22 10:50:25

2011.10.22 10:51:25

2011.10.22 10:55:25

2011.10.22 10:57:25

2011.10.22 10:58:25

2011.10.22 11:04:26

2011.10.22 11:05:26

2011.10.22 11:06:26

2011.10.22 11:07:260

100

200

Microsoft Excel 97-2003 Worksheet

© 2009/2010 Pythian


• Seconds count between 2 timestamps

select s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME, s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME DTIME, -- Returns “Interval”

EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) H, EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) M, EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) S,

EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+ EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+ EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) SECS,

phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) -– Write you own fun() (cast(s.END_INTERVAL_TIME as date) - cast(s.BEGIN_INTERVAL_TIME as date)) *24*60*60from DBA_HIST_SNAPSHOT swhere 1=1 and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE)order by s.BEGIN_INTERVAL_TIME;

23

© 2009/2010 Pythian

AWR Things to keep in mind …select SNAP_INTERVAL, RETENTION from

DBA_HIST_WR_CONTROL c, V$DATABASE dwhere

c.DBID = d.DBID;

SNAP_INTERVAL RETENTION------------------------------ ------------------------------+00000 01:00:00.0 +00007 00:00:00.0

select DBID, INSTANCE_NUMBER, count(*) C, min(BEGIN_INTERVAL_TIME) OLDEST, max(BEGIN_INTERVAL_TIME) YUNGESTfrom DBA_HIST_SNAPSHOTgroup by DBID, INSTANCE_NUMBER;

DBID INSTANCE_NUMBER C OLDEST YUNGEST---------- --------------- ---------- ------------------------- -------------------------3244685755 1 179 13-AUG-11 07.00.30.233 PM 21-AUG-11 05.00.01.855 AM3244685755 2 179 13-AUG-11 07.00.30.309 PM 21-AUG-11 05.00.01.761 AM

24

© 2009/2010 Pythian

Trends Analysis Example (1) …

select s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME, ( t.VALUE-

LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) ) DVALUE,

(t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/ phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) VAL_SECfrom DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT twhere 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = 'parse count (hard)'order by s.BEGIN_INTERVAL_TIME;

25

DBA_HIST_SYSSTAT & DBA_HIST_SYSTEM_EVENT

© 2009/2010 Pythian 26


© 2009/2010 Pythian

select s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME, ( t.VALUE-

LAG (t.VALUE) OVER (ORDER BY s.END_INTERVAL_TIME) ) DVALUE,

(t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.END_INTERVAL_TIME))/ phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) VAL_SECfrom DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT twhere 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = 'parse count (hard)'order by s.END_INTERVAL_TIME;

27

DBA_HIST_SYSSTAT & DBA_HIST_SYSTEM_EVENT


© 2009/2010 Pythian

SQL Bad performance Example (2) …

• Called by a client to troubleshoot a SQL with bad performance

• Sometimes the SQL hangs (never finishes) and needs to be killed and re-executed

• Upon re-execution, it always finishes successfully in a few minutes

• The client demanded a resolution ASAP …

28

© 2009/2010 Pythian

select st.SQL_ID , st.PLAN_HASH_VALUE , sum(st.EXECUTIONS_DELTA) EXECUTIONS, sum(st.ROWS_PROCESSED_DELTA) CROWS, trunc(sum(st.CPU_TIME_DELTA)/1000000/60) CPU_MINS, trunc(sum(st.ELAPSED_TIME_DELTA)/1000000/60) ELA_MINSfrom DBA_HIST_SQLSTAT stwhere st.SQL_ID in ('5ppdcygtcw7p6','gpj32cqd0qy9a') group by st.SQL_ID , st.PLAN_HASH_VALUEorder by st.SQL_ID, CPU_MINS;

29

DBA_HIST_SQLSTAT


© 2009/2010 Pythian

SQL_ID PLAN_HASH_VALUE EXECUTIONS CROWS CPU_MINS ELA_MINS------------- --------------- ---------- ---------- ---------------- ----------------5ppdcygtcw7p6 436796090 20 82733 1 35ppdcygtcw7p6 863350916 71 478268 5 115ppdcygtcw7p6 2817686509 9 32278 2,557 2,765

gpj32cqd0qy9a 3094138997 30 58400 1 3gpj32cqd0qy9a 1700210966 36 69973 1 7gpj32cqd0qy9a 1168845432 2 441 482 554gpj32cqd0qy9a 2667660534 4 1489 1,501 1,642

30

DBA_HIST_SQLSTAT


© 2009/2010 Pythian

select st.SQL_ID , st.PLAN_HASH_VALUE , sum(st.EXECUTIONS_DELTA) EXECUTIONS, sum(st.ROWS_PROCESSED_DELTA) CROWS, trunc(sum(st.CPU_TIME_DELTA)/1000000/60) CPU_MINS, trunc(sum(st.ELAPSED_TIME_DELTA)/1000000/60) ELA_MINSfrom DBA_HIST_SQLSTAT stwhere st.SQL_ID in ('5ppdcygtcw7p6','gpj32cqd0qy9a') group by st.SQL_ID , st.PLAN_HASH_VALUEorder by st.SQL_ID, CPU_MINS;

31

DBA_HIST_SQLSTAT & DBA_HIST_SEG_STAT


© 2009/2010 Pythian

• In the result …

• Two different jobs were gathering statistics on a daily basis1. “ANALYZE …” part of other batch job (developer)2. “DBMS_STATS…” traditional (DBA)

• Sometimes “DBMS_STATS…“ is not completed before the batch job starts (+/- 10 minutes).

• After the job got killed (typically well after 10 mins since start) the new “correct” statistics were in place.

• Takeaways …A. Don’t change your statistics that frequently (should be consistent)

B. AWR data helps to spot such issues easily

32


© 2009/2010 Pythian

SQL Plan flipping Example (3) …

• I asked myself: Well !

• If you find that the execution plan for a SQL has changed from a good (quick) to a bad one (slow), how do you know if there are other affected SQLs?

• And if there are, how many and which ones?

• Would SQL Profiles (Outlines) help address those?

33

© 2009/2010 Pythian

SELECT st2.SQL_ID , st2.PLAN_HASH_VALUE , st_long.PLAN_HASH_VALUE l_PLAN_HASH_VALUE , st2.CPU_MINS , st_long.CPU_MINS l_CPU_MINS , st2.ELA_MINS , st_long.ELA_MINS l_ELA_MINS , st2.EXECUTIONS , st_long.EXECUTIONS l_EXECUTIONS , st2.CROWS , st_long.CROWS l_CROWS , st2.CPU_MINS_PER_ROW , st_long.CPU_MINS_PER_ROW l_CPU_MINS_PER_ROWFROM (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , SUM(st.EXECUTIONS_DELTA) EXECUTIONS , SUM(st.ROWS_PROCESSED_DELTA) CROWS , TRUNC(SUM(st.CPU_TIME_DELTA) /1000000/60) CPU_MINS , DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW , TRUNC(SUM(st.ELAPSED_TIME_DELTA) /1000000/60) ELA_MINS FROM DBA_HIST_SQLSTAT st WHERE 1 =1 AND ( st.CPU_TIME_DELTA !=0 OR st.ROWS_PROCESSED_DELTA !=0) GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st2, (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , SUM(st.EXECUTIONS_DELTA) EXECUTIONS , SUM(st.ROWS_PROCESSED_DELTA) CROWS , TRUNC(SUM(st.CPU_TIME_DELTA) /1000000/60) CPU_MINS , DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW , TRUNC(SUM(st.ELAPSED_TIME_DELTA) /1000000/60) ELA_MINS FROM DBA_HIST_SQLSTAT st WHERE 1 =1 AND ( st.CPU_TIME_DELTA !=0 OR st.ROWS_PROCESSED_DELTA !=0) HAVING TRUNC(SUM(st.CPU_TIME_DELTA)/1000000/60) > 10 GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st_longWHERE 1 =1AND st2.SQL_ID = st_long.SQL_IDAND st_long.CPU_MINS_PER_ROW/DECODE(st2.CPU_MINS_PER_ROW,0,1,st2.CPU_MINS_PER_ROW) > 2ORDER BY l_CPU_MINS DESC, st2.SQL_ID, st_long.CPU_MINS DESC, st2.PLAN_HASH_VALUE;

34


© 2009/2010 Pythian

SELECT ...FROM (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , ... DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW , ... FROM DBA_HIST_SQLSTAT st WHERE 1 =1 ... GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st2, (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , ...HAVING trunc(sum(st.CPU_TIME_DELTA)/1000000/60) > 10GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st_longWHERE 1 =1AND st2.SQL_ID = st_long.SQL_IDAND st_long.CPU_MINS_PER_ROW/DECODE(st2.CPU_MINS_PER_ROW,0,1,st2.CPU_MINS_PER_ROW) > 2ORDER BY l_CPU_MINS DESC, st2.SQL_ID, st_long.CPU_MINS DESC, st2.PLAN_HASH_VALUE;

35


© 2009/2010 Pythian

SQL_ID PLAN_HASH_VALUE L_PLAN_HASH_VALUE CPU_MINS L_CPU_MINS ELA_MINS L_ELA_MINS EXECUTIONS L_EXECUTIONS CROWS L_CROWS CPU_MINS_PE------------- --------------- ----------------- ---------- ---------- ---------- ---------- ---------- ------------ ---------- ---------- ---db8yz0rfhvufm 3387634876 619162475 17 2673 21 4074 3106638 193 2121380 131453 8.3979E-06 .02035ppdcygtcw7p6 436796090 2817686509 1 2557 3 2765 20 9 82733 32278 .000016381 .0795ppdcygtcw7p6 863350916 2817686509 5 2557 11 2765 71 9 478268 32278 .000011503 .0791tab7mjut8j9h 875484785 911605088 9 2112 23 2284 980 1436 808 606 .011678951 3.4851tab7mjut8j9h 2484900321 911605088 6 2112 6 2284 1912 1436 1516 606 .003998529 3.4851tab7mjut8j9h 3141038411 911605088 50 2112 57 2284 32117 1436 26048 606 .00195365 3.485gpj32cqd0qy9a 1700210966 2667660534 1 1501 7 1642 36 4 69973 1489 .000022548 1.00gpj32cqd0qy9a 3094138997 2667660534 1 1501 3 1642 30 4 58400 1489 .000025147 1.002tf4p2anpwpk2 825403357 1679851684 6 824 71 913 17 13 21558 253 .000314321 3.260csvwu3kqu43j4 3860135778 2851322291 0 784 0 874 1 2 1546 204 .000165255 3.8460q9hpmtk8c1hf 3860135778 2851322291 0 779 0 867 1 2 4075 479 .000068144 1.6272frwhbxvg1j69 3860135778 2851322291 0 776 0 865 1 2 1950 492 .000139004 1.5784nzsxm3d9rspt 3860135778 2851322291 0 754 0 846 1 2 1901 3445 .000074031 .21901pc2npdb1kbp6 9772089 2800812079 0 511 0 3000 7 695 383 14021 .000007121 .0364gpj32cqd0qy9a 1700210966 1168845432 1 482 7 554 36 2 69973 441 .000022548 1.093gpj32cqd0qy9a 3094138997 1168845432 1 482 3 554 30 2 58400 441 .000025147 1.093 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *4bcx6kbbrg6bv 3781789023 2248191382 0 11 0 41 2 2 34 34 .000797865 .32986wh3untj05apd 3457450300 3233890669 0 11 0 131 1 20 21 393 .003310346 .02856wh3untj05apd 3477405755 3233890669 0 11 1 131 2 20 17 393 .007030779 .02858pzsjt5p64xfu 3998876049 3667423051 0 11 5 44 3 18 12 72 .065142643 .1566bpfzx2hxf5x7f 1890295626 774548604 0 11 0 26 1 24 488580 11279752 .000000477 .0000g67nkxd2nqqqd 1308088852 4202046543 0 11 1 57 1 49 32 393 .011233152 .0282g67nkxd2nqqqd 1308088852 1991738870 0 11 1 39 1 38 32 414 .011233152 .0269g67nkxd2nqqqd 2154937993 1991738870 1 11 27 39 72 38 371 414 .003401926 .0269g67nkxd2nqqqd 2154937993 4202046543 1 11 27 57 72 49 371 393 .003401926 .0282

92 rows selected.

Elapsed: 00:00:02.53SQL>

36


© 2009/2010 Pythian

• In the result …

• Load on the system was reduced by 5 times • Takeaways …

A. SQL Plans may flip from good plans to …B. SQL Outlines/Profiles may help some timesC. AWR provides good input for such analysis

• Why SQL Plans may flip?1. Bind variable peeking / adaptive cursor sharing2. Statistics change (including difference in partitions stats)

3. Adding/Removing indexes4. Session/System init.ora parameters (nls_sort/optimizer_mode)

5. Dynamic statistics gathering (sampling)

6. Profiles/Outlines/Baselines evolution

37


© 2009/2010 Pythian

• AWR = DBA_HIST% views ( snapshots from V$% views )

• Sometimes it is the only source of information

• AWR contains much more information that default AWR reports and Grid Control could provide you

• Be careful mining data (there are some gotchas)

• Don’t be afraid to discover/mine the AWR data

I can show you the door … … but it is you who should walk through

it

38

Conclusions …

© 2009/2010 Pythian 39

Pythian Facts• Founded in 1997, over 14 years• 100+ DBAs ! (140 employees)• 5 offices in 5 countries (true follow the sun model)• Employ

• 6 Oracle ACEs (Including 1 ACE director)• Several Oracle Masters• Plenty of technical geeks

• Platinum level partner in the Oracle Partner Network• Actively supports technical communities via

• Blogging• Conferences• SIGs and other events

© 2009/2010 Pythian

Additional Resources

• www.oracle.com/scan• www.pythian.com/exadata• www.pythian.com/news/tag/exadata -

Exadata Blog• www.pythian.com/news_and_events/

in_the_newsArticle: “Making the Most of Oracle Exadata”

My Oracle Support notes 888828.1 and 757552.1

Thank you!

40

Google Yury Oracle [LinkedIn, twitter, blog, email, mobile, …] Email me to get the presentation

Mission

Let you remember/consider AWR next time you troubleshoot

Performance issue!

Pythian• Like Pythian on

facebook: http://on.fb.me/pythianfacebook• Follow us on LinkedIn: http://linkd.in/pythian• Follow Pythian on Twitter @pythian

(www.twitter.com/pythian)

Technology

Oracle AWR Data mining