Multiple awr reports_parser

2015© Trivadis

BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA

2015 © Trivadis

Multiple AWR Reports Parser

Jacques Kostic

Senior Consultant IMS

[email protected]

7-mar-2015

TechEvent: Multiple AWR Reports Parser

1

2015© Trivadis

AGENDA

1. Customer overview

2. Architecture overview

3. Why such tool?

4. Reports parsing phase

5. Reports analysis phase

6. Conclusions

7. Q&A

7-mar-2015


2

2015© Trivadis

7-mar-2015


Customer Overview

3

2015© Trivadis

Customer Overview

The name will not be disclosed but the most relevant

characteristics to the project are reported below.

Server and database names are fictive.

Medium size customer of the Public sector.

Data Warehouse environment.

More than 30 databases involved with various sizes up to 900 GB.

Huge amount of processing involving many business entities.

Highly consolidated Oracle infrastructure.

Strong growing factor.

Constant criticality increase.

7-mar-2015


4

Customer

Environment

2015© Trivadis

Customer Overview

Goal of the project:

Identify factors that cause important performance deviations during

Data Warehouse load processing

Propose solutions to resolve the issues if identified

Propose cost effective solutions to efficiently absorb load

processing and satisfy new business needs

Deliverables:

Evidences and explanations of deviations during load processing

Possible improvement areas

7-mar-2015


5

2015© Trivadis

7-mar-2015


Architecture

overview

6

2015© Trivadis

Architecture overview

SAN: VSP Hitachi with Dynamic and Thin Provisioning

Production: OS, Solaris 1 CPU X5460 4 cores, RAM, 64GB

2 * FC 4Gb, 4 * NIC Gb

ZFS file systems in the ZPOOL

9 databases : 11.2.0.3

QAS/DEV: OS, Solaris 1 CPU X5460 4 cores, RAM, 64GB

2 * FC 4Gb, 4 * NIC Gb

ZFS file systems in the ZPOOL

13 databases : 11.2.0.3

VMs: a couple of VMs under VMWare that host some Talend and

Cognos repositories

7-mar-2015


7

2015© Trivadis

7-mar-2015


Why such tool?

8

2015© Trivadis

Why such tool?

The idea came to me during the kickoff meeting when I realize that I

would not have the expected, level of support, access and time to do the

work.

Key constraints :

Limited internal DBA support

Limited internal BI team support

No access to the system

No OEM access

Requires complete analysis of two consecutive weeks (last and first of next

month)

Very short time to run the project (7 days)

The outcome must focus on key factors with relevant and proven facts.

7-mar-2015


9

2015© Trivadis

Why such tool?

It has been agreed that:

System team would send me all graphs for servers and storage.

DBA would send all AWR reports for all databases for the two weeks.

A script was provided to DBAs for reports generation

With 30 minutes AWR window basis, the total number of reports is:

24*2*7*2= 672 reports per database

672 * 22 databases = 14’784 reports!

Solution?

Parse all reports and extract what is relevant to have the full trend and

being able to highlight bottlenecks

7-mar-2015


10

2015© Trivadis

7-mar-2015


Reports parsing phase

11

2015© Trivadis


During the development phase, my college @ludodba told me that

@flashdba has already build something similar. As I almost completed the

script and some other ideas where coming, I decided to continue spending

some time on the utility. Finally our versions are differing somehow…

Here are some key points and features (at the moment):

Works fine starting from 11g and 12c ready

Most relevant data are captured

Sorting of the output file on the snapshot time

Adjust window time to a given period

Generate cross references for multiple databases impact analysis

Generate events summary of TOP 6 foreground events of all reports

Fast (336 reports parsed in less than 15 seconds)

7-mar-2015


12

The tool

http://twitter.com/ludodba

http://twitter.com/flashdba

2015© Trivadis


7-mar-2015


13

Extract information from reports

$> tvd_perl parse_awr.pl -o <output folder> -s <source folder

containing the reports> -t <TimeWindow (30,60,... minutes)>)

$> tvd_perl parse_awr.pl -o Outputs -s WeekOne/prod/dbPRD01 -t 30

Parsing file WeekOne\prod\dbPRD01/awrrpt_1_15454_15455_20141127_08_30_20141127_09_00.txt



…




Processed 336 files

Saving result...

Writing header...

Sorting by snapshot date/time...

Writing data...

Done!

2015© Trivadis


7-mar-2015


14

Generate cross references

$> tvd_perl consolidate_awr.pl [-p <filename>] -s <Sourcefolder> -c

<col_number> -o <OutputFile>

$> tvd_perl consolidate_awr.pl -p srvlPRD_dbPRD01.csv

Column:0 -> DB -> String

Column:1 -> INST_NAME -> String

Column:2 -> INST_NUM -> Numeric

Column:3 -> Host -> String

Column:4 -> FileName -> String

Column:5 -> CPUs -> Numeric

Column:6 -> DtSort -> Numeric

Column:7 -> BSnap -> String

Column:8 -> ESnap -> String

Column:9 -> ElapsTime -> Numeric

Column:10 -> DBTime -> Numeric

…

Column:101 -> FGEvent6 -> String

Column:102 -> FGWaits6 -> Numeric

Column:103 -> FGTOut6 -> Numeric

Column:104 -> FGTime6 -> Numeric

Column:105 -> FGEAvg6 -> Numeric

Column:106 -> FGPctDb6 -> Numeric

2015© Trivadis


7-mar-2015


15

Generate cross references

$>tvd_perl consolidate_awr.pl -s . -c 10 -o output/dbtime.csv

Collecting columns infos from file srvlPRD_dbPRD01.csv

Processing file srvlPRD_dbPRD01.csv







Writing result in output/dbtime.csv

Done!

dbtime.csv

…Begin Time dbPRD01-DBTime dbPRD02-DBTime dbPRD03-DBTime dbPRD04-DBTime

24/11/2014 00:00 0,29 0,06 0,14 0,34

24/11/2014 00:30 0,22 0,18 0,95 0,46

24/11/2014 01:00 0,11 0,06 0,16 0,28

24/11/2014 01:30 0,24 0,18 0,94 0,33

24/11/2014 02:00 0,1 0,06 0,16 0,3

24/11/2014 02:30 0,22 0,21 1,48 0,28

…

2015© Trivadis


7-mar-2015


16

Generate events summary

$> tvd_perl event_rpt.pl -s <Sourcefile> -o <OutputFile>

$> tvd_perl event_rpt.pl -s srvlPRD_dbPRD01.csv –o Events_dbPRD01.csv


Writing result in Events_dbPRD01.csv

Summary of top 6 foreground events

Done!

Event Name Count Time (Min) Min (ms) Max (ms) Avg (ms)Times in

top 6% Occ % Time

db file sequential read 43’629’296 132 0 121 0.000182 224 11.1 25.7

SQL*Net message from client 5’227 126 1’457 1’457 1.457050 1 0.04 24.5

read by other session 3’677’004 93 1 19 0.001533 11 0.54 18.2

…

direct path write temp 18’595 2 4 59 0.009572 13 0.64 0.5

control file parallel write 22’169 2 3 36 0.007308 169 8.38 0.5

library cache: mutex X 8’717 2 5 139 0.019732 51 2.52 0.5

PX Deq: Slave Session Stat 23’319 2 5 8 0.005489 8 0.39 0.4

Events_dbPRD01.csv

2015© Trivadis


All reports consolidation

One file per database and per week

Cross references

Db_time

Db_cpu

Redo size/s

Logical reads/s

Physical reads/s

Physical writes/s

Block changes/s

Host statistics

…

Event summary for each database on top 6 foreground

7-mar-2015


17

Extracted data for analysis

2015© Trivadis

7-mar-2015


Reports analysis phase

18

2015© Trivadis


Using db_time

For trend analysis we keep only DBPRD2, DBPRD3, DBPRD4 and DBPRD5

7-mar-2015


19

Identify significant activity

2015© Trivadis


In green, CPU graphs given by system team, in blue run queue extracted from

AWR reports analysis. We can clearly observe a perfect correlation.

7-mar-2015


20

CPU ConsumptionFrom 24 au 30 Nov

From 1 au 7 Dec

2015© Trivadis


In green, CPU graphs given by system team, in blue run queue extracted from

AWR reports analysis. We can clearly observe a perfect correlation.

7-mar-2015


21

CPU ConsumptionFrom 24 au 30 Nov

From 1 au 7 Dec

2015© Trivadis

Headline

Text Text

Text

- Text

Text

Text

Text Text

Text

- Text

Text

Text

Text Text

Text

- Text

Text

Text

7-mar-2015


22

2015© Trivadis

Headline

Text Text

Text Text Text Text

Text Text

Text Text Text Text

SQL > EXEC DBMS_STATS.GATHER_SCHEMA_STATS

> ownname => user

> estimate_percent. => DBMS_STATS.AUTO_SAMPLE_SIZE)

7-mar-2015


23

2015© Trivadis

Headline

Text Text Text Text Text









Text Text

7-mar-2015


24

2015© Trivadis

Questions and answers ...

2015 © Trivadis

BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA

Jacques Kostic

Senior Consultant IMS

[email protected]

7-mar-2015


25

Technology

Multiple awr reports_parser