Surviving the Crisis With the Help of Oracle Database Resource Manager

© 2

010

Tie

to C

orpo

ratio

n

Surviving the Crisis with the Help of Oracle Database

Resource ManagerUKOUG Conference Series

Technology & E-Business Suite 2010

Māris Elsiņš

Senior Oracle Applications DBATieto Latvia,[email protected]

© 2010 Tieto Corporation2

Who I am?• 8 years in IT

• 3 years – PL/SQL developer• 5 years – Oracle [Apps] DBA (started with 11.5.7 and 8.1.7)

• Certificates• 10g OCM• 9i / 10g / 11g OCP• 11i Applications Database Administrator OCP• 11i System Administrator OCE

• Conferences• UKOUG 2007/2008/2010• LVOUG 2009/2010• EMEA Harmony 2010

• Current employer – Tieto Latvia• All kinds of oracle DBA tasks - patching, upgrade, performance tuning,

troubleshooting, planning and implementation of backup and recovery procedures, cross platform migration, etc.

• Planning of system architecture, design and implementation of HA solutions (RAC, Data Guard, Custom cold failover)

• Implementation of system specific monitoring, automation of routine tasks• Technical project planning and coordination, management of team of DBAs

2010-11-29


I come from Latvia!

Latvia is a small country…

…with a huge economic crisis.

That’s where the topic comes from!

Surviving the Crisis with the Help of Oracle Database Resource Manager

2010-11-29


Contents• Presentation will contain:

• Short Introduction to the Resource Manager• DEMO• Evaluation of Performance impact

• The Goal is:• Not to read the documentation for you…• …but to show why and when the resource manager can be very

useful

2010-11-29


The problem• OS doesn’t care enough about DB sessions/processes

according to what business requires• Assigns the same priority for all processes• CPU resources are equally distributed among all processes• Inability to manage DB-specific resources

• Parallel processes• Active sessions• Sessions holding latches• Undo usage

• Context switching overhead when many processes running• Problems start when there’s not enough CPU gor everyone• CPU starvation can be hard to recover from (snowball

effect)• CPU starvation makes online troubleshooting hard to do

2010-11-29


Some possible problem scenarios• Some samples of problems

• running reports cause too much load on the OLTP system• One of the sessions allocate all parallel query slaves therefore other

sessions can’t use any of them• Application support team runs heavy queries to anayze the data leaving less

resources for business processing• Too wide search criteria cause “hangs” in the search form• 3 of 8 CPU cores are idle, my query runs without parallel execution, I could

use the idle CPUs to provide results faster.• Users don’t log out and leave idle sessions• Sessions with incomplete transactions have locked some rows and other

sessions have stuck therefore.

• Does Oracle Resource Manager solve all these problems?

2010-11-29

• Some samples of problems• running reports cause too much load on the OLTP system• One of the sessions allocate all parallel query slaves therefore other

sessions can’t use any of them• Application support team runs heavy queries to anayze the data leaving less

resources for business processing• Too wide search criteria cause “hangs” in the search form• 3 of 8 CPU cores are idle, my query runs without parallel execution, I could

use the idle CPUs to provide results faster.• Users don’t log out and leave idle sessions• Sessions with incomplete transactions have locked some rows and other

sessions have stuck therefore.

• Does Oracle Resource Manager solve all these problems?


The concept of the Resource manager• Resource manager

• Included in Oracle EE licnese• Allows prioritization of sessions according to the defined business

requirements• Allows defining the guaranteed amount of allocated resources for

each type of sessions (consumer group)• Resources not used by higher priority sessions, can be used by

lower priority sessions• Prioritization archieved by changing the process states to

running/sleeping• Resource manager does not solve the «lack of CPU

resources» problem, it just controls there execution queue• Resource manager uses some resources too, the 3rd part of

the presentation will estimate the overhead

2010-11-29


Which side is the RM enabled for?

2010-11-29

48 active sessions all doing CPU work in both casesCheck Running/Sleeping process states!


The concept of the Resource manager• Consumer group

• Set of sessions having similar requirements for server resources

• Resources are allocated to the consumer group, not individual sessions

• dba_rsrc_consumer_groups

• Directives• Rules that define resource allocation

to the consumer group• dba_rsrc_plan_directives

• Resource plan• Set of directives defining the

distribution of resources among consumer groups

• dba_rsrc_plans

2010-11-29


Features of Resource Manager

2010-11-29

9.2 10.2 11.1 11.2

CPU resource allocation to consumer groups by ratio or percentage

Limit of the degree of parallelism

active session pool

Automated change of consumer group if session has used or is estimated to use the defined amount of resources (CPU, Est CPU, IO_MB, IO_REQ)

CPU, Est CPU

CPU, Est CPU

CPU,Est CPU, IO_MB, IO_REQ

CPU,Est CPU, IO_MB, IO_REQ

Limit of estimated execution time

Limit size of undo used by uncommitted sessions

Termination of idle sessions

Termination of idle blocking sessions

L0 70% CPU _ORACLE_BACKGROUND_GROUP_ hidden consumer group for background processes

Instance caging /CPU_COUNT + resource plan/

Max CPU Utilization limit


Setting up the resource manager• Create the resource consumer groups, resource plan and

directives• Use DB Console or EM Grid Control• Use API (DBMS_RESOURCE_MANAGER package)

• Define rules to assign sessions to particular consumer groups• Connect time (ORACLE_USER, SERVICE_NAME, CLIENT_OS_USER,

CLIENT_PROGRAM, CLIENT_MACHINE)• Run time (MODULE_NAME, MODULE_NAME_ACTION, SERVICE_MODULE,

SERVICE_MODULE_ACTION)

• Enable resource plan• ALTER SYSTEM SET resource_manager_plan=‘ALL_OK';

Excellent documentation! Oracle® Database Administrator's Guide 11g Release 2 (11.2)

27 Managing Resource Allocation with Oracle Database Resource Manager

http://download.oracle.com/docs/cd/E11882_01/server.112/e17120/dbrm.htm#ADMIN027

2010-11-29





Important to keep in mind• 11g has resource manager enabled by default for

maintenance tasks (can cause problems after upgrade to 11g)• MAINTENANCE_WINDOW_GROUP scheduler window group• DEFAULT_MAINTENANCE_PLAN plan

• Waitevent «rsmgr: cpu quantum» for sleeping processes• Common mistake is to forget about v$session.state

• WAITING – session is currently waiting• WAITED KNOWN TIME – session is running on CPU• WAITED SHORT TIME – session is running on CPU

• (same warning is valid for other wait events too)

2010-11-29


DEMO

2010-11-29


DEMO on SAMPLE_PLAN3• SYS_GROUP

• 5% L1 CPU• ONLINE3

• 95% L1 CPU• Switch the session to LONG3

group if the execution takes longer then 30 seconds

• OTHER_GROUPS• 100% L2 CPU• KILL blocking session if it’s

idle for longer then 90 seconds• LONG3

• 20% L1 CPU• KILL session inactive for

longer then then 20 seconds

2010-11-29


Evaluation of performance impact

2010-11-29


Evaluation of performance impact• Test platforms

• 11.1.0.7 on RHEL 4.4, 4Cores + HT, RAID5 Local HDD, 6G RAM• 11.2.0.1 on HP-UX v11.31, vPAR on ia64 hp superdome server

SD32B, 4Cores, NetApp storage (could have affected the test results), 32G RAM

• Concept• Identically repeatable workload• Testing is the only activity (no schedules, jobs, awr snapshots, etc.)• Each test repeated 3 times, average execution time recorded• DBs configured with ASYNC and DIRECT IO (FS cache does not

affect testing)• DB Buffer cache flushed before every IO test

• Goal• Identify if enabling the resource manager causes visible impact on

performance

2010-11-29


Tests• «CPU bottleneck» on RHEL with 8, 24, 48 processes

• No RM plan in use• Simple RM plan in use, processes evenly distributed among CGs• Complex RM plan in use, processes evenly distributed among CGs

• «CPU bottleneck» on HP-UX with 4, 12, 24, 48 processes• No RM plan in use• Simple RM plan in use, processes evenly distributed among CGs• Complex RM plan in use, processes evenly distributed among CGs

• «IO bottleneck» on RHEL with 24 processes• No RM plan in use• Complex RM plan in use

• «IO bottleneck» on HP-UX with 24 processes• No RM plan in use• Complex RM plan in use

2010-11-29


Testing with simple RM plan

2010-11-29


Testing with complex RM plan

2010-11-29


«CPU bottleneck»---noramloadcpu.sql---

set verify off feed off timing on serverout on time on

declare

n number;

x number:=10000000*&1;

z number;

date1 date;

begin

select sysdate into date1 from dual;

DBMS_APPLICATION_INFO.SET_MODULE(module_name => 'TEST',

action_name => '&2');

for n in x+1..x+10000000 loop

z:=sqrt(n);

end loop;

dbms_output.put_line(to_char(date1,'HH24:MI:SS')||' : &&2 : &&1');

end;

/

sqlplus -s apps/nopassword@TTBI @noramloadcpu.sql 1 Online_group &

sqlplus -s apps/nopassword@TTBI @noramloadcpu.sql 2 Batch_group &

sqlplus -s apps/nopassword@TTBI @noramloadcpu.sql 3 Bug_Maint_group &

sqlplus -s apps/nopassword@TTBI @noramloadcpu.sql 4 Users_group &

sqlplus -s apps/nopassword@TTBI @noramloadcpu.sql 5 Postman_group &

sqlplus -s apps/nopassword@TTBI @noramloadcpu.sql 6 Mail_Maint_group &



















echo "---------------------------"

wait

echo "---------------------------"

2010-11-29

• Practically no IO• Each process spinning on

CPU


«CPU bottleneck» resultsRHEL 4.4 and 11.1.0.7 HP-UX v11.31 and 11.2.0.1

2010-11-29


«CPU bottleneck» results--------------WITHOUT RESOURCE MANAGER--------------

09:28:05 : LOW : 14 Elapsed: 00:00:15.3909:28:05 : LOW : 19 Elapsed: 00:00:15.4909:28:06 : HIGH : 43 Elapsed: 00:00:16.9309:28:07 : HIGH : 46 Elapsed: 00:00:17.4509:28:07 : HIGH : 41 Elapsed: 00:00:20.4209:28:05 : LOW : 13 Elapsed: 00:00:24.8209:28:15 : HIGH : 39 Elapsed: 00:00:17.4009:28:09 : HIGH : 35 Elapsed: 00:00:24.0609:28:22 : HIGH : 38 Elapsed: 00:00:11.1909:28:19 : HIGH : 34 Elapsed: 00:00:17.7209:28:30 : LOW : 4 Elapsed: 00:00:11.3009:28:33 : HIGH : 47 Elapsed: 00:00:10.8009:28:34 : HIGH : 25 Elapsed: 00:00:10.8809:28:37 : LOW : 20 Elapsed: 00:00:11.0309:28:41 : HIGH : 29 Elapsed: 00:00:11.4909:28:44 : LOW : 12 Elapsed: 00:00:11.6609:28:45 : LOW : 15 Elapsed: 00:00:13.8709:28:48 : LOW : 11 Elapsed: 00:00:15.7409:28:50 : LOW : 6 Elapsed: 00:00:17.6809:28:55 : LOW : 17 Elapsed: 00:00:16.7009:28:54 : HIGH : 33 Elapsed: 00:00:19.7609:28:50 : HIGH : 30 Elapsed: 00:00:23.2009:28:59 : LOW : 5 Elapsed: 00:00:15.6009:29:03 : HIGH : 37 Elapsed: 00:00:20.4809:29:08 : LOW : 23 Elapsed: 00:00:15.4309:29:02 : HIGH : 28 Elapsed: 00:00:21.5909:29:06 : LOW : 22 Elapsed: 00:00:21.6309:29:13 : HIGH : 45 Elapsed: 00:00:16.4709:29:17 : LOW : 1 Elapsed: 00:00:15.2809:29:15 : LOW : 18 Elapsed: 00:00:18.1009:29:20 : HIGH : 40 Elapsed: 00:00:16.7209:29:24 : LOW : 24 Elapsed: 00:00:12.6909:29:27 : HIGH : 44 Elapsed: 00:00:14.9209:29:33 : HIGH : 31 Elapsed: 00:00:11.9509:29:37 : LOW : 8 Elapsed: 00:00:11.0409:29:36 : LOW : 21 Elapsed: 00:00:14.4609:29:37 : LOW : 16 Elapsed: 00:00:13.7909:29:43 : LOW : 7 Elapsed: 00:00:11.8809:29:45 : HIGH : 36 Elapsed: 00:00:10.8809:29:50 : HIGH : 32 Elapsed: 00:00:13.2209:29:51 : LOW : 2 Elapsed: 00:00:13.3209:29:53 : HIGH : 42 Elapsed: 00:00:11.3709:29:56 : LOW : 9 Elapsed: 00:00:10.1209:29:57 : HIGH : 26 Elapsed: 00:00:17.0409:30:03 : LOW : 3 Elapsed: 00:00:11.8409:30:05 : LOW : 10 Elapsed: 00:00:10.9209:29:59 : HIGH : 27 Elapsed: 00:00:17.6309:30:08 : HIGH : 48 Elapsed: 00:00:13.25

---------------------------real 2m17.292s

--------------WITH RESOURCE MANAGER--------------09:24:50 : HIGH : 25 Elapsed: 00:00:12.0709:24:54 : HIGH : 42 Elapsed: 00:00:14.7709:24:54 : HIGH : 29 Elapsed: 00:00:17.0309:24:53 : HIGH : 39 Elapsed: 00:00:20.5709:24:55 : HIGH : 48 Elapsed: 00:00:19.7009:25:04 : HIGH : 36 Elapsed: 00:00:23.1009:25:04 : HIGH : 30 Elapsed: 00:00:23.7909:25:03 : HIGH : 28 Elapsed: 00:00:24.3809:25:11 : HIGH : 38 Elapsed: 00:00:30.3409:25:11 : HIGH : 44 Elapsed: 00:00:31.5809:25:12 : HIGH : 35 Elapsed: 00:00:31.2809:25:19 : HIGH : 46 Elapsed: 00:00:38.9309:25:20 : HIGH : 40 Elapsed: 00:00:38.4809:25:21 : HIGH : 34 Elapsed: 00:00:38.8709:25:23 : HIGH : 45 Elapsed: 00:00:37.6809:25:23 : HIGH : 32 Elapsed: 00:00:37.3709:25:22 : HIGH : 31 Elapsed: 00:00:39.7409:25:26 : HIGH : 47 Elapsed: 00:00:35.9409:25:23 : HIGH : 33 Elapsed: 00:00:39.2909:25:25 : HIGH : 37 Elapsed: 00:00:38.0009:25:29 : HIGH : 26 Elapsed: 00:00:34.9409:25:28 : HIGH : 27 Elapsed: 00:00:35.7209:25:31 : HIGH : 41 Elapsed: 00:00:34.1909:25:49 : HIGH : 43 Elapsed: 00:00:21.1709:24:50 : LOW : 24 Elapsed: 00:01:55.3809:24:51 : LOW : 21 Elapsed: 00:02:01.8409:24:50 : LOW : 22 Elapsed: 00:02:03.0709:24:53 : LOW : 8 Elapsed: 00:02:04.9409:24:53 : LOW : 5 Elapsed: 00:02:04.8309:25:05 : LOW : 3 Elapsed: 00:01:53.8809:25:17 : LOW : 2 Elapsed: 00:01:44.0309:25:28 : LOW : 7 Elapsed: 00:01:34.1109:25:17 : LOW : 9 Elapsed: 00:01:45.8609:25:19 : LOW : 6 Elapsed: 00:01:43.9009:25:19 : LOW : 4 Elapsed: 00:01:44.0209:25:01 : LOW : 10 Elapsed: 00:02:02.0509:25:03 : LOW : 1 Elapsed: 00:02:00.2209:25:22 : LOW : 20 Elapsed: 00:01:41.6009:24:55 : LOW : 16 Elapsed: 00:02:08.8009:24:55 : LOW : 11 Elapsed: 00:02:08.0109:25:29 : LOW : 18 Elapsed: 00:01:34.7409:25:10 : LOW : 15 Elapsed: 00:01:54.6909:25:12 : LOW : 17 Elapsed: 00:01:51.9909:25:09 : LOW : 23 Elapsed: 00:01:55.0309:25:21 : LOW : 12 Elapsed: 00:01:42.8709:25:11 : LOW : 19 Elapsed: 00:01:53.6809:25:25 : LOW : 13 Elapsed: 00:01:39.4509:25:15 : LOW : 14 Elapsed: 00:01:50.21

---------------------------real 2m15.818s

2010-11-29

First

Last

First

Last


«IO bottleneck»---prepare data---

create table MEL_IO_TEST as select cast(rownum as number)x, lpad('a',400,'a') y from dual connect by level <=2654208;

insert into MEL_IO_TEST select * from MEL_IO_TEST;

commit;

create index MEL_IO_TEST_IDX on MEL_IO_TEST(x);

--- ioloadtest.sql---

set verify off feed off timing on serverout on time on

declare

nn number;

date1 date;

begin

select sysdate into date1 from dual;

DBMS_APPLICATION_INFO.SET_MODULE(module_name => 'TEST',

action_name => '&3');

select /*+ index(a MEL_IO_TEST_IDX)*/avg(length(y)+length(x)) into nn from MEL_IO_TEST a where x between 2654208/&2*(&1-1) and 2654208/&&2*&&1;

dbms_output.put_line(to_char(date1,'HH24:MI:SS')||' : &&3 : &&1 : &&2');

end;

/

sqlplus -s apps/apps@MEL @ioloadtest_flush.sql

sqlplus -s apps/apps@MEL @ioloadtest.sql 1 24 Online_group &

sqlplus -s apps/apps@MEL @ioloadtest.sql 2 24 Batch_group &

sqlplus -s apps/apps@MEL @ioloadtest.sql 3 24 Bug_Maint_group &

sqlplus -s apps/apps@MEL @ioloadtest.sql 4 24 Users_group &

sqlplus -s apps/apps@MEL @ioloadtest.sql 5 24 Postman_group &

sqlplus -s apps/apps@MEL @ioloadtest.sql 6 24 Mail_Maint_group &



















echo "------------"

wait

echo "------------"

2010-11-29


«IO bottleneck» results

2010-11-29

RHEL 4.4 and 11.1.0.7• Average execution time:

• No RM: 227.09s• Complex RM: 226.68s

HP-UX v11.31 and 11.2.0.1• Average execution time:

• No RM: 36,30s• Complex RM: 37,42s


«IO bottleneck» results------------WITHOUT RESOURCE MANAGER------------13:23:47 : HIGH : 23 : 64 Elapsed: 00:02:34.5113:23:47 : LOW : 1 : 64 Elapsed: 00:02:55.0713:23:47 : LOW : 2 : 64 Elapsed: 00:02:57.0513:23:47 : HIGH : 24 : 64 Elapsed: 00:03:27.2013:23:47 : LOW : 12 : 64 Elapsed: 00:03:43.3813:23:47 : LOW : 8 : 64 Elapsed: 00:03:44.7913:23:47 : LOW : 10 : 64 Elapsed: 00:03:44.8113:23:47 : LOW : 11 : 64 Elapsed: 00:03:45.0213:23:47 : LOW : 9 : 64 Elapsed: 00:03:45.0713:23:47 : HIGH : 16 : 64 Elapsed: 00:03:45.2313:23:47 : HIGH : 14 : 64 Elapsed: 00:03:45.2613:23:47 : HIGH : 15 : 64 Elapsed: 00:03:45.3413:23:47 : LOW : 7 : 64 Elapsed: 00:03:45.4213:23:47 : HIGH : 13 : 64 Elapsed: 00:03:45.6113:23:47 : HIGH : 20 : 64 Elapsed: 00:03:46.3513:23:47 : HIGH : 19 : 64 Elapsed: 00:03:46.4413:23:47 : LOW : 5 : 64 Elapsed: 00:03:46.4513:23:47 : LOW : 4 : 64 Elapsed: 00:03:46.8013:23:47 : HIGH : 22 : 64 Elapsed: 00:03:47.0513:23:47 : HIGH : 18 : 64 Elapsed: 00:03:47.2113:23:47 : LOW : 6 : 64 Elapsed: 00:03:47.3513:23:47 : HIGH : 17 : 64 Elapsed: 00:03:47.3513:23:47 : LOW : 3 : 64 Elapsed: 00:03:47.3813:23:47 : HIGH : 21 : 64 Elapsed: 00:03:47.54

------------real 3m48.504s

------------WITH RESOURCE MANAGER------------13:32:14 : LOW : 7 : 64 Elapsed: 00:02:52.4713:32:14 : LOW : 2 : 64 Elapsed: 00:02:52.8213:32:15 : HIGH : 24 : 64 Elapsed: 00:03:23.3613:32:15 : HIGH : 23 : 64 Elapsed: 00:03:25.9713:32:14 : HIGH : 13 : 64 Elapsed: 00:03:43.7413:32:15 : HIGH : 16 : 64 Elapsed: 00:03:43.8213:32:14 : LOW : 9 : 64 Elapsed: 00:03:44.0313:32:15 : HIGH : 22 : 64 Elapsed: 00:03:44.0913:32:14 : HIGH : 19 : 64 Elapsed: 00:03:44.6013:32:14 : LOW : 10 : 64 Elapsed: 00:03:44.7213:32:14 : LOW : 11 : 64 Elapsed: 00:03:45.0513:32:15 : HIGH : 21 : 64 Elapsed: 00:03:45.1213:32:15 : LOW : 3 : 64 Elapsed: 00:03:45.3213:32:15 : HIGH : 20 : 64 Elapsed: 00:03:45.3713:32:14 : LOW : 8 : 64 Elapsed: 00:03:45.8313:32:14 : LOW : 6 : 64 Elapsed: 00:03:45.8913:32:15 : HIGH : 17 : 64 Elapsed: 00:03:45.6813:32:14 : HIGH : 14 : 64 Elapsed: 00:03:45.8513:32:14 : LOW : 12 : 64 Elapsed: 00:03:46.0913:32:14 : LOW : 1 : 64 Elapsed: 00:03:46.2813:32:15 : HIGH : 15 : 64 Elapsed: 00:03:46.1213:32:15 : HIGH : 18 : 64 Elapsed: 00:03:46.3013:32:14 : LOW : 4 : 64 Elapsed: 00:03:46.4413:32:14 : LOW : 5 : 64 Elapsed: 00:03:46.46

------------real 3m47.104s

2010-11-29

• RM does not help if the system is IO-bound


Summary• Very useful if CPU utilization close to 100%

• Ensures most critical sessions get the resources• Reduces overall load on the server by setting least important

processes to «sleep»• Resource Manager does not solve the problem – just allows

prioritization• Performance overhead is practically invisible, (could depend

on platform) – test yourself!• Included in EE license• No downtime needed to implement it, setup can be changed

dynamically • Can be used as a short-term workaround of some

performance problems.• Some bugs can show up as additional code layer introduced

– test it!

2010-11-29

© 2010 Tieto Corporation27 2010-11-29

Thank you!

?http://appsdbalife.wordpress.com – Comment my blog and I’ll answer!

http://appsdbalife.wordpress.com/

http://appsdbalife.wordpress.com/

© 2

010

Tie

to C

orpo

ratio

n Māris Elsiņš

Senior Oracle Applications DBATieto Latvia,[email protected]