Proof of Concept with Real Application Testing 12c

Luís Marques - @drune - http://lcmarques.com

PROOF OF CONCEPTwith Real Application Testing 12c

Me

Name: Luís Marques

http://lcmarques.com / @drune / [email protected]


http://lcmarques.com/



mailto:[email protected]

mailto:[email protected]


We will discuss

Database Replay features on 12c version and how we can apply stress test and workload scale-up techniques to ensure a real world proof of concept

SQL Perfomance Analyzer

Actual PoC results

We will not to discuss


PoC – What really is?

An huge amount of work to plan and execute (and re-execute until the end of available time period)

Most work is for DBA if Real Application Testing is used.


The (real) way it was…

Hardware arrivesSetup required

Recent Backup required

Restore your DB into new hardware

Flashback Guaranteed Restore PointSetup Application:

- New hardware - Install and Configure:- AS, Clients, Network, etc

We are ready!

2 weeks of infinite fun and infinite problems


The (real) way it was…

Database scheduler Jobs created on-the-fly

Small reference in docs(not even in restrictions area)


The way it should be

Working with the vendor to provide support for Real Application Testing

Client requested a simple and effective way to test new platform and software releases


The way it should be

Hardware arrivesSetup required

Recent Backup required

Restore your DB into new hardware using START_SCN from Capture

Flashback Guaranteed Restore Point

DB ReplayDB Capture


Database Replay - Capture


Before start any capture…RAT Patches on Production and PoC database (Doc ID 560977.1)

Capture Restrictions: direct path load external files with SQL*Loader, scheduler jobs, flashback querys, distributed trx, etc..

Workload is important: sysdate use,avoid maintenance windows or EM activity

No ongoing transactions during capture - Plan you database shutdown and start capture immediately after startup

RAC – Parallel SQL tunning set capture not supported (capture_sts)


Workload Capture

Payment processing starts at 19:00:

18:35 – SQL> shutdown immediate18:45 – SQL> startup restrict18:50 – $ sqlplus / as sysdba18:50 – SQL> @capture_payment_day.sql

DBMS_WORKLOAD_CAPTURE.ADD_FILTER ('PaymentUser', 'USER’, 'PAY_A');

DBMS_WORKLOAD_CAPTURE.START_CAPTURE(name =>'capture_payment_day', dir => 'DB_REPLAY_DIR', auto_unrestrict => TRUE, default_action => 'EXCLUDE', sts_cap_interval => 300);


Workload Capture

Populate DBA_WORKLOAD_CAPTURES / FILTERS

SQL> exec dbms_workload_capture.get_capture_info('DB_REPLAY_DIR');

Save the START_SCN to use with RMAN UNTIL SCN

to restore your database on PoC database.


Workload Capture Export AWR data to compare performance between

Production and PoC database:

SQL> exec dbms_workload_capture.export_awr(2);

Dump of AWR will be used after replay to compare AWR on

PoC database

AWR data


Workload Capture – The intelligent stuff

Workload Intelligence – Java programs to analyze Captured data

Don’t run it on production system (Heavy memory consumption)

Allow you to know more about the capture by patterns

Create user wi

Create Workload

Job

Create Workload

Model

Identify Patterns

Generate Report

Java –classpath … oracle.dbreplay.workload.intelligence.LoadInfo -job wi_job1 -cdir /home/oracle/scripts/rat/db_replay_dir -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi

Java –classpath … oracle.dbreplay.workload.intelligence.BuildModel -job wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi

Java –classpath … oracle.dbreplay.workload.intelligence.FindPatterns -job wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi

Java –classpath … oracle.dbreplay.workload.intelligence.GenerateReport -job wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi –top 5 –out intel_report.html


How an intelligence report looks… Pattern is a sequence of transactions that run under “certain order”.

Allow you to identify mostly used “code paths” – Example: create a customer or process a visa card “code paths”

Patterns are ordered by different columns (DB Time, Nr. Executions and Length)

Pattern id = 2 Create a customer run 2333 times during the capture with a DB Time 9 seconds (0,78% of total DB Time)

Pattern id = 2

Pattern id =1


Database Replay – Replay


Before start any replay…Setup guarantee restore points before any replay

Run java utility “Workload Analyzer” – Check the quality

of capture (insufficient data, errors, etc)

Run a small test replay (~ 20 min) before full replay - Avoid potential issues with full replay

Set up your system date to the same time as time of capture start if application relies on sysdate

Before performance analysis perform a replay analysis and review divergences


Before start replay - sysdate

Set system clock on PoC system to the start of time of capture

Make sure that NTP daemon is disabled

Make sure that AWR snapshots are still created automatically after system clock changed

Restart PoC database after change in the clock.


Before start replay – The Workload Analyzer

Java utility provided by Oracle to analyze the quality of

your capture.

$ java -classpath $ORACLE_HOME/jdbc/lib/ojdbc6.jar:$ORACLE_HOME/rdbms/jlib/dbrparser.jar:$ORACLE_HOME/rdbms/jlib/dbranalyzer.jar: oracle.dbreplay.workload.checker.CaptureChecker /home/oracle/scripts/rat/db_replay_dir jdbc:oracle:thin:@baco.localdomain:1521:prodpay2

Almost all sessions have been captured in-flight.

PL/SQL blocks or functions have 'complicated' logic or

multiple commits in them, they are hard to synchronize and they behavior might change during replay.

wcr_cap_analysis.html


Captured workload: Sequential payment processing by Card type

Goal: Parallelized payment processing by Card Type

Scenario #1


Consolidate Replay – Workload folding

Consolidate Database Replay: Slicing an existing captured

workload into subsets by specifying a point in time

Allows scale-in testing replaying workload subsets together

Our Goal: Parallelized payment processing by Card Type

Start of Day Processing

VISA Payment Process

MasterCard Payment Process

OthersCardPayment Process

WorkloadTimeWL Subset

#1WL Subset #2

WL Subset #3



Workload from 1 day (7pm to 6am) – capture_payment_day

3 subsets of workload derived from 1 capture:

Start of Day Processing



OthersCard Payment Process

WorkloadTime

Day Finish

Replay this!



BEGIN_TIME and END_TIME are offset in seconds from the start of the capture (tricky)

Incomplete calls are discarded to avoid problematic results

Example (VISA workload subset #1):• 4500 seconds after start of capture (1 hour and 15 minutes)• Ends after 8100 seconds from the start of capture (1 hour of duration)

SQL> EXEC DBMS_WORKLOAD_REPLAY.GENERATE_CAPTURE_SUBSET ( INPUT_CAPTURE_DIR => 'DB_REPLAY_DIR', OUTPUT_CAPTURE_DIR => 'DB_REPLAY_VISA', NEW_CAPTURE_NAME => 'CardVisa_wkld’, BEGIN_TIME => 4500, BEGIN_INCLUDE_INCOMPLETE=> FALSE, END_TIME => 8100, END_INCLUDE_INCOMPLETE=> FALSE, PARALLEL_LEVEL=>1)



Preprocess capture from all directories:

EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_VISA');EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_MCARD');EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_OTHERS');

Create a consolidate replay schedule

EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE ('visa_mcard_others_schedule');

SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_VISA') FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_MCARD') FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_OTHERS') FROM dual;

EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;



Remap connections (DBA_WORKLOAD_CONNECTION_MAP) – You know how to do it

Prepare your consolidation replay:

SQL> EXEC DBMS_WORKLOAD_REPLAY.PREPARE_CONSOLIDATED_REPLAY(synchronization=>’OBJECT_ID’, think_time_auto_correct => TRUE )

• Each capture subset have a different SCN. SCN-based sync is not possible

• Only OBJECT_ID and OFF are supported as syncronization options

• Connect_time_scale and think_time_scale were left at default

• Best Results avoiding connect time modifications or think time modifications


Consolidate Replay – OBJECT_ID A “more advanced” synchronization scheme is used

Ensures that SCN’s for different captures do not clash

Better dealing with concurrency when actions do not touch the same object

Bugs happen:

If you find problems with OBJECT_ID sync you are left to no syncronization

Monitor the progress of your replay. A collection of scripts here help - Scripts to Debug Slow Replay (Doc ID 760402.1)



Start replay clients – At least one per subset

wrc system/oracle mode=replay replaydir=/home/oracle/scripts/rat/db_replay_dir/

sleep 1 wrc system/oracle mode=replay replaydir=/home/oracle/scripts/ rat/db_replay_dir/

…

Start database replay

SQL> EXEC DBMS_WORKLOAD_REPLAY.START_CONSOLIDATED_REPLAY;


Review divergences after replayApplication Level divergences:

• Example: Check if all the VISA cards were processed (manual SQL Script)

Data Level divergences:• Different number of rows returned by DML or SQL queries – Don’t match with capture

Error Level divergences:• Errors on Capture that are not on replay• Errors on Replay that were not captured• Different errors on capture and replay

select DBMS_WORKLOAD_REPLAY.REPORT(replay_id => 1, format => 'HTML') from dual;


Sequential vs Parallel: Compare it!

Import AWR from previous sequential replay to compare to parallel replay

dbms_workload_replay.import_awr(replay_id => 12, staging_schema => 'PAY_A');

Generate a compare replays report

spool report_compare_consolidate.htmlVAR v_clob CLOBBEGIN dbms_workload_replay.compare_period_report (replay_id1 => 11, replay_id2 => 12,format => DBMS_WORKLOAD_REPLAY.TYPE_HTML, result => :v_clob );END;/PRINT v_clobspool off


Sequential vs Parallel: Compare it!

Compare DB Time between periods (Sequential TP1 vs Parallel TP2);

Easier than read an AWR


Workload: Parallelized payment processing by Card Type

Goal: Parallelized payment processing by Card Type and ATM machines payment processing from another database capture

Scenario #2


Consolidate Replay – Time shifting

Allows shifting a replay to a time in the future and align with other replays

Stress testing by align peak loads

Peak number of sessionsAdd another workload here!



Align “Multibanco ATM processing” and Card Payment Processing to start exactly at same time



OthersCard Payment Process

ValidationsMultibanco

ATM processing

Workload time

Workload time

Delay time



Preprocess and delay the workloads to match ATM replay

EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE('payment_peak_align');SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_ATM') FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_VISA', 600) FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_MCARD', 650) FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_OTHERS', 660) FROM dual;EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;

EXEC DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY ('payment_peak_replay', 'payment_peak_align');

Delaysecs


Workload: Single workload capture on ATM machines payment

Goal: Stress test ATM machine payment processing on a very busy day – December 24th

Scenario #3


Consolidate Replay – Schema Remapping

A single workload consolidated into two different instances of the application

Original capture schema: MULTIBANCO_ATM (original)

Remapped schema: MULTIBANCO_ATM_B (remapped)

Validations

Multibanco ATM processing

Multibanco ATM processing

Validations

Fees Calc

Fees Calc

MULTIBANCO_ATM_B

Workload time


Consolidate Replay – Schema Remapping

Add the same workload twice to a consolidate replay

SQL> EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE(’multibanco_stress_schedule');

SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_ATM') FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_ATM') FROM dual;

EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;EXEC DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY ('payment_peak_replay', ‘multibanco_stress_schedule');

EXEC DBMS_WORKLOAD_REPLAY.SET_USER_MAPPING (2, ’MULTIBANCO_ATM', ’MULTIBANCO_ATM_B');


Workload: Applied to all previous scenarios

Goal: Get a “even” more realist parallelized payment processing

Scenario #4


Query Only – Warm up my buffer cache

When you start your replay, remember that your cache is "cold”

Replay may not be realistic or take too much time – capture is done with already some blocks in buffer cache.

Warm-up the buffer cache (if needed) with Query Only Replay before the real workload.

Query Only Replay replays read-only statements


Query Only – Warm up my buffer cache example

EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE ('visa_mcard_others_schedule');

SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_VISA’, query_only => TRUE) FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_MCARD’, query_only => TRUE) FROM dual;SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_OTHERS’,query_only => TRUE) FROM dual;

EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;


Final Notes

Consolidate Replay can be really helpful on stress testing and scaling up on “brand new” PoC system

Comparing periods is important: Capture vs Replay; Consolidate Replay vs Sequential Replays, etc

Follow capture and replay recommendations. Make sure that you know your capture and his subsets and start your replay with the “defaults”

Try different synchronization options if you find in trouble with OBJECT_ID. Remember that OBJECT_ID and OFF are the only options for Consolidated Replay


Q & A

Technology

Proof of Concept with Real Application Testing 12c