Upload
oracleonthebrain
View
2.609
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Oracle's Real Application Testing solution provides a cost-effective and easy-to-use solution for assessing the impact of change on production Oracle databases by capturing and replaying actual database workload and workflows. The solution provides greater business agility and reduces the risk and cost associated with change.
Citation preview
Oracle Database 11g
Real Application Testing
and Enterprise Manager Virtual ClassroomLynn Ferrante and Michael Whelan
Oracle Database 11g
Real Application Testing
and Enterprise Manager Virtual ClassroomLynn Ferrante and Michael Whelan
3
Database Testing Agenda
• Overview
• Database testing
• Eliminate the risk of change by assessing the impact of
change on production databases
• Database masking
• Eliminate risk of moving data to non-production environments
4
Database Testing
• Despite expensive testing success rate low
• Many issues go undetected
• System availability and performance negatively impacted
• Cause of low success rate
• Existing tools provide inadequate testing
• Simulate synthetic workload instead of replaying actual
production workload
• Provide partial workflow coverage
Database Replay makes real-world testing possible
5
1 2 3 4 5
Typical Steps in Test Phase
6
1: Install & Setup Test System
0 10 20 30 40 50 60 70 80 90 100
110
120
130
140
150
Real Application
Testing
Traditional Approach
Duration (Days)
24 Days
4 Days
7
2: Understand Application Usage
0 10 20 30 40 50 60 70 80 90 100
110
120
130
140
150
Real Application
Testing
Traditional Approach
Duration (Days)
+20 Days
+0 Days
8
3: Identify Key Transactions
0 10 20 30 40 50 60 70 80 90 100
110
120
130
140
150
Real Application
Testing
Traditional Approach
Duration (Days)
+20 Days
+0 Days
9
4: Develop Test Scripts
0 10 20 30 40 50 60 70 80 90 100
110
120
130
140
150
Real Application
Testing
Traditional Approach
Duration (Days)
+80 Days
+2 Days
10
5: Run Test Workload
0 10 20 30 40 50 60 70 80 90 100
110
120
130
140
150
Real Application
Testing
Traditional Approach
Duration (Days)
+5 Days
+5 Days
11
Innovate Faster
0 10 20 30 40 50 60 70 80 90 100
110
120
130
140
150
Real Application
Testing
Traditional Approach
Duration (Days)
149 Days
11 Days
12
0
20
40
60
80
Tim
e T
ak
en
(D
ays)
Install & Setup Understand
Application
Usage
Identify Key
Transactions
Generate
Workload
Run Test
DB Replay
LoadRunner
2
20
54
80
2420
DB Replay: 2 weeks
“Traditional” Testing: 30 weeks
Total Testing Time
500
Comparison of LoadRunner & Oracle Real
Application with e-Business Suite
13
Load Testing andAccelerators for
Packaged Apps & Web Services
Functional Testing andAccelerators for
Packaged Apps & Web Services
Test Management
Real Application Testing
Data Masking
Real UserExperience Monitoring
Service Level Management
Packaged Application Management
SOA Application Management
Diagnostic and Tuning
Provisioning, Configuration Management
Change Tracking and Synchronization
Oracle Enterprise Manager
ApplicationOperations
Management
Top-Down Application Management
ApplicationQuality
Management
Application Testing Suite
14
Real Application Testing
15
Real Application Testing
Remediate
Test
Deploy
• Value
• Rapid technology adoption
• Higher testing quality
• Business Benefit
• Lower cost
• Lower risk
Change
16
Ovum Summit
Oracle Real Application Testing – business agility
through superior testinghttp://www.oracle.com/corporate/analyst/reports/infrastructure/dbms/ovum
-real-application-testing.pdf
ROI
17
Real Application Testing
SQL
PERFORMANCE
ANALYZER
DATABASE
REPLAY
Identify SQL statement with performance regressions
caused by system changes
Captures a production workload and allows replay of the workload on a test system
Execute SQL concurrently
Execute SQL serially
18
Real Application Testing
Database Replay
19
Database Replay
• Replay actual production database workload in test environment
• Identify, analyze and fix potential instabilities before making changes to production
• Capture Workload in Production
• Capture full production workload with real load, timing & concurrency characteristics
• Move the captured workload to test system
• Replay Workload in Test
• Make the desired changes in test system
• Replay workload with full production characteristics
• Honor commit ordering
• Analyze & Report
• Errors
• Data divergence
• Performance divergence
Analysis & Reporting
20
Database Replay: Supported Changes
Changes
Unsupported
Changes Supported
•Database Upgrades, Patches
•Schema, Parameters
•RAC nodes, Interconnect
•OS Platforms, OS Upgrades
•CPU, Memory
•Storage
•Etc.
ClientClient
…Client
Middle Tier
Storage
Recording of External Client
Requests
21
Common Usage Scenarios
• Database initialization/config parameter changes
• Schema changes
• Changes to operating systems and hardware
• Gathering statistics
• Database upgrades including patch deployments
• Implementation of tuning recommendations
• Partitioning
• Compression
• etc
22
…
…
Database Replay Workflow
Production (10.2.0.4) Test (11.1)
Capture ReplayAnalysis &
ReportingProcess
StorageStorage
Mid-Tier
Replay Driver
Clients
23
Step 1: Workload Capture
File 1
File 2
File n
…
Production System • All external client requests
captured in binary files
• System background, internal activity excluded
• Minimal performance overhead for capture
• For RAC, shared and local file system supported
• Specify interesting time period for capture, e.g., peak workload, month-end processing, etc.
• Can capture on 10.2.0.4 and replay on 11g
File System
ClientClient
…Client
Middle Tier
Storage
25
Step 2: Process Workload Files
File 1
File 2
File n
…
File 1
File 2
File n
…
Metadata
Capture Files Replay Files
Test System• Setup test system
• Test DB is at same point in time
as before production capture
• Use RMAN to physically restore
production db from backup
• Use Snapshot standby
• Use imp/exp, Data Pump, etc.
• Processing transforms captured
data into replayable format
• Once processed, workload can be
replayed many times
• For RAC copy all capture files to
single location for processing
26
Step 3: Replay Workload
• Replays workload preserving timing, concurrency and dependencies of the capture system
• Replay Driver is a special client program that consumes processed workload and sends requests to the replay system
• Replay Driver consists of one or more clients. For workloads with high concurrency, it may be necessary to start multiple clients to drive workload
Test System
File 1
File 2
File n
…
Metadata
Replay Files
Replay Driver
27
Analysis & Reporting
• Comprehensive reports are provided for analysis purposes
• There (3) types of divergences are reported
• Data Divergence: Number of rows returned by each call are
compared and divergences reported
• Error Divergence: For each call error divergence is reported
• New: Error encountered during replay not seen during capture
• Not Found: Error encountered during capture not seen during
replay
• Mutated: Different error produced in replay than during capture
• Performance Divergence
• Capture and Replay Report: Provides high-level performance
information
• ADDM Report: Provides in-depth performance analysis
• AWR, ASH Report: Facilitates comparative or skew analysis
28
Demo
29
Real Application Testing
SQL Performance Analyzer
30
• Enables identification of SQL performance regressions beforeend-users can be impacted
• SPA can help with any change that impacts SQL execution plan
• DB upgrades
• Optimizer statistics refresh
• New indexes, Materialized Views, Partitions, etc.
• Automates SQL performance tracking of hundreds of thousands of SQL statements – impossible to do manually
• Captures SQL workload with low overhead
• Integrated with SQL Tuning Advisor and SQL Plan Baselines for regression remediation
SPA Benefits
31
……
ClientClient
…Client
Capture SQL
Middle Tier
Storage
Oracle DB
Re-execute SQL Queries
Production Test
Use SQL Tuning Advisor to tune regression
32
SQL Performance Analyzer: Workflow
Production Test
Capture
SQL (STS)
Transport
STS
Execute SQL
Pre-change
Execute SQL
Post-change
Compare
Perf.
Steps (1) (2) (3) (4) (5)
(6) Reiterate
(7)
No
Yes(7)
Done?
Make Change
Production Change / Tuning Deployment
Tuned System
33
Step 1: Capture SQL Workload
• SQL Tuning Set (STS) used to store SQL
workload
• STS includes:
• SQL Text
• Bind variables
• Execution plans
• Execution statistics
• Incremental capture used to populate
STS from cursor cache over a time
period
• SQL tuning set’s filtering and ranking
capabilities filters out undesirable SQL
• SQL workload captured in 10.2.0.1 and
higher can be used for SPA tasks in 11gProduction Database
Cursor Cache
SQL Tuning Set
Incremental Capture
34
Step 2: Move SQL Workload to Test System
Production Database
Cursor Cache
SQL Tuning Set
Test Database
SQL Tuning Set
• Use Enterprise Manager
Export/Import
35
Step 3: Execute SQL Before Making Change
• Establishes SQL workload performance
baseline
• SQL execution plan and statistics
captured
• SQL executed serially (no concurrency)
• Each SQL executed only once
• DDL/DML skipped
• Option to do Explain Plan only analysis
SQL Tuning Set
Fetch Next SQL
SQL Performance Analyzer
Execution Plan &
Statistics
Test Execute
Save Result
36
Step 4: Execute SQL After Making Change
• Re-execute SQL after change
• Gathers new SQL execution plans
and statistics
SQL Tuning Set
SQL Performance Analyzer
Fetch Next SQL
Execution Plan &
Statistics
Test Execute
Save Result
Completed
• Manually implement the planned change
• Database upgrade, patches
• Optimizer statistics refresh
• Schema changes
• Database parameter changes
• Tuning actions, e.g., SQL Profile creation
37
Step 5: Compare & Analyze Performance
Completed Completed
Compare
SQL Performance
SQL Performance Analyzer
Analysis Report
• Compare performance using
different metrics, e.g.,
• Elapsed Time
• CPU Time
• Optimizer Cost
• Buffer Gets
• SPA Report shows impact of
change for each SQL
• Improved SQL
• Regressed SQL
• Unchanged SQL
• Fix regressed SQL using SQL
Tuning Advisor or SQL Plan
Baselines
38
SPA Report
39
SPA Report
40
Demo
41
Real Application Testing Summary
• Provides cost-effective and easy-to-use solution of assessing impact of change on production systems
• Holistic and actual workload testing results in lower risk
• Brings down testing cycle from months down to days
• Lowers hardware costs by eliminating need for mid-tier and application setup on test system
• Maximizes ROI by leveraging Diagnostics and Tuning Pack to remediate regressions
• With Real Application Testing businesses can
• Stay competitive
• Improve profitability
• Be compliant
42
Real Application Testing: Tools of the Trade
SQL Performance Analyzer Database Replay
What is it? • Predicts SQL performance deviations
before end-users can be impacted,
helps assess impact of change on
SQL response time
• Replays real database workload
on test system, helps assess
impact of change on workload
throughput
How it works? • Executes each SQL, stored in SQL
Tuning Set, in isolation using
production context and then
compares before and after execution
plans and run-time statistics
• Captures workloads and replays it
with production characteristics
including concurrency,
synchronization &
dependencies
When to use? • Unit testing of SQL with the goal to
identify the set of SQL statements
with improved/regressed performance
• Comprehensive testing of all
sub-systems of the database
server using real production
workload
SQLSQL
Dependency
Concurrency
Speed up/down
43
Database Diagnostic Pack
44
Database Diagnostic Pack
Introduction
• Part of the database management pack family
• Provides unique functionalities
• Industry’s first performance self-diagnosing engine
• Lightweight, Automatic, Out-of-box performance data warehouse
• Guided problem resolution
• Introduces a new, easy to use performance tuning methodology
• Not just another tool
• Core functionality built right into the database server
• Makes it most efficient, integrated and accurate
45
Database Diagnostic Pack
Key Features
• Automatic Workload Repository (AWR)
• Active Session History (ASH)
• Automatic Database Diagnostic Monitor (ADDM)
46
SQL
Advisor
High-load
SQL
IO / CPU
issues RAC issues
Automatic Diagnostic Engine
Snapshots in
Automatic Workload
Repository
Self-Diagnostic Engine
System
Resource
Advice
Network +
DB config
Advice
• Top Down Analysis Using
AWR Snapshots
• Throughput centric - Focus on
reducing time ‘DB time’
• Classification Tree - based on
decades of Oracle
performance tuning expertise
• Real-time results
• Don’t need to wait hours to
see the results
• Pinpoints root cause
• Distinguishes symptoms
from the root cause
• Reports non-problem areas
• E.g. I/O is not a problem
How Does ADDM Work?
47
Database Tuning Pack
Introduction
• Part of the database management pack family
• Provides unique, automatic and deterministic SQL
tuning functionality
• Core functionality built in the Oracle Database 10g
kernel
• SQL Tuning Advisor
• SQL Access Advisor
• Requires diagnostic pack
48
SQL Profiles
Contains auxiliary information specific to a given SQL statement
– Customized optimizer settings
Based on past execution history
– Compensation for missing or stale statistics
– Compensation for errors in optimizer estimates
Estimation errors occur due to data skews and correlations, complex filters and joins
Doesn’t freeze the execution plan
Doesn’t require any change to the SQL text– Ideal for Packaged Apps
Persistence: Works across shutdowns & upgrades
Transportable across databases (10.2)
49
Data Masking Pack
50
Oracle Data Masking Data Masking for Non-Production Environments
• Eliminate risk of moving data to
non-production environments
• Share data with partners
without exposing sensitive data
• No impact on production
environments
• Sensitive data never leaves the
database (masked in place)
• Automated masking process
preserves application integrity
LAST_NAME SSN SALARY
AGUILAR 203-33-3234 40,000
BENSON 323-22-2943 60,000
LAST_NAME SSN SALARY
ANSKEKSL 111—23-1111 40,000
BKJHHEIEDK 111-34-1345 60,000
Production
Non-Production
51
Summary
• Real Application Testing for managing risk• Provides cost-effective and easy-to-use solution of assessing
impact of change on production systems
• Lowers hardware costs by eliminating need for mid-tier and application setup on test system
• Maximizes ROI by leveraging Diagnostics and Tuning Pack to remediate regressions
• Diagnostic and Tuning Pack for proactive
management
• Industry’s first performance self-diagnosing engine
• Data Masking for security of data
52
0
20
40
60
80
Tim
e T
ak
en
(D
ays)
Install & Setup Understand
Application
Usage
Identify Key
Transactions
Generate
Workload
Run Test
DB Replay
LoadRunner
2
20
54
80
2420
DB Replay: 2 weeks
“Traditional” Testing: 30 weeks
Total Testing Time
500
Comparison of Traditional Approach &
Oracle Real Application with e-Business
Suite