Upgrade to Oracle Database 19cWith or without Multitenant

Mike Dietrich

Distinguished Product Manager

Oracle Global Services

Distinguished Product Manager

Database Upgrade and Migrations

Mike Dietrich

https://MikeDietrichDE.com

@MikeDietrichDE

mikedietrich

Slides | https://MikeDietrichDE.com

Twitter | twitter.com/MikeDietrichDE

Content Overview

Performance Stability

5AutoUpgrade

3Before Upgrade

2Release Strategy

1Migration to Multitenant

4

Ph

oto

by

Sp

ace

X o

n U

nsp

lash

Database Upgrade

Why you "want" to upgrade

Lifetime Support Policy

20

09

20

10

20

11

20

12

20

13

20

14

20

15

20

16

20

17

20

18

20

19

20

20

20

21

20

22

20

23

20

24

20

25

20

26

20

27

Oracle 18(12.2.0.2)

EXTENDEDOracle 12.1

EXTENDEDOracle 19(12.2.0.3)

EXTENDEDWaived EXTENDEDOracle 11.2 MARKETDRIVEN

Oracle 12.2.0.1 LIMITED

Premier Support Waived Extended Support Paid Extended Support Market Driven Support Limited Error Correction

Innovation

EXTENDEDLong Term Support

Innovation

Premier Support Paid Extended Support

Release Types | Long Term Support vs Innovation Releases

Long Term Support Release

• 5 years of Premier Support followed by 3 years of Extended Support

Innovation Release

• 2 years of Premier Support, but there is no Extended Support

• MOS Note: 742060.1 - Release Schedule of Current Database Releases

Oracle 12.2 Release Family

Includes:

• Oracle 12.2.0.1, Oracle 18c (12.2.0.2), Oracle 19c (12.2.0.3)

• MOS Note:742060.1 - The Single Source of Truth

• MOS Note:161818.1 - Releases Support Status Summary

20

16

20

17

20

18

20

19

20

20

20

21

20

22

20

23

20

24

20

25

20

26

20

27

Oracle 18(12.2.0.2)

EXTENDED

Oracle 12.2.0.1

Oracle 19(12.2.0.3)

LIMITED

Lifetime Support Policy Brochure

http://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdf

Old Format

12.2.0.1.0 Year. Update. Revision

New Format

19.8.0

Yearly Releases | 3 Number Format

Why will I speak only about Oracle 19c?

https://docs.oracle.com/en/database/oracle/oracle-database/20/rnrdm/deprecated-desupported-features.html#GUID-61D44BE2-21DE-463C-BEA9-8F8F1C75E45F

Your key to successful Database Upgrades

1 2 3 4

Download and install Oracle 19c

Download and install newest RU

Download and usenewest AutoUpgrade

eDelivery MOS: 2521164.1 MOS: 2485457.1

Get started | Quick Start Guide

• Simple overview

• Read it, try it

• Download from oracle.com

Check

Before Upgrade

Ph

oto

by

Pa

blo

He

imp

latz

on

Un

spla

sh

Supportability | OS Certification

Supportability | OS Certification

Supportability | OS Certification

More Links on the Upgrade Blog

Oracle Clusterware Certification on OL8/RHEL8https://mikedietrichde.com/2020/05/14/of-course-oracle-clusterware-is-certified-on-ol8-rhel8-as-well/

Oracle Database 19c Certification on OL8/RHEL8https://mikedietrichde.com/2020/05/11/oracle-database-19c-is-certified-on-ol8-and-rhel8/

Database Upgrade | Supported Releases

11.2.0.4

12.1.0.2

12.2.0.1

19c

18c

https://mikedietrichde.com/2019/02/14/oracle-database-19c-requires-ol7-rhel7-or-sles12-or-newer/

Oracle 19c | Installation

Gold Image

1. Create ORACLE_HOME directory

2. Download image

3. Unpack into ORACLE_HOME

4. ./runInstaller

5. root.sh

Oracle 19c | RPM Installation

RPM

• yum install -y oracle-database-preinstall-19c

• yum -y localinstall oracle-database-ee-19c-1.0-1.x86_64.rpm

• rpm installs always into: /opt/oracle/product/19c/dbhome_1

• Not practical for typical environments

Oracle 19c | Read-Only Home

Separate configuration and binary files

• ORACLE_BASE_HOME

• ORACLE_BASE_CONFIG

• Please read more:https://docs.oracle.com/en/database/oracle/oracle-database/19/ladbi/understanding-read-only-oracle-homes.html#GUID-AE1973D7-0325-4AD8-9FC9-D7BA3C3C3F37

In r/w: equal to ORACLE_HOME

APEX | Upgrade APEX upfront

APEX upgrade

• Not part of the database upgrade

• MOS Note: 1088970.1 - Master Note APEX Upgrades

APEX certification

• Minimum APEX Version for Oracle 19c: APEX 18.2

• MOS Note: 1344948.1 - APEX Database and Web Server Certification

Download the newest APEX

• https://www.oracle.com/tools/downloads/apex-v191-downloads.html

Upgrade 19c | Speed it up

• Check when dictionary stats have been gathered the last time

• Refresh stats a day before the upgrade

column OPERATION format a40

set linesize 200

select to_char(max(END_TIME),'DD-MON-YY hh24:mi') LATEST, OPERATION

from DBA_OPTSTAT_OPERATIONS

where OPERATION in ('gather_dictionary_stats', 'gather_fixed_objects_stats’)

group by operation;

Upgrade 19c | Make it run faster …

• Stats not gathered on several clustered indexes (bug 25286819)

• Wait … autoupgrade.jar does this for you automatically!

exec dbms_stats.gather_schema_stats('SYS');

exec dbms_stats.gather_index_stats('SYS','I_OBJ#');

exec dbms_stats.gather_index_stats('SYS','I_FILE#_BLOCK#');

exec dbms_stats.gather_index_stats('SYS','I_TS#');

exec dbms_stats.gather_index_stats('SYS','I_USER#');

exec dbms_stats.gather_index_stats('SYS','I_TOID_VERSION#');

exec dbms_stats.gather_index_stats('SYS','I_MLOG#');

exec dbms_stats.gather_index_stats('SYS','I_RG#');

Upgrade 19c | Effect of having good stats

• The larger the dictionary, the bigger the effect

• This example has been done with one of the tiny Hands-On Lab databases

Duration Reduction

No dictionary and fixed objects stats 15 min 55 sec

Gathered dictionary and fixed objects stats 14 min 10 sec 11 %

Gathered schema and cluster index stats 13 min 41 sec 3.4 % to previous

Total downtime saved 2 min 14 sec 14 % overall

Operating System | Recommendations

• MOS Note: 1587357.1Oracle Database on Unix AIX, HP-UX, Linux, Solaris and MS Windows –Installation and Configuration Requirements Quick Reference

AutoUpgrade

The ONLY recommended way to upgrade databases

Ph

oto

by

Jo

shu

a S

ort

ino

on

Un

spla

sh

Download | Always get the latest autoupgrade.jar

MOS Note: 2485457.1

Upgrade | AutoUpgrade

upg>

Job Console

11.2.0.412.1.0.2

12.2.0.1

18c

autoupgrade.jarMOS Note: 2485457.1

2MB

config.cfg

autoupgrade.jar

12.2.0.1

18.5.0

19c

19c

AutoUpgrade | Architecture

UpgradeJob Manager

UpgradeDispatcher

config.cfg

Database Post-Upgrade

Stats Refresh Time Zone Adjustment /etc/oratab

Database Upgrade

Command Line

Database Pre-Checks

preupgrade.jar Automatic Fix-Ups

System Checks

Space Check Requirements

AutoUpgrade | Requirements

• Java 8 or newer

• Default in your $ORACLE_HOME since 12.1.0.2

• Check: $ORACLE_HOME/jdk/bin/java -version

• The AutoUpgrade package

• A jar file of 2 MB in size

AutoUpgrade | Availability and Support

• CDB and non-CDB upgrades

• All operating systems running Oracle Database servers

• All editions (SE2, EE)

• Single instance and RAC databases

• Upgrades from version:

• 11.2.0.4 and newer

• Upgrades to version:

• 12.2.0.1 with Jan 2019 Update or newer

• 18.5.0 (includes Jan 2019 Update) or newer

• 19.3.0 or newer

AutoUpgrade | What you don't need …

• Agents

• EM Cloud Control

• DBUA

• Extra license

AutoUpgrade | Coming soon!

• Data Guard

• Disable DG Broker and defer log transport

• Unplug/plug/upgrade

AutoUpgrade | What it doesn't do

• Grid Infrastructure upgrade

• This is still done by the OUI when you invoke rootupgrade.sh

• Create a copy of your database

• Move your database to another host

• Create a new CDB to plugin after upgrade

AutoUpgrade | Blog Post Series

https://mikedietrichde.com/2019/04/29/the-new-autoupgrade-utility-in-oracle-19c/

AutoUpgrade | Use the newest version

• $ java -jar autoupgrade.jar –version

build.hash 3b51369

build.version 19.9.1

build.date 2020/07/10 11:20:49

build.max_target_version 19

build.supported_target_versions 12.2,18,19

build.type production

Oracle 19.7.0:

build.hash 04dd9f2

build.version 19.7.5

build.date 2020/02/11 15:28:49

build.max_target_version 19

build.type production

Download the newer version

AutoUpgrade | Create sample config file

$ java –jar autoupgrade.jar -create_sample_file config

Created sample configuration file/home/oracle/sample_config.cfg

AutoUpgrade | Example config file

global.autoupg_log_dir=/home/oracle/logs

upg1.dbname=DB12

upg1.start_time=NOW

upg1.source_home=/u01/app/oracle/product/12.2.0.1

upg1.target_home=/u01/app/oracle/product/19

upg1.sid=DB12

upg1.log_dir=/home/oracle/aux/DB12

upg1.upgrade_node=localhost

upg1.target_version=19

upg2.dbname=UPGR

upg2.start_time=NOW

upg2.source_home=/u01/app/oracle/product/11.2.0.4

. . .

Clarification

• One config file can be used for hundreds and thousands of databases

• You can start now

• But you can configure much more …

• Important: You don't need to!

AutoUpgrade | Adjust init parameters

#

# Example global pfile configuration entries

#

global.del_during_upgrade_pfile=/database/pfiles/global_during_delinit.ora

global.add_during_upgrade_pfile=/database/pfiles/global_during_addinit.ora

global.del_after_upgrade_pfile=/database/pfiles/global_after_delinit.ora

global.add_after_upgrade_pfile=/database/pfiles/global_after_addinit.ora

#

# Example local pfile configuration entries

#

upg2.del_during_upgrade_pfile=/database/pfiles/hr_during_delinit.ora

upg2.add_during_upgrade_pfile=/database/pfiles/hr_during_addinit.ora

upg2.del_after_upgrade_pfile=/database/pfiles/hr_after_delinit.ora

upg2.add_after_upgrade_pfile=/database/pfiles/hr_after_addinit.ora

AutoUpgrade | Execute external scripts

#

# Example global script execution

#

global.before_action=/database/scripts/run_this_before.sh Y

global.after_action=/database/scripts/run_this_after.sh

#

# Example local script execution

#

upg2.before_action=/database/scripts/run_this_on_UPG2_before.sh

upg2.after_action=/database/scripts/run_this_on_UPG2_after.sh

• Permitted extension options:

• Unix shell (.sh)

• Microsoft Windows batch (.bat, .cmd)

• Microsoft Windows PowerShell (.ps1)

Pass return codeback to AutoUpgrade

AutoUpgrade | Protection - GRP options

#

# By default a GRP will be created. Default for EE: Yes

#

upg2.restoration=no

#

# By default a GRP will be kept. This option overwrites the default.

#

upg2.drop_grp_after_upgrade=yes

AutoUpgrade | Other options

#

# Example global directive - Default: Underscores stay as before

#

global.remove_underscore_parameters=yes

#

# If unset, utlrp.sql won't be run - Default: YES

#

upgr2.run_utlrp=no

#

# If unset, time zone upgrade won't be performed - Default: YES

# As Timezone can't be downgraded, it may be useful to postpone and skip it

#

upgr2.timezone_upg=no

AutoUpgrade | Start time

• VALID values

• Format: DD/MM/YYYY HH24:MI:SS

• NOW [this is the default]

• 30/12/2018 15:30:00

• 01/11/2019 01:30:15

• 2/5/2019 3:30:50

Example | Adjust COMPATIBLE post upgrade

Add to your config file:

• upg1.drop_grp_after_upgrade=YES

• upg1.after_action=/home/oracle/scripts/change_compatible.sh

#!/bin/bash

. db19

sqlplus /nolog @/home/oracle/scripts/change_compatible.sql

connect / as sysdba

alter system set compatible='19.0.0' scope=spfile sid='*';

shutdown immediate

startup

exit

AutoUpgrade | 4 Operation Modes

autoupgrade.jar

-deploy

-analyze -fixups -upgrade

AutoUpgrade | Best Practice

$ java -jar autoupgrade.jar -config config.cfg -mode analyze

autoupgrade.jar

-deploy

-analyze

$ java -jar autoupgrade.jar -config config.cfg -mode deploy

AutoUpgrade | Move to new hardware

$ java -jar autoupgrade.jar -config config.cfg -mode analyze

autoupgrade.jar

-analyze -fixups

autoupgrade.jar

-upgrade

$ java -jar autoupgrade.jar -config config.cfg -mode deploy

$ java -jar autoupgrade.jar -config config.cfg -mode upgrade

Source System Target System

AutoUpgrade | Move to new hardware

autoupgrade.jar

-analyze

Source System

autoupgrade.jar

-fixups

global.autoupg_log_dir=/home/oracle/upg_logs

upg1.dbname=DB12

upg1.start_time=NOW

upg1.source_home=/u01/app/oracle/product/12.1.0.2

upg1.target_home=/u01/app/oracle/product/19

upg1.sid=DB12

upg1.log_dir=/home/oracle/upg_logs

upg1.upgrade_node=localhost

upg1.target_version=19

upg1.restoration=no

AutoUpgrade | Move to new hardware

autoupgrade.jar

-upgrade

Target System

global.autoupg_log_dir=/home/oracle/logs

upg1.dbname=DB12

upg1.start_time=NOW

upg1.source_home=/tmp

upg1.target_home=/u01/app/oracle/product/19

upg1.sid=DB12

upg1.log_dir=/home/oracle/logs

upg1.upgrade_node=localhost

upg1.target_version=19

upg1.restoration=no

• https://mikedietrichde.com/2020/08/03/oracle-autoupgrade-between-two-servers/

AutoUpgrade | Job Console

• Most important commands

upg>

lsj // List jobs

resume -job <number> // Restarts a job

status -job <number> // Show job status

restore -job <number> // Restores database from GRP

abort -job <number> // Aborts the specified job

AutoUpgrade | Log File Structure

• Logs written in TEXT and JSON format

• /cfgtoollogs

• ./upgrade/auto/status

• /database_1

• ./job_number

• ./prechecks

• ./preupgrade

• ./prefixups

• ./drain

• ./dbupgrade

• ./postupgrade

• ./temp

• /database_2

• …

Status Logs

HTML Report

Upgrade Logs

AutoUpgrade | Monitor the Progress

• cd ~/upg_logs/cfgtoollogs/upgrade/auto

• python -m SimpleHTTPServer 8080

• Open a browser:

• http://127.0.0.1:8080/state.html

• https://mikedietrichde.com/2020/07/16/autoupgrade-refresh-status-information-automatically/

AutoUpgrade | What if ...

1. ZIP the entire global log directory:

java -jar autoupgrade.jar -config config.cfg -zip

2. Add the alert.log

3. Upload it to MOS

AutoUpgrade | What if ... you have RAC?

• Since AutoUpgrade 19.8 (2020/03/27)

• RAC database upgrades are supported

• Works best with SPFILE in ASM

• Oracle Grid Infrastructure upgrade must be done separately upfront

After qualifying the new AutoUpgrade tool on a representative portion of our database landscape we found that tool was doing a great job and is production-ready. In our automation tool we have removed a lot of “home-grown” code, and replaced it with AutoUpgrade functionality.

Since August 2019 all upgrades at SimCorp have been executed using the AutoUpgrade tool.

Daniel Overby Hansen

Lead DeveloperSimCorp A/S - Denmark

Slides | https://MikeDietrichDE.com

Migration Strategies

Migration to Oracle Multitenant

Ph

oto

by

Ma

thia

s Je

nse

no

n U

nsp

lash

Oracle Multitenant | Oracle 19c

• Oracle 18c and Oracle 19c belong to the Oracle Database 12.2 release family

• Oracle 19c supports both non-CDB and PDB databases!

• For further details see MOS Note: 742060.1

Please note: After Oracle 19c, there will be only the CDB architecture

https://docs.oracle.com/en/database/oracle/oracle-database/20/upgrd/behavior-changes-deprecated-desupport-oracle-database.html#GUID-BA926361-95C9-4EB9-B4A7-FD3CC674A395

And by the way …

Since Oracle 19c you can have 3 user-created PDBs without need for a Multitenant license!!!

Upgrade to 19c | Migration to Multitenant

AutoUpgrade One-Command Orchestration

Oracle 11.2.0.4

AutoUpgrade

Oracle 19.8.0

Oracle 19.8.0

CDB

PDB$SEED

Oracle 19.8.0

CDB

PDB$SEED

1️⃣2️⃣ Plug into CDB

3️⃣Conversion withnoncdb_to_pdb.sql

AutoUpgrade | Migration to Multitenant

• Use these parameters in your config file

• The receiving CDB must be precreated already

# Specify the SID of the target CDB to plug into after upgrade

# Mandatory if you'd like to plugin

upg2.target_cdb=CDB2

# Give the non-CDB a new name when it becomes a PDB

# Optional

upg2.target_pdb_name=PDB1

# Define whether you'd like to use the COPY option

# Optional - if not specified, NOCOPY will be used

upg2.target_pdb_copy_option=file_name_convert('ORCL','PDB1')

Move to Multitenant | Automation?

• AutoUpgrade automates non-CDB-to-PDB migration

• Requires customer creating the destination CDB

• Can't lift the migration constraints (downtime, fallback)

• No automation for Endianness migrations

Videos | Upgrade Channel on Youtube

• https://www.youtube.com/channel/UCTKvCZwAbOeTSRFQDAh7yXw

• AutoUpgrade

• https://youtu.be/kr3aZrqA9Go [11:57 min]

• https://youtu.be/dO_y8sOasjs [DOAG 2019]

• Preupgrade Steps

• https://youtu.be/2R0Plcn7Udk [6:27 min]

• Database Upgrade

• https://youtu.be/ZqW3hS64hng [4:17 min]

• Postupgrade and Time Zone change

• https://youtu.be/kEBLqYDmspw [7:58 min]

DEMO

AutoUpgrade | Secret

• You can use AutoUpgrade with these parameters to plugin any 19c database

• Even without doing an upgrade

global.autoupg_log_dir=/home/oracle/logs

upg1.dbname=DB12

upg1.start_time=NOW

upg1.source_home=/u01/app/oracle/product/19

upg1.target_home=/u01/app/oracle/product/19

upg1.sid=DB12

upg1.log_dir=/home/oracle/logs

upg1.upgrade_node=localhost

upg1.target_version=19

upg1.restoration=no

upg1.target_cdb=CDB2

upg1.target_pdb_name=PDB1

Upgrade | AutoUpgrade Plugin Demo Video

https://mikedietrichde.com/2020/05/20/autoupgrade-and-plug-in-to-a-cdb-with-a-single-command/

Time for a break

After Upgrade

Important things to do after the upgrade completed successfully

Things to do right after upgrade

Configure statistics history retention period

• Check space usage:

• Check retention

• Default: 31 days

• Adjust setting

• Example: 10 days

SQL> select SPACE_USAGE_KBYTES/1024 MB

from V$SYSAUX_OCCUPANTS where

OCCUPANT_NAME='SM/OPTSTAT';

SQL> select

DBMS_STATS.GET_STATS_HISTORY_RETENTION from DUAL;

SQL> exec

DBMS_STATS.ALTER_STATS_HISTORY_RETENTION(10);

Post Upgrade | Time Zone Change

• Adjust time zone post upgrade

• Newest time zone patch:

• MOS Note:412160.1

• New scripts in ?/rdbms/admin:

• utltz_countstar.sql

• utltz_upg_check.sql

• utltz_upg_apply.sql

Oracle Database Release Default TZ Version

11.2.0.2 - 11.2.0.4 DST V14

12.1.0.1, 12.1.0.2 DST V18

12.2.0.1 DST V26

18c DST V31

19c DST V32

Post Upgrade | Unified Audit Trail

• Internal structure changed to improve query performance of UNIFIED_AUDIT_TRAIL view

• Old records must be converted to new format

• To convert:SQL> EXEC DBMS_AUDIT_MGMT.TRANSFER_UNIFIED_AUDIT_RECORDS;

• Only applicable if upgrading from Oracle Database 12.1

More Information | Client Connectivity

MOS Note: 207303.1Client / Server Interoperability Support Matrix for Different Oracle Versions

Things to know

Best Practices and things to take care on

Best Practice | Database COMPATIBLE Parameter

• When should you change COMPATIBLE?

• 7-10 days after the upgrade

• Which value?

• 11.2.0

• 12.1.0

• 12.2.0

• 18.0.0

• 19.0.0

• Change COMPATIBLE for RUs?

• NEVER!

• Flashback to GRP• Downgrade

Oracle 19c | Multimedia Removal

• The Oracle Multimedia API gets removed during upgrade

• The Locator still exists and works

• ORDIM component remains VALID in DBA_REGISTRY

• Recommendation

• Check, if you use Oracle Multimedia

• If not, you can remove it before upgrade

• More information

• https://mikedietrichde.com/2019/02/18/oracle-multimedia-will-be-removed-in-oracle-database-19c/

• MOS Note: 2347372.1 – Oracle Multimedia Statement of Direction

• MOS Note: 2375644.1 – How To Migrate Data From Oracle Multimedia Data Types to BLOB columns

Oracle 19c | Streams Removal

• The Oracle Streams API gets removed during upgrade

• See:

• https://docs.oracle.com/en/database/oracle/oracle-database/19/upgrd/behavior-changes-deprecated-desupport-oracle-database.html#GUID-7E90C2D3-853A-45B2-AC6D-C9326798E82B

Oracle 19c | DBMS_JOB

DBMS_JOB package is still supported

• Grant the CREATE JOB privilege

• During and after upgrade, for each job in DBMS_JOB a corresponding entry will be created with DBMS_SCHEDULER

• The check in preupgrade.jar is only checking for inconsistencies or any issues

MOS Note: 2380601.1 - Database Preupgrade tool check list

Health Check | hcheck.sql

If your database is highly important, do a health check

• hcheck.sql - Script to Check for Known Problems (MOS Note: 136697.1)

Your key to successful Database Upgrades

1 2 3 4

Download and install Oracle 19c

Download and install newest RU

Download and usenewest AutoUpgrade

eDelivery MOS: 2521164.1 MOS: 2485457.1

SPM, STA, RAT for Performance Stability

Testing | Typical Mistakes

• Only 10% of real data used

• Data sets artificially created

• Tests done on a laptop

• No testing tools used

• Data from 6 months ago

• No statistics refreshed

• Testing?? Waste of time!

• Real experts fix it after go-live …

Ph

oto

by

Elij

ah

O'D

on

ne

ll o

n U

nsp

lash

Best Practice | COMPATIBLE vs Optimizer

• COMPATIBLE and OPTIMIZER_FEATURES_ENABLE

• Fully independent from each other

• Set COMPATIBLE to the default of the release, e.g. 19.0.0

• Change OPTIMIZER_FEATURES_ENABLE only when you have to

• Avoid it if possible

• This is not a Swiss Army knife!

• You will turn off a lot of great features

Testing Tools | Workflow

Collect execution plans before upgrade

Compare AWR Snapshots

Verify them with SQL Performance

Analyzer

Regressed plans?

SQL Plan Management

SQL Tuning Advisor

Verify functionality and performance with

Database Replay

Testing Tools | Collect Plans on Production

STS

Capturing from cursor cache and AWR into an STS does not require an extra license for EE anymore:

Find it under: "SQL Tuning Sets" at:https://docs.oracle.com/database/121/DBLIC/editions.htm#DBLIC109

Testing Tools | Comparison

Collect execution plans before upgrade

Compare AWR Snapshots

Verify them with SQL Performance

Analyzer

Regressed plans?

SQL Plan Management

SQL Tuning Advisor

Verify functionality and performance with

Database Replay

Testing Tools | AWR Diff Reports

Upgrade / Migrate

11.2.0.4 19.8.0

Snapshot Interval

Compare

AWR Diff Report: awrddrpt.sql Snapshot Interval

Testing Tools | SQL Performance Analyzer

Collect execution plans before upgrade

Compare AWR Snapshots

Verify them with SQL Performance

Analyzer

Regressed plans?

SQL Plan Management

SQL Tuning Advisor

Verify functionality and performance with

Database Replay

Testing Tools | SQL Performance Analyzer

SQL Performance Analyzer

STS

Capture SQLsinto a STS

Load STS

STS

Evaluate, compare and tune –over and over again!

11.2.0.4 19.8.0

Upgrade / Migrate

Testing Tools | SPA - Regressed Report

Testing Tools | SQL Plan Management

Collect execution plans before upgrade

Compare AWR Snapshots

Verify them with SQL Performance

Analyzer

Regressed plans?

SQL Plan Management

Verify functionality and performance with

Database Replay

Testing Tools | Situation after change

• Challenging to "freeze" execution plans

Parse ExecutePlan acceptable

Plan possiblynot acceptable

Conditions change:New statistics, parameter change, upgrade

GB

HJ

...HJ

GB

NL

...NL

SQL

SQL

ExecuteParse

SQL Plan Management | Plan Stability

GB

HJ

...HJ

Plan Baseline

Plan History

STS

STSExport/Import for Migrations

Upgrade

Plans will be takenfrom STS and addedto the SQL Plan Baseline

DBMS_SPM.LOAD_PLANS_FROM_SQLSET

Testing Tools | SQL Tuning Advisor

Collect execution plans before upgrade

Compare AWR Snapshots

Verify them with SQL Performance

Analyzer

Regressed plans?

SQL Plan Management

SQL Tuning Advisor

Verify functionality and performance with

Database Replay

SQL Tuning Advisor | SQL Profiles

• SQL Profiles provide more information to the optimizer

• Part of the Tuning Pack

• FORCE_MATCH=TRUE

• Don't change the application code

• Persistent, transportable (MOS Note:751068.1), can be tested without risk

• Example:

Testing Tools | Database Replay

Collect execution plans before upgrade

Compare AWR Snapshots

Verify them with SQL Performance

Analyzer

Regressed plans?

SQL Plan Management

SQL Tuning Advisor

Verify functionality and performance with

Database Replay

Testing Tools | Database Replay

Database Replay

Capture Workload

Replay Workload

Compare – then restoreand run again with changes

11.2.0.4 19.8.0

Migrate

Preprocess Workload

Help me - I have an upgrade problem …

In>95%of all cases

Ph

oto

by

nik

ko m

aca

spa

co

n U

nsp

lash

“ ”

an "upgrade problem" in reality is a performance issue after upgrade, or not a database upgrade topic.

There is exactly one way to mitigate the risk:

TESTING!

Automatic SQL Tuning Set

In Oracle 19.7.0, a Automatic SQL Tuning Set gets populated

• Some customers reported high growth and consumption in SYSAUX

• See: https://mikedietrichde.com/2020/05/28/do-you-love-unexpected-surprises-sys_auto_sts-in-oracle-19-7-0/

• If you want to disable it:• BEGIN

DBMS_AUTO_TASK_ADMIN.DISABLE(client_name=>'Auto STS Capture Task', operation=>NULL, window_name=>NULL);

END;

/

• Task is not enabled by default from Oracle 19.8.0 on

Oracle 19c | _cursor_obsolete_threshold

• MOS Note: 2431353.1High Version Counts for SQL statements (>1024) post upgrade To 12.2 and above causing database slow performance

• Since Oracle 12.2.0.1, the old default of 1024 has been raised to 8192

• This may lead to various drastic performance issues

• Recommendation:

• Set it to the old default to avoid mutex concurrency and other issues:

_cursor_obsolete_threshold=1024

Oracle 19c | deferred_segment_creation

• MOS Note: 1216282.1All sorts of issues and bugs - from corruption to mutex contention to customer scripts not operating correctly anymore

• Recommendation:Set it todeferred_segment_creation=false

unless you have a valid reason

Oracle 19c | Adaptive Features

• Recommendation: Set optimizer_adaptive_statistics explicitly in your SPFILE

• See:

• https://mikedietrichde.com/2018/01/18/additional-info-adaptive-features-fixes-oracle-12-1-0-2/

• MOS Note: 2187449.1 - Recommendations for Adaptive Features in Oracle Database 12.1

Oracle 12.1 optimizer_adaptive_features = TRUE

Oracle 12.2

AdaptiveStatistics

optimizer_adaptive_statistics

Default: FALSE

AdaptivePlans

optimizer_adaptive_plans

Default: TRUE

Oracle 19c | _sql_plan_directive_mgmt_control

• MOS Note: 2209560.1 - How To Disable SQL Plan Directive (SPD)

• In order to fully disable SQL Plan Directives, you need to set:

• _sql_plan_directive_mgmt_control=0

• Otherwise the database collects SPDs in the background, but won't use it

• Having optimizer_adaptive_statistics=false - which is the default - disables only the usage

of SPDs but not their creation

Getting ready for the future

Upgrade / Migrate and Consolidate to Multitenantat La Mobilière, Switzerland

Customer Case | La Mobilière

• Swiss Mobiliar

• Switzerland's most personal insurer

• Founded 1826 in Bern, oldest Swiss insurance

• Legal form:

• Cooperative association (mutual company

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks

Customer Case | La Mobilière

Customer

Project

Constraints

Preparation

Upgrade

Success?

Remarks

Customer Case | La Mobilière

• Parallel upgrade catctl.pl unfolds its full power when upgrading many

PDBs at the same time

• 50 PDBs upgraded in less than 4 hours

• We feel very well prepared for the annual Oracle releases

• When we encounter issues, we fix them before going live

• Follow their projects on: https://mobiliar.ch/db-blog

• 100% Multitenant Consolidation reached in Oct 1, 2019

Customer

Project 2017

Constraints

Preparation

Upgrade

Success?

Remarks

Customer Case | La Mobilière

• High increase in number of PDBs, often driven by Microservices

• Self-developed DBaaS interface to privision, alter and drop PDBs

• Used to create >1000 PDBs

Customer

Project 2017

Constraints

Preparation

Upgrade

Success?

Remarks

Migrating to the Future

Exadata Migrationat La Mobilière, Switzerland

Upgrade NOW!

Upgrade to Oracle 19cat La Mobilière, Switzerland

Ph

oto

by

Ric

ard

o G

om

ez

An

ge

lon

Un

spla

sh

Customer

Project 2020

Constraints

Preparation

Upgrade

Success?

Remarks

Customer Case | La Mobilière

• Upgrade 2000 PDBs

• Oracle Database 12.2.0.1 to 19c

• Up to 50 PDBs per CDB in PROD

• Up to 150 PDBs per CDB in DEV/TEST

Customer

Project 2020

Constraints

Preparation

Upgrade

Success?

Remarks

Customer Case | La Mobilière

• CPU resources limited

• Solution: Sequential upgrades

• One DBA covers 1-2 CDB upgrades

• Once done, next DBA steps in

Customer

Project 2020

Constraints

Preparation

Upgrade

Success?

Remarks

Customer Case | La Mobilière

• Adopt AutoUpgrade

• Download newest version from MOS: 2485457.1

• Phase 1: 735 PDBs on a single weekend

• CDB1 144 PDBs• CDB2 148 PDBs• CDB3 148 PDBs• CDB4 147 PDBs• CDB5 148 PDBs

Customer

Project 2020

Constraints

Preparation

Upgrade

Success?

Remarks

* Logs of CDB5 are lost

Customer Case | La Mobilière

Upgrade timings *

Customer

Project 2020

Constraints

Preparation

Upgrade

Success?

Remarks

Customer Case | La Mobilière

• Upgrade timings

• Average: 6 - 7.5 hours

• Including recompilation

• Only 10 oCPUs used per CDB

Customer

Project 2020

Constraints

Preparation

Upgrade

Success?

Remarks

Customer Case | La Mobilière

• Post-fixups issue

• Solution: Use the newest OPatch

• Processes parameter

• Upgrade in PDBs spawns a lot of processes

• Solution: Increase it for the upgrade

• Recompile PDBs before upgrade

• ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl

-n 32 -e -d $ORACLE_HOME/rdbms/admin -l /tmp -b utlrp_log

utlrp.sql

• Read the full story here:

• https://mobiliardbblog.wordpress.com/2020/05/05/what-does-it-take-to-migrate-735-databases-to-19c-in-a-weekend-the-multitenant-architecture-and-great-dbas/

"We migrated 735 databases to 19c, and the task was mostly relatively relaxed. Start the AutoUpgrade tool and monitor the progress from time to time.

Sitting in front of the screen the whole time is not necessary."

Alain Fuhrer

Head IT Database Services

La Mobilière

Bern, Switzerland

Further Information

Finally …

Slides | https://MikeDietrichDE.com

HOL | https://MikeDietrichDE.com

Watch the entire seminar | https://MikeDietrichDE.com/videos

https://mikedietrichde.com/2020/07/06/upgrade-seminars-2020-recap-and-videos-on-demand/

Thank You!