Session: 226 Customer Experiences with 11gR2 Database · PDF fileSession: 226 Customer Experiences with 11gR2 Database on Linux on System z Time: Mon, Apr 23, 2012 (11:00 AM -12:00

Session: 226

Customer Experiences with 11gR2

Database on Linux on System z

Time: Mon, Apr 23, 2012

(11:00 AM - 12:00 PM)

Agenda

• Introduction

• Linux on System z Updates

• Migration & Upgrades

• Performance/Customer Experiences

Oracle Database 11g Release 2 on System z

� Oracle has delivered on Linux for System z since 2002 (9i, 10gR1,10gR2 & 11gR2),

before that on IBM z/OS – The zSeries Oracle SIG has been an independent user group since

1986.

� In 1Q 2011 Oracle delivered as scheduled 11g Release 2 (11.2.0.2) Database for

Linux on IBM System z Servers.

� In 4Q 2011 Oracle delivered as scheduled 11g Release 2 (11.2.0.3) Database for Linux on IBM

System z Servers LESS THAN 3 Months after the Initial release on x86.

� Oracle 11g Release 2,, has many new options including:

• Real Application Testing - the ability to capture live workloads and rerun

them on another system.

• Huge page memory support.

Enterprise Manager Grid Control

• Oracle Enterprise Manager – 10.2.0.5 Agent supports running Oracle 11gR2 databases.

• Oracle Support has published a release Date of Q4 Financial Year 2012, for the new

Enterprise manager 12c Grid Control Agent – which means between Now and May

2012 for Linux on System z.

Red Hat 6.1 Support

• Oracle announced, that Red Hat 6 will be supported (in general) within 90 days. http://www.oracle.com/us/corporate/press/1563775

• No Official Date for Red Hat System z has been published – but it is expected to be certified with Red Hat 6.1 for System z Linux.

• Oracle Support will take support calls for Red Hat 6.1 and an rpm checker is available.

Oracle Patch Set Update – Apr 17, 2012

• Released on the Same date as other Platforms!

• When applying this PSU, to either the RDBMS (13696216) or Grid Infrastructure (13696251) Homes, the relink phase might generate the following warnings:

ins_srvm.mk:68: warning: overriding commands for target `libsrvm11.so'

ins_srvm.mk:31: warning: ignoring old commands for target `libsrvm11.so'

ins_srvm.mk:71: warning: overriding commands for target `libsrvmocr11.so'

ins_srvm.mk:34: warning: ignoring old commands for target `libsrvmocr11.so'

ins_srvm.mk:74: warning: overriding commands for target `libsrvmhas11.so'

ins_srvm.mk:37: warning: ignoring old commands for target `libsrvmhas11.so'

*** This is a warning only which opatch is reporting. The Patch has applied successfully and the warning output can be safely ignored.

Compat-libstdc++-33: SLES 11 Installs (Safe to Ignore)

Compat-libstdc++-33 Error• libstdc++33 package replaces the compat-libstdc++ package

• rpm -q --provides libstdc++33.rpm

compat-libstdc++

libstdc++5 = 3.3.3-7.8.1

libstdc++.so.5()(64bit)

libstdc++.so.5(CXXABI_1.2)(64bit)

libstdc++.so.5(CXXABI_1.2.1)(64bit)

libstdc++.so.5(CXXABI_1.2.2)(64bit)

libstdc++.so.5(GLIBCPP_3.2)(64bit)

libstdc++.so.5(GLIBCPP_3.2.1)(64bit)





libstdc++.so.5(libstdc++.so.5)(64bit)

libstdc++33 = 3.3.3-7.8.1

• To complete the installation,click the Ignore Requirements box, then, hit the install/next button to finish the Oracle 11g R2 installation.

• Suse Link -> http://www.novell.com/support/dynamickc.do?cmd=show&forward=nonthreadedKC&docType=kc&externalId=7004995&sliceId=1

Oracle on System z Tips

• ORA-600[KFDADD03] WHEN CREATING A DISKGROUP USING FCP/SCSI STORAGE

– Bug 12346221 when creating ASM disk group

– See note for long term ASMLib direction - Oracle ASMLib Software Update

Policy for Red Hat Enterprise Linux Supported by Red Hat [ID 1089399.1] –

no plans for RH6

– Recommendation use UDEV rules opposed to ASMLib for DASD disks.

• When NLS_LANG and LANG values are set to different character set, DBCA can't

be launched.

– In Japan, DBCA related processes can't be terminated normally. Sending Ctrl-

C doesn't work and processes remained as zombie. – open SR

– No problems with French in Quebec or in Latin America – Mexico

– Recommendation -> unset any Oracle ENVIRONMENT variables when

installing per the release notes.

ECKD / DASD Disk Require’s a Partition

147a1d000: ffffffffffffffff ffffffffffffffff ................



147a1d030: ffffffffffffffff 00b38f0000000081 ................

147a1d040: 0000000205d5fdb0 00000000864d3a3c .............M:<

147a1d050: 0e575248245f5347 41535441545f5503 .WRH$_SGASTAT_U.

147a1d060: 535953015404504f 4f4c044e414d4504 SYS.T.POOL.NAME.

147a1d070: 4442494407534e41 505f49440f494e53 DBID.SNAP_ID.INS

147a1d080: 54414e43455f4e55 4d424552e5e5e5e5 TANCE_NUMBER....

147a1d090: e5e5e5e5e5e5e5e5 e5e5e5e5e5e5e5e5 ................

...

147a1dff0: e5e5e5e5e5e5e5e5 e5e5e5e5e5e5e5e5 ................

• ECKD/DASD Disks it’s mandatory to create at least one partition

• example below shows an Oracle session’s memory region being

• Linux was not expecting this and wrote “e5e5e5e5e5” to sector 0 (as Linux

assumes disks are partitioned)

• Moral of story: Always Partition DASD disk for LVM, ASM and swap

Oracle Automatic Memory – MEMORY_TARGET

• New memory parameter MEMORY_TARGET(AMM – Automatic Memory management)

• Combines SGA_TARGET and PGA_AGGREGATE_TARGET into one parameter.

• If you set MEMORY_TARGET you may see…(or just do a default Install)

ORA-00845: MEMORY_TARGET not supported on this system

Oracle alert log shows:

WARNING: You are trying to use the MEMORY_TARGET feature. This feature requires the /dev/shm file system to be mounted for at least 847249408 bytes.

• The error is really that the MEMORY_TARGET needs a larger /dev/shm file systemRun the following to resize tmpfs:# umount tmpfs

# mount -t tmpfs shmfs -o size=1300m /dev/shm

# df -k /dev/shm

Filesystem 1K-blocks Used Available Use% Mounted on

shmfs 1331200 0 1331200 0% /dev/shm *** make permanent in /etc/fstab file

Memory Sizing Oracle on System z Linux and 11gR2

• Customer attempted installing 11gR2 (Oracle’s latest release) with 512mb – could not re-link on install. Oracle recommends 4GB for all Linux Platforms, smallest we would recommend is 2GB of Virtual Memory for a Single Oracle instance.

• One customer experienced with 10gR2 to 11gR2 upgrade a 200mb

• Size the Virtual Memory based on What Oracle needs taking into account SGA, PGA, User Connections to the database (4.5mb), Linux Page Tables and Linux Kernel Memory. Right sizing is important.

• Production workloads 1 to 1.5:1 Virtual to Physical Memory, for Test and Dev 2 to 3:1 are possible.

Configuring Swap Space for Linux

• Example of VDISK for 1st and or 2nd Level Swap to

memory and 3rd priority to Disk

Multipathing for FCP/SCSI LUNsmultipath {

wwid 3600507630bffc2ce0000000000001112

alias lun40

path_grouping_policy failover

uid 501

gid 501

mode 660

}

• No longer require a disk partition for FCP/SCSI and 11gR2! OS Vendors recommend this as well. ECKD DASD Still requires a Partition.

• Required for Device Persistence (tied to WWID)

• Required for Oracle grid user file permissions

• Use the /dev/mapper/<alias name> as the ASM Diskstring

Linux UDEV Rules for Oracle

From Oracle we can then work with the new ASM Disk Device for example:

ALTER DISKGROUP DG2 add disk '/dev/ASM0305';

ALTER DISKGROUP DG2 rebalance power 2;

Oracle RAC Inter-Connect Considerations

• With System z you can use an Open System’s Adapter

Network Interface (OSA) e.g. eth1 or a Real Hipersocket

configured on Layer 2 for Oracle RAC Interconnect.

• Must Have Linux interface configured with ARP enabled

due to new Redundant Interconnect feature in 11.2+

• System z’s SAP processor’s assist with Network offload

of cpu utilization.

New in 11gR2 Oracle VKTM process (Virtual Time Keeper)

• VKTM is responsible for providing centralized time tracking• wall-clock time (updated every second)

• reference-time counter (updated every 20 ms) • When System is CPU Idle – vktm still runs.

• Non idle Linux Guest z/VM consistently stays in Q3 Status (which

means it will never swap/release it’s memory).

• If DB is stopped the database the Linux Guest goes to Q1 (or Q2)releasing memory. Restart DB, the machine stays in Q3.

• You can Disable tracing -> 11.2.0.3 + Oracle Note: 1381270.1To turn off VKRM tracing:alter system set events '10720 trace name context forever, level 0x10000000';To turn off VKTM tracing: alter system set events '10795 trace name context forever, level 2';

Linux strace of Oracle’s VKTM Process

$ ps -ef | grep vktm

oracle 6723 1 0 09:08 ? 00:00:00 ora_vktm_TEST11202

[root@orarac1 ~]# strace -p 6723

Process 6723 attached - interrupt to quit

gettimeofday({1323711237, 10495}, NULL) = 0


nanosleep({0, 10000000}, {1323711237, 10495}) = 0



nanosleep({0, 10000000}, {1323711237, 21947}) = 0

• Oracle does a lot of Linux gettimeofday calls (up to 100 per second) particularly with statistics_level set to 'typical' or 'all‘

• SuSe 11 SP1 and Red Hat 6.1(Perhaps Future 5.x) versions have reduced cpu consumption for gettimeofday()

Reference -> http://linuxmain.blogspot.com/2011/11/vdso-or-how-to-read-time-faster.html

One Customer’s 11.2.0.3 Experience with VKTM process:

• Oracle’s VKTM process still uses almost the same amount of CPU minutes

(about 0.08 vs. 0.09 with 11.2.0.2)

• However, we can see a great improvement with ora_dia0 process. (about

0.07 sec cpu/minute vs. 0.28 with 11.2.0.2)

• database 1:

installed with NO options

The "gettimeofday" function is called 300 times every 15 seconds.

database 2:

installed with all options : (java, xml, Text, spatial, APEX, etc ....... )

The "gettimeofday" function is called 1500 times every 15 seconds.

- RMAN Command -> CONFIGURE COMPRESSION ALGORITHM ‘Low‘

- Oracle Advanced Compression Feature required for Low, Medium, High

- Very High CPU observed with BZIP2

- Secure File LOBs can utilize this compression Technology

73.0 %

47.26 MB/s

357.03 MB00:28 (28 s)‘Low’

(LZO) Compression

77.6 %

19.46 MB/s

295.53 MB 01:08 (68 s)‘Medium’

(ZLIB) Compression

83.0 %

2.54 MB/s

224.82 MB08:41 (521 s)‘High’ 11gR2

(BZIP2) Compression

78.9 %

7.89 MB/s

278.95 MB02:48 (168 s)‘Basic’ 10gR2

(BZIP2) Compression

% Compression /

Input MB/s

Compression Size

Source DB - 1.29 GB

Backup TimeBackup Compression

Oracle RAC One Node - Omotion

z/VM

Linux

Guest

z/OS

shared storage

z/VM

# IFLsMemory

# OSA Card

# FC Cards

# IFLsMemory

# OSA Card

# FC Cards

# IFLsMemory

# OSA Card

# FC Cards

LPAR 1 LPAR 2 LPAR 3

z/VM

OCR and Voting Disks

ASM Disk Groups

Linux

Guest

Linux

Guest

Singl Inst 1 Singl Inst 3 Singl Inst 5Singl Inst 4

Oracle Clusterware

ASM

Singl Inst 2Singl Inst 2

• Patch Oracle binaries, modify Linux parameters, etc..

Source: Cloud Infrastructure – Deploying an elastic and heterogeneous application - Paul Bramy

RAC ONE – Transparent Application Failover - TAF

• Steps are in Oracle Support Note – 453293.1

• Key step that needs to be configured Post Install of Clusterware:

srvctl modify service -d test -s testone -P BASIC -e SELECT -z 180 -w 5 -m BASIC -j SHORT

*** Demo of Oracle RAC ONE with IBM Linux on System z ***

• In addition, z/VM Live Guest Migration moves a running guest from one z/VM system to another with at most a brief interruption to its service.

Migrations & Upgrades

Oracle 11gR2 – Changes in Mutex Locking11gR2 Experience -> If using cursor_sharing = “FORCE” or “SIMILAR”

1) ORA-600 errors as workload increases [kkspsc0: basehd]

or [kglLockOwnersListAppend-ovf] - applied patches to address

2) AWR showing -> cursor: mutex S and library cache lock

1. Download and apply the 11.2.0.2.3PSU Patch 11724916

2. Enable event 106001 to address Bug 10187168.

To enable the fix "_cursor_features_enabled" needs to be set

3) Oracle 11.2.0.2.2 PSU (Patch Set Update) includes new parameters that you can tweak based on workload characteristics. Even more fixes have been added

Note: 10411618 - Enhancement to add different "Mutex" wait schemes [ID 10411618.8]

4) 11.2.0.3 Has many Mutex enhancement’s

Migration Pre-requisite’s

• Oracle Note->Things to Consider Before

Upgrading to 11.2.0.3 Grid

Infrastructure/ASM [ID 1363369.1]

• In that note it states, that it's mandatory that

we have Grid Patch - 9952245 applied on top

of 10.2.0.5 before hand (so 10.2.0.5.2).

Oracle 10gR2 -> 11gR2 Upgrade

TIP => Safe to Ignore the OCR Integrity Check for Upgrades

Oracle 10gR2 – 11gR2 Upgrades

• Upgrades of the Database use – The DBUA (Database

Upgrade Assistance) – Easier than trying to

memorize/script the commands

• The Database Upgrade Assistant – will do all the system

checks and steps for the database upgrade.

Purge the Recycle Bin for 11gR2 Upgrade

• The recycle bin is enabled by default in 10g

• When the recycle bin is enabled, any tables that you

drop do not get deleted, Oracle renames the table and

its associated objects (indexes, triggers, LOBs, etc)

• If upgrading from 10g or 11g, purge the recycle bin

before you start the upgrade to help save time.

SQL> purge DBA_RECYCLEBIN;

Database Migration With Transportable

Database or Tablespace Options

• Transportable Database Methodology is the easiest if DB is same “endian binary format”, AIX, Solaris, HP-UX, Linux on System z are are all Big.

• Transportable tablespaces is another methodology which can be used when going from Little Endianto Big Endian.

Oracle Platform Migration Tips

• If Possible perform SAN Disk Replication between Source and

Target for the migration file system.

– Migration went from 60+ hours with NFS mount to 12

hours by unmounting the transportable file system,

performing SAN replication, and then remounting.

• Additional Performance Suggestions:

– Added more Network Cards to the source system.

– Parallelized as many threads as possible.

– Dynamically allocated more cpu for the Migration

Transportable Tablespace Suggestions

• Perform an Assessment of the source database.select * from nls_database_parameters where parameter like '%SET%';

NLS_NCHAR_CHARACTERSET AL16UTF16

NLS_CHARACTERSET AL32UTF8

**When you create your target database you will need to create with these values.

• Are the tables in the transportable set that use TIMESTAMP WITH TIMEZONE (TSTZ) columns?col name for a24col value$ for a10select name,value$ from props$ where name='DST_PRIMARY_TT_VERSION‘;

- Source DB :

NAME VALUE$

------------------------ ----------

DST_PRIMARY_TT_VERSION 14

Time Zone Update

Time Zone for Upgrades

• Most recent Time Zone definitions as of April 2012 is DST V17

• DSTv18 is targeted to incorporate the latest TZ update as of May

15th 2012 and to be available on June 15th 2012.

• Recommend updating the Time Zone before Upgrading as

DataPump Import can fail if TZ versions are different.

Performance Experiences with

Oracle on zEnterprise

Use HyperPAV, PAV or FCP/SCSI for Database files

Oracle Storage – Testing with ORION

ORION Simulates Oracle reads and writes, without having to create a database

./orion_zlinux -run oltp -testname mytest -num_disks 2 -duration 30 -simulate raid0

ORION VERSION 11.2.0.0.1

Commandline: -run oltp -testname mytest -num_disks 2 -duration 30 -simulate raid0

This maps to this test: Test: mytest

Small IO size: 8 KB Large IO size: 1024 KB

IO Types: Small Random IOs, Large Random IOs

Simulated Array Type: RAID 0 Stripe Depth: 1024 KB

Write: 0% Cache Size: Not Entered

Duration for each Data Point: 30 seconds

Small Columns:, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40

Large Columns:, 0 Total Data Points: 22

Name: /dev/dasdq1 Size: 2461679616

Name: /dev/dasdr1 Size: 2461679616

2 FILEs found.

Maximum Small IOPS=5035 @ Small=40 and Large=0

Minimum Small Latency=0.55 @ Small=2 and Large=0

Orion Testing Continued

-run oltp -testname mytest -num_disks 2 -duration 30 -simulate raid0

This maps to this test:

Test: mytest

Small IO size: 8 KB Large IO size: 1024 KB

IO Types: Small Random IOs, Large Random IOs

Simulated Array Type: RAID 0 Stripe Depth: 1024 KB

Write: 0%

Cache Size: Not Entered

Duration for each Data Point: 30 seconds

Small Columns:, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, Large Columns:, 0

Total Data Points: 22

Name: /dev/sda1 Size: 10737401856

Name: /dev/sdb1 Size: 10737401856

2 FILEs found.

Maximum Small IOPS=24945 @ Small=24 and Large=0

Minimum Small Latency=0.60 @ Small=12 and Large=0

For this test, the IOPS (Inserts per second) went from 5035 on ECKD to 24945 using FCP/SCSI storage.

Oracle HugePages – With Linux 4K Page Tables after 70 minutes

Linux Swap and PageTables using 87.7 GB of Memory!

74 ms response time and 0 TPS

1 MB Huge Pages 576 vs 4K Pages 485 TPS and 45ms vs 58 ms Response

(When running Good)

Two Hours later still Running strong…

Page tables: 0.371 GB and No Swap

Linux HugePage Considerations

• Can not use with Oracle Automatic Memory Management –

MEMORY_TARGET

• Not swappable: Huge Pages are not Linux swappable. There is no

page-in/page-out mechanism.

• General guideline consider when combined Oracle SGA’s are

greater than 8 GB (particularly if a lots of connections)

• Decreased page table overhead; more memory can be freed up for

other uses. For example Oracle SGA memory

Oracle Performance Suggestions:1) I/O scheduler on Red Hat - zipl.conf

parameters"elevator=noop" helps with reducing cpu usage.

2) Reduce Read - ahead for LVM file systems containing Oracle datafiles only.

lvchange -r none <lv device name>

lvdisplay /dev/oradb-vg/oradb-lv

3) Oracle parameter –

_fastpin_enable=1 will result in utilizing

"Consistent Gets from cache (fast path)”

4) Oracle Parameters that can reduce cpu 5-10%

statistics_level = 'basic'

timed_statistics=TRUE

**Can reduce cpu utilization by 5-10%, but effects data on AWR reports.

Oracle Real Application Testing:

Record Production Load on Test System and Replay on Another System:

Capture Considerations• Planning:

– Adequate disk space for captured workload (binary files) with nfs read-write

– Database restart is needed (care is needed)

• Startup restrict

– Capture will un-restrict

– If RAC Start on one node – other nodes down then bring other nodes up.

– A way to restore database for replay purposes:

• Physical restore (scn/time provided)

• Logical restore of application data

• Flashback/snapshot-standby

– Filters can be specified to capture subset of workload.

• Overhead:

– Performance overhead ~ 4.5%

– Memory overhead : 64 KB per session

– Disk space

Oracle Real Application Testing - Replay

Workload Wizard

Customer Experiences with Oracle

on zEnterprise

Customer Benchmark Test:

Source: At the Fun House – Brian Gormanly Marist College

Oracle RAC Performance Benchmark – z10 vs z196

• Source: http://public.dhe.ibm.com/software/dw/linux390/perf/ZSW03185-USEN-02.PDF Dr. Juergen Doelle, Margaret Phillips

Questions

The following are trademarks of the International Business Machines Corporation in the United States and/or other countries.

The following are trademarks or registered trademarks of other companies.* Registered trademarks of IBM Corporation

* All other products may be trademarks or registered trademarks of their respective companies.

Intel is a trademark of Intel Corporation in the United States, other countries, or both.

Java and all Java-related trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countries

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation.

Oracle is a trademark of Oracle Corporation in the United States, other countries, or both.

Red Hat, the Red Hat "Shadow Man" logo, and all Red Hat-based trademarks and logos are trademarks or registered trademarks of Red Hat, Inc., in the United States and other countries.

SET and Secure Electronic Transaction are trademarks owned by SET Secure Electronic Transaction LLC.

Notes:

Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here.

IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.

All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.

This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.

All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

AIX*

APPN*

CICS*

DB2*

DB2 Connect

DirMaint

DRDA*

Distributed Relational Database Architecture

e-business logo*

ECKD

Enterprise Storage Server*

ESCON*

FICON*

GDPS*

Geographically Dispersed Parallel Sysplex

HiperSockets

HyperSwap

IBM*

IBM eServer

IBM e(logo)server*

IBM logo*

IMS

InfoPrint*

Language Environment*

MQSeries*

Multiprise*

NetView*

On demand business logo

OS/390*

Parallel Sysplex*

PR/SM

Processor Resource/Systems Manager

RACF*

Resource Link

RMF

S/390*

Sysplex Timer*

System z

System z9

TotalStorage*

Virtualization Engine

VSE/ESA

VSE/ESA

VTAM*

WebSphere*

z/Architecture

z/OS*

z/VM*

z/VSE

zSeries*

Documents

Session: 226 Customer Experiences with 11gR2 Database · PDF fileSession: 226 Customer Experiences with 11gR2 Database on Linux on System z Time: Mon, Apr 23, 2012 (11:00 AM -12:00