Utah Oracle User Group Tech Day

March 2017Charles Kim, Oracle ACE Director, VMware vExpert

• Oracle ACE Director

• Founder and President of Viscosity

• Over 25 years of Oracle Expertise: Mission Critical Databases,

RAC, Data Guard, ASM, RMAN, Attunity/GoldenGate

• Specializing in “Automation of Everything” done more than once!!

• President - IOUG Cloud Computing SIG

• Blog Sites: http://DBAExpert.com/blog

• Oracle Exadata Certified Implementation Specialist, (2014, 2016)

• Certified RAC Expert

Charles Kim @racdba

3

Audience Poll

• Get a show of hands of people running RH 6 or 7, OEL 6 or 7, SUSE?

• Get a show hands of people running RAC on Linux?

• Show of people running 11g R2 or 12cR1• Show of hands of who are running VM for databases

…VMWARE? OVM? RAC on VM?

• Show of hands who are DBAs? SAs? Developers?

• Show of hands who are running VM on laptops or personal desktops?

4

Where can we download Linux?OEL:http://edelivery.oracle.com/linux

30 day evaluation from Red Hat:https://www.redhat.com/rhel/details/eval/

Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2) [ID 169706.1]

Benefits of OEL?Oracle validated install

Single Vendor (no one to point blame to)Comprehensive Support

Evolution of the DBA

5

Kind of DBA Timeline

CLI DBA Early 90’s DBAs

GUI DBA Late 90’s and Dot Com

Google DBA Dot Com and 2000’s

iDBA Dot Com, IOUG iDBA Master Curriculum

RAC DBAs 2000+ after 9.2 (but major spike with 10.2)

DMA 2010+ Database Machine Administrator

vDBA / vRAC DBA 2010+ Evolving role of a DBA in the virtual world

Cloud DBA 2011+ Database Consolidation with Private Database CloudOracle Database 12c Launches June 2013

Public Cloud DBA 2015+ Oracle Public Cloud with Database Cloud Service, Database Backup Cloud Service, Storage Cloud Service, IaaS Cloud Service

PDBAs 2017+ Multi-Tenant with Oracle Database 12c Release 2GA – March 15, 2017

“It is not the strongest or the most intelligent who will survive but those who can best

manage [adapt to] change.”

My Speaking Events 2017

March 13-14 Utah Oracle Users Group Training Days Salt Lake City, UT

March 15-16 Cloud Experience @GaOUG Tech Day Atlanta GA

March 28 IT Transformation with Oracle 12.2 Houston, TX

April TBD Sun Coast OUG Tech Day - Cloud Experience Tampa, FL

April 18 Data Transformation with Oracle 12.2 Dallas

April 2-6 IOUG Collaborate17 at Mandalay Bay Las Vegas, NV

May 17-18 Great Lakes OUG Conference Cleveland, OH

June 25-29 ODTUG Kscope17 @ JW Marriot San Antonio, TX

Nov 8 Michigan Oracle User Group Detroit, MI

Collaborate – Independent Oracle User GroupCloud Experience @IOUG 2017

A boot camp design by Oracle User Community Technology Experts for Oracle Technology

Professionals to experience uses of the Oracle Public Cloud and Oracle Technology on-premise to accomplish work critical for their businesses

and careers.

IOUG Cloud SIGioug.org/cloud@cloud_sig

OPERATING SYSTEMCHOICES

A Word on Oracle UEK

It’s your ace in the hole. If you run UEK, you get much better support form Oracle J

UEK has loads of benefits for database workloads (even on virtualized environments)•Improved memory management•Better IO scheduling

Formerly Known as Oracle-Validated on Red Hat

10

wgetwget

# yum install oracle-ebs-server-R12-preinstall

EBS ISOhttp://dbaexpert.com/ebs/

OS Template - Red Hat 6.4+ ASMLIB SupportAs of May 2013ASMLib requires kmod-oracleasm (RHEL Supplementary Channel), oracleasmlib and

oracleasm-support packages

Convert from udev to ASMLIB – http://dbaexpert.com/blog/

11

12

VM Template - ssh setup – The Alternative Way

http://www.dbaexpert.com/blog/ssh-setup-the-oracle-way/http://www.dbaexpert.com/blog/sshsetup-on-a-7-node-rac/

$ pwd/nfs/software/12c/grid/sshsetup

$ ./sshUserSetup.sh -user oracle -hosts "rac1 rac2" \-noPromptPassphrase -advanced –exverify

The output of this script is also logged into /tmp/sshUserSetup_2013-07-09-16-31-50.log

Space

13

VM Template - OS Watcher – The Alternative Way

Install OS Watcher Service RPM:sudo rpm -ihv oswbb-service-1.1.5-1.noarch.rpm

Modify /etc/oswbb.conf# Set OSW_HOME to the directory where you unpacked OSW or OSWbbaOSW_HOME='/u01/app/oracle/oswbb'# Set OSW_INTERVAL to the number of seconds between collectionsOSW_INTERVAL='30'# Set OSW_RETENTION to the number of hours logs are to be retainedOSW_RETENTION='168'# Set OSW_USER to the owner of the OSW_HOME directoryOSW_USER='root'# Set OSW_COMPRESSION to the desired compression facilityOSW_COMPRESSION='gzip’

http://www.dbaexpert.com/blog/the-better-way-to-configure-oswatcher/How To Start OSWatcher Black Box (OSWBB) Every System Boot [ID 580513.1]

Start OSWatcher:sudo /sbin/service oswbb start

Setup Auto-Start:sudo /sbin/chkconfig oswbb on

14

AGENDA

LINUX TUNING>TOP 10 MISTAKES FOUND IN ORACLE SYSTEMS

State of the Penguin

15http://www.cisco.com/go/semreg/es_viscosityoracleevent/pdf

Top Ten Mistakes Found in Oracle Systems

1. Bad connection management2. Bad use of cursors and shared pool3. Bad SQL4. Use of nonStandardized initialization Parameters5. Getting database I/O wrong6. Online redo log setup problems7. Serialization of data blocks in the buffer cache due to lack of free

lists, free list groups, INITRANS, and shortage of RBS8. Long full table scans9. High amounts of recursive (SYS) SQL10. Deployment and migration errors

16

My Top 10 Mistakes in Oracle Systems1. Bad SQL2. Getting database I/O wrong

• Improper ASM Configuration• Wrong Storage Configuration

3. Online redo log setup problems4. Long full table scans5. Scalability to RAC

• Non-RAC on AIX/Solaris/HP-UX to RAC on Linux cannot scale

• Serialization of data blocks in the buffer cache due to lack of free lists, free list groups, INITRANS, and shortage of RBS

17

6. Incorrect Linux Configuration7. Bad Statistics (Histograms)8. Too many indexes, wrong index usage9. Over-commitment of CPU in a

consolidated environment10. No Partitions when partitions are

warranted• Lack of Oracle Features due to $$

18

AGENDA

LINUX TUNING>STORAGE

Partition Alignment

If you choose to create separate disk groups for OCR and Vote disk, create a partition alignment of 1M and 4M offset

# --# -- Partition alignment of OCR / Vote Disks with 1MB offsetecho "2048,," | sfdisk -uS /dev/mapper/ov_disk000p1echo "2048,," | sfdisk -uS /dev/mapper/ov_disk001p1echo "2048,," | sfdisk -uS /dev/mapper/ov_disk002p1

# --# -- Partition alignment of Data / FRA disks with 4MB offsetecho "8192,," | sfdisk -uS /dev/mapper/data101_disk000p1echo "8192,," | sfdisk -uS /dev/mapper/data101_disk001p1echo "8192,," | sfdisk -uS /dev/mapper/data101_disk002p1echo "8192,," | sfdisk -uS /dev/mapper/data101_disk003p1echo "8192,," | sfdisk -uS /dev/mapper/data101_disk004p1

sfdisk# echo "2048,," | sfdisk -uS /dev/emcpowerbChecking that no-one is using this disk right now ...OKDisk /dev/emcpowerb: 1018 cylinders, 166 heads, 62 sectors/tracksfdisk: ERROR: sector 0 does not have an msdos signature/dev/emcpowerb: unrecognized partition table typeOld situation:No partitions foundNew situation:Units = sectors of 512 bytes, counting from 0

Device Boot Start End #sectors Id System/dev/emcpowerb1 2048 10477255 10475208 83 Linux/dev/emcpowerb2 0 - 0 0 Empty/dev/emcpowerb3 0 - 0 0 Empty/dev/emcpowerb4 0 - 0 0 EmptyWarning: no primary partition is marked bootable (active)This does not matter for LILO, but the DOS MBR will not boot this disk.Successfully wrote the new partition tableRe-reading the partition table ...If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1

(See fdisk(8).)

20

Verify Partition Alignment

# sfdisk -uS -l /dev/emcpowerb

Disk /dev/emcpowerb: 1018 cylinders, 166 heads, 62 sectors/trackUnits = sectors of 512 bytes, counting from 0

Device Boot Start End #sectors Id System/dev/emcpowerb1 2048 10477255 10475208 83 Linux/dev/emcpowerb2 0 - 0 0 Empty/dev/emcpowerb3 0 - 0 0 Empty/dev/emcpowerb4 0 - 0 0 Empty

Best Practice – Determine IOPS and Throughput

ORION (Part of 11.2 now)sudo -u root ./orion_linux_x86-64 -run advanced –testname myload -num_disks 20 -cache_size 8000 -duration 240 -matrix basic

SLOB2 (Simple Little Oracle Benchmark)

Calibrate I/O – Native to Oracle starting in 11.1SQL> declare2 l_latency integer;3 l_iops integer; 4 l_mbps integer; 5 begin 6 dbms_resource_manager.calibrate_io7 (5,10,l_iops,l_mbps,l_latency);8 dbms_output.put_line ('max_iops = '||l_iops);9 dbms_output.put_line (’latency = '||l_latency);10 dbms_output.put_line ('max_mbps = '||l_mbps);11 end;12 /max_iops = 5348latency = 10max_mbps = 641

Other Free Tools:• Swingbench

• TPC Benchmark • Custom scripts

How do you know for sure?RAT - Database Replay

tuned-adm

23

# yum install tuned# chkconfig tuned on# service tuned start

# tuned-adm profile enterprise-storageSwitching to profile 'enterprise-storage'Applying deadline elevator: dm-0 dm-1 dm-2 sda sdb sdc sdd [ OK ]sdg sdh Applying ktune sysctl settings:/etc/ktune.d/tunedadm.conf: [ OK ]Calling '/etc/ktune.d/tunedadm.sh start': [ OK ]Applying sysctl settings from /etc/sysctl.confStarting tuned: [ OK ]

# tuned-adm list |grep ^- |wc12 24 219

tuned-adm

24

Tuneables Default (CFQ) Enterprise-Storage (Deadline)

CPU Governer OnDemand Performance

File-System barrier on off

vm.dirty_ratio 20% 40%

Disk read-ahead 1x 4x# blockdev --getra /dev/sdh1256

# blockdev --getra /dev/sdh11024

tuned-adm

25

# yum install tuned# chkconfig tuned on# service tuned start

# tuned-adm profile enterprise-storageSwitching to profile 'enterprise-storage'Applying deadline elevator: dm-0 dm-1 dm-2 sda sdb sdc sdd [ OK ]sdg sdh Applying ktune sysctl settings:/etc/ktune.d/tunedadm.conf: [ OK ]Calling '/etc/ktune.d/tunedadm.sh start': [ OK ]Applying sysctl settings from /etc/sysctl.confStarting tuned: [ OK ]

# tuned-adm list |grep ^- |wc12 24 219

IO schedulersDeadline CFQ (Completely Fair Queueing) NOOP

• Default except for SATA devices

• Attempts to provide guaranteed latency for requests from the point at which requests reach the I/O scheduler

• Suitable for most use cases especially where read > write operations

• Default scheduler for SATA disks

• Divides processes into three separate classes: real time, best effort, and idle

• Real time class performed before best effort which is performed before idle class

• Real time class can starve both best effort and idle processes of processor time

• Default is best effort class

• Implements a simple FIFO (first-in first-out) scheduling algorithm

• Requests merged at the generic block layer through a simple last-hit cache

• Best scheduler for CPU-bound systems using fast storage

Recommendation for Virtualized Infrastructure:• Set GOS disk scheduling algorithm to ‘noop’• ‘noop’ (No Op) scheduler lets ESXi perform I/O optimization & queueing

I/O Schedulers# echo deadline > /sys/block/${ASM_DISK}/queue/scheduler

Default in OL6 UEK:# cat /sys/block/sda/queue/schedulernoop [deadline] cfq

• Completely Fair Queuing (CFQ) scheduler is the default algorithm in Red Hat Enterprise Linux 4, 5, 6

Permanent Changes to /etc/grub.conftitle Red Hat Enterprise Linux Server (2.6.9-67.EL)

root (hd0,0) kernel /vmlinuz-2.6.9-67.EL ro root=/dev/vg0/lv0 elevator=deadlineinitrd /initrd-2.6.9-67.EL.img

27

Kernel Parameter that affect Database Performance – Virtual Memory and I/O

dirty_background_ratio % of system memory that can be filled with dirty pages

dirty_expire_centisecs How long a dirty page should sit in RAM prior to being written to diskIf dirty pages are older than value, written to disk asynchRecommended Value: 500 à 5 Seconds

vm.dirty_ratio Maximum amount of system memory that can be filled with dirty pages before committed to disk

dirty_writeback_centisecs How often the flush process wakes up to determine need to flush any dirty pages

vm.swappiness Default Value: 60 in RHEL 6

fs.aio-max-nr Sets the maximum number of concurrent asynchronous I/O requestsOracle Recommended Value: 1048576fs.aio-max-nr = 1048576

fs.file-max Maximum number of open file handles assigned to the OSCalculated by multiplying ( 512* PROCESSSES ) + in addition to the open file handles by the OSOracle Recommended Value: Not <= fs.file-max = 6815744Need to revisit this value as we increase databases and processes parameters28

29

AGENDA

LINUX TUNING>KERNEL

30

OEL Validated Kernel Parameters - /etc/sysctl.confIncluding comments

# For 11g, recommended value for ip_local_port_range is 9000 65500net.ipv4.ip_local_port_range = 9000 65500# For 10g, uncomment 'net.ipv4.ip_local_port_range 1024 65000'# net.ipv4.ip_local_port_range:1024 65000# Added min_free_kbytes 50MB to avoid OOM killer on EL4/EL5vm.min_free_kbytes = 51200# For 11g, recommended value for net.core.rmem_max is 4194304net.core.rmem_max = 4194304# For 10g, uncomment 'net.core.rmem_max 2097152’

# For 11g, recommended value for wmem_max is 1048576net.core.wmem_max = 1048576# For 10g, uncomment 'net.core.wmem_max 262144'# net.core.wmem_max:262144

# Controls the maximum number of shared memory segments, in pageskernel.shmall = 1073741824# For 11g, recommended value for file-max is 6815744fs.file-max = 6815744# For 10g, uncomment 'fs.file-max 327679'# fs.file-max:327679kernel.msgmni = 2878kernel.sem = 250 32000 100 142kernel.shmmni = 4096net.core.rmem_default = 262144# net.core.rmem_max=2097152

Dynamically Load Kernel Parameters # sysctl –p

List Kernel Parameters# sysctl -a

Tip #1:Cheat from Oracle’s

validated installations

Tip #2: Starting from Oracle Linux 6:

31

OEL Validated Kernel Parameters - /etc/sysctl.confIncluding comments

# oracle-rdbms-server-12cR1-preinstall setting for net.ipv4.ip_local_port_range is 9000 65500net.ipv4.ip_local_port_range = 9000 65500

# Recommended value for kernel.panic_on_oopskernel.panic_on_oops = 1 # Value of 1 will reboot; value of 0 will try to continue the operation……# Controls the maximum number of shared memory segments, in pageskernel.shmall = 4294967296

# oracle-rdbms-server-12cR1-preinstall setting for fs.file-max is 6815744fs.file-max = 6815744

# oracle-rdbms-server-12cR1-preinstall setting for kernel.sem is '250 32000 100 128'kernel.sem = 250 32000 100 128

# oracle-rdbms-server-12cR1-preinstall setting for kernel.shmmni is 4096kernel.shmmni = 4096

# oracle-rdbms-server-12cR1-preinstall setting for kernel.shmall is 1073741824 on x86_64

# oracle-rdbms-server-12cR1-preinstall setting for kernel.shmmax is 4398046511104 on x86_64kernel.shmmax = 4398046511104

# oracle-rdbms-server-12cR1-preinstall setting for net.core.rmem_default is 262144net.core.rmem_default = 262144

# oracle-rdbms-server-12cR1-preinstall setting for net.core.rmem_max is 4194304net.core.rmem_max = 4194304

Kernel Parameter that affect Oracle Database Performance - Memory

sem Comprised of 4 parameterskernel.sem = 250 32000 100 128

shmall Maximum total amount of shared memory pages; # getconf PAGE_SIZE4096Example: Physical RAM 128GB of Memory:128 * 1024*1024*1024 / 4096 =33554432kernel.shmall = 4294967296

shmmax maximum size in bytes of a single shared memory segmentGood Starting Place ➔ ½ of Physical RAM 128GB of Memory: 128/2 * 1024*1024*1024 = 68719476736

shmmni Maximum number of semaphore sets for the entire system ➔ Oracle Recommended Value: 4096 kernel.shmmni = 4096

vm.swappiness 100 to force aggressive swapping;

32

Kernel Parameter that affect Oracle Database Performance – Network

net.core.mem_default Default OS receive buffer sizenet.core.rmem_default = 262144

net.core.rmem_max Max OS receive buffer sizenet.core.rmem_max = 4194304

net.core.wmem_default Default OS send buffer sizenet.core.wmem_default = 262144

net.core.wmem_max Max OS send buffer sizenet.core.wmem_max = 1048576

net.ipv4.ip_local_port_range net.ipv4.ip_local_port_range = 9000 65500

33

34

AGENDA

LINUX TUNING>NETWORK

Enable Jumbo Frames – Check to see

Will Suceedping -M do -s 8972 -c 2 rac01a-privping -M do -s 8972 -c 2 rac01b-privping -M do -s 8972 -c 2 rac02a-privping -M do -s 8972 -c 2 rac02b-privPING rac01a (10.17.33.31) 8972(9000) bytes of data.8980 bytes from rac01a-priv (10.17.33.31): icmp_seq=1 ttl=64 time=0.017 ms8980 bytes from rac01a-priv (10.17.33.31): icmp_seq=2 ttl=64 time=0.018 ms

Will Failping -M do -s 8973 -c 2 rac01a-privping -M do -s 8973 -c 2 rac01b-privping -M do -s 8973 -c 2 rac02a-privping -M do -s 8973 -c 2 rac02b-priv

Make sure: switch support is enabled

Must have end-to-end support

Enable Jumbo Frames# cat /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0IPADDR=192.168.89.90NETMASK=255.255.255.0ONBOOT=yesBOOTPROTO=noneUSERCTL=noVLAN=yes

MTU=9000

36

37

Agenda – CPU and Memory Management

Another look at top

z – color mode / no color modeShift M - list processes by Memory UsageShift P – list processes by CPU usageShift W – to save top preferences to a file - /home/oracle/.toprcShift H – to display threads for processes

#1 – list the CPU threadsc – provides a full command line syntaxd – change refresh rate from 3 seconds (default) to another value* Change the value to like 1000 and manually refresh by hitting the return key

k – to kill a process ID

38

# freetotal used free shared buffers cached

Mem: 3922696 3574024 348672 0 56460 2067188-/+ buffers/cache: 1450376 2472320Swap: 4063224 8236 405498

Best Practices - Hugepages

Not /etc/security/limits.conf to set soft and hard limitsInstead, /etc/security/limits.d/99-oracle-limits.conf

oracle soft nofile 131072oracle hard nofile 131072

oracle soft nproc 131072oracle hard nproc 131072oracle soft core unlimitedoracle hard core unlimited

# -- The following entries need to adjusted with HugePages settings# oracle soft memlock [Sum SGA + wiggle]# oracle hard memlock [Sum SGA + wiggle]

oracle soft memlock 50331648oracle hard memlock 50331648

$ ulimit –l50331648 /etc/sysctl.conf

vm.nr_hugepages = 24576

HugepagesSQL> alter system set memory_target=0;System altered. SQL> alter system set memory_max_target=0;System altered.

USE_LARGE_PAGES=only [only|true|false] ..true is default [11.2.0.2+]SGA_TARGET=47GSGA_MAX_SIZE=47GMEMORY_MAX_TARGET=0MEMORY_TARGET=0

hugepages_settings.sh from MOS ID: 401749.1

# /etc/sysctl.confvm.nr_hugepages = 22960 # number of pages specified in 2M increments

# grep ^Huge /proc/meminfo à HugePages_Rsvd: number of hugepages that are in use.

40

Top CPU Consumers[oracle@rac01 ~]$ ps auxww --sort=lstart | sort -r -k3,4 | headUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDoracle 1718 6.0 2.0 1542388 82132 ? Ss 20:08 0:42 ora_lmd0_VPROD1oracle 1728 5.0 1.9 1528292 76292 ? Ss 20:08 0:36 ora_lck1_VPROD1oracle 6323 3.8 0.1 1093736 5376 ? Ss 18:13 4:49 mdb_vktm_-MGMTDBoracle 17073 3.6 0.1 1437804 5724 ? Ss 17:42 5:42 asm_vktm_+ASM1oracle 1694 3.2 0.5 1526500 20224 ? Ss 20:08 0:22 ora_vktm_VPROD1oracle 2296 29.6 12.9 1613132 508324 ? Rs 20:09 3:11 oracleVPROD1

(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))oracle 5469 27.8 4.3 1351840 170916 ? Sl 20:19 0:04

/u01/app/12.1.0/grid_1/jdk/bin/java -server -Xms128M -Xmx384M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Dstdstream.filesize=100 -Dstdstream.filenumber=10 -DTRACING.ENABLED=false -Doracle.wlm.dbwlmlogger.logging.level=INFO -jar /u01/app/12.1.0/grid_1/oc4j/j2ee/home/oc4j.jar -config /u01/app/12.1.0/grid_1/oc4j/j2ee/home/OC4J_DBWLM_config/server.xml -properties -ports default-web-site:http:8888,rmi:23792 -out /u01/app/12.1.0/grid_1/oc4j/j2ee/home/log/oc4j.out -err /u01/app/12.1.0/grid_1/oc4j/j2ee/home/log/oc4j.err

root 17315 2.7 1.2 815024 47168 ? Ssl 17:42 4:21 /u01/app/12.1.0/grid_1/bin/crsd.bin reboot

oracle 1720 2.5 2.8 1542244 110240 ? Ss 20:08 0:18 ora_lms0_VPROD1

41

Top Memory Consumer[oracle@rac01 ~]$ ps aux --sort -rss | headUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDoracle 2296 29.3 13.7 1642416 539020 ? Rs 20:09 3:18 oracleVPROD1

(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))oracle 1700 0.0 9.9 1526500 389840 ? Ss 20:08 0:00 ora_mman_VPROD1root 21244 0.6 7.6 1455980 299148 ? Sl 17:31 1:07 /u01/app/12.1.0/grid_1/jdk/jre/bin/java -Xms128m -

Xmx512m -classpath /u01/app/12.1.0/grid_1/tfa/rac01/tfa_home/jlib/RATFA.jar:/u01/app/12.1.0/grid_1/tfa/rac01/tfa_home/jlib/je-5.0.84.jar:/u01/app/12.1.0/grid_1/tfa/rac01/tfa_home/jlib/ojdbc6.jar:/u01/app/12.1.0/grid_1/tfa/rac01/tfa_home/jlib/commons-io-2.2.jar oracle.rat.tfa.TFAMain /u01/app/12.1.0/grid_1/tfa/rac01/tfa_home

oracle 5610 20.5 5.2 1610004 207260 ? Rs 20:20 0:05 ora_m000_VPROD1oracle 16191 1.8 4.9 725260 193612 ? SLsl 17:41 2:53 /u01/app/12.1.0/grid_1/bin/ocssd.bin oracle 6329 0.0 4.7 1093736 185216 ? Ss 18:13 0:00 mdb_mman_-MGMTDBroot 16359 1.4 4.7 755128 184604 ? S<Lsl 17:41 2:22 /u01/app/12.1.0/grid_1/bin/ologgerd -M -d

/u01/app/12.1.0/grid_1/crf/db/rac01oracle 12017 0.8 4.6 1197448 180476 pts/1 Sl+ 19:16 0:32

/u01/app/oracle/product/12.1.0/dbhome_2/jdk/jre/bin/java -Doracle.installer.not_bootstrap=true -DCV_HOME=/u01/app/oracle/product/12.1.0/dbhome_2 -DORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_2 -Dsun.java2d.font.DisableAlgorithmicStyles=true -DSET_LAF= -Dice.pilots.html4.ignoreNonGenericFonts=true -mx128m -XX:MaxPermSize=128M -classpath /u01/app/oracle/product/12.1.0/dbhome_2/assistants/dbua/jlib/dbma.jar:/u01/app/oracle/product/12.1.0/dbhome_2/jlib/ewt3.jar:/u01/app/oracle/product/12.1.0/dbhome_2/jlib/ewtcompat-3_3_15.jar:/u01/app/oracle/product/12.1.0/dbhome_2/jlib/swingall-1_1_1.jar:/u01/app/oracle/product/12.1.0/dbhome_2/jlib/share.jar:/u01/app/oracle/product/12.1.0/dbhome_2/jlib/orai18n.jar:/u01/app/oracle/product/12.1.0/dbhome_2/assistants/jlib/assistantsCommon.jar:/u01/app/oracle/product/12.1.0/dbhome_2/assistants/jlib/rconfig.jar:/u01/app/oracle/product/12.1.0/dbhome_2/assistants/netca/jlib/netca.jar:/u01/app/oracle/product/12.1.0/dbhome_2/jlib/help4.jar:/u01/app/oracle/product/12.1.0/dbhome_2/oui/jlib/jewt4.jar:/u01/app/oracle/product/12.1.0/dbhome_2/jlib/oracle_ice.jar:/u01/app/oracle/product/12.1.0/dbh

42

Our Favorite Performance Monitoring Toolstop - Displays real-time view of a running system$ top –u oracle$ top -u oracle -n 2 -d 3

tcpdump – Network Packet AnalyzerDump traffic on a network# tcpdump –i eth0# tcpdump –i eth0 -n target_host

netstat – Monitor incoming and outgoing network and packet statistics.

# netstat –a |more# netstat -nlpt : Get process name/pid and user id# netstat -rn: Display kernel routing information# netstat -i : Print network interfaces# netstat -atnp | grep ESTA : Print active connections

iotop –Simple top like I/O monitor for I/O & Processes# iotop –u oracle

nmon$ nmon -ft -s 30 -c 120 : Leverage nmon Analyzer

free - Display amount of free and used memory in the system$ free –m or $ free -g : Memory free in MB or GB

iostat - Report CPU and I/O statistics # iostat –d: displays only disks I/O statistics of all partitions # iostat –c: displays only CPU statistics# iostat –t 2 5: iostat ‘X’ seconds and (‘N’number of times)

with timestamp

vmstat: - Report virtual memory statistics#vmstat –t 2 5 :vmstat ‘X’ seconds and (‘N’number of times)

with timestamp#vmstat –s: displays event counters and memory statistics

sar - System Activity Reporter# sar -u 2 10 : Report CPU utilization for each 2 seconds x 10

lsof: Display the list of all opened files lsof -i TCP:22

Others: dtrace, strace, perf, collectl, wireshark, tethereal43

44

AGENDA

LINUX TUNING>SUMMARY

oratop - Utility for Near Real-time Monitoring of Databases, RAC and Single Instance

(Doc ID 1500864.1)

DEMO Time

47

AGENDA

LINUX TUNING>CLEANUP / MAINTENANCE

48

Find Tips That Will Save You Time

Removing files older than 14 Days find . -type f -mtime +14 -exec rm -f {} \;

Finding the top 5 largest files find . -ls | sort -nrk 7 | head -5

Find files larger than 100MB find . -size +100000k

Delete audit records that’s older than 30 days find $ORACLE_HOME/rdbms/audit -name "*.aud" -mtime +30 -exec rm {} \;

Delete files in /tmp that’s older than 30 days find /tmp -group dba -type f -mtime +5 -exec rm -f {} \;find /tmp/dba -group dba -type f -mtime +5 -exec rm -f {} \;

Delete *.trc files more than 5 days old. find $TRACE_DIR -name '*.trc' -type f -mtime+5 -exec rm {} \;

49

More Commands to Locate Large Files

Display top 5 largest files in a current directory

ls -alS | head -5

Largest space-consuming directories(Does not include sub-directories)

du -Sm . | sort -nr | head -5

report the sum of space consumed by a directory and its subdirectories

du . | sort -nr | head -5

export DB_SIZE=$(find . -newer $RMAN_BACKUP_MARKER_FILE -name '*.DB' |xargs du -sk |awk '{SUM += $1} END {print SUM/1024}')export ARCH_SIZE=$(find . -newer $RMAN_BACKUP_MARKER_FILE -name '*.A' |xargs du -sk |awk '{SUM += $1} END {print SUM/1024}')echo "DB Backup Size is: $DB_SIZE"echo "Archive Backup Size is: $ARCH_SIZE

List files in a directory greater than 300M ls -l | awk '{if ($5 >314572800) print $0}' |sort +4

50

tar & scp One-Liners to rememberTar and un-tar on the same machine

Make a clone of ASM Oracle Home to DB Oracle Home

cd /u01/app/oracle/product/12.2.0/asmtar cvf - * | ( cd /u01/app/oracle/product/12.2.0/db; tar xvfp - )

Tar and un-tar to a remote server

Push the ASM Home Directory from Server#1 to the DB Home on Server #2

tar cvf - DIR_NAME |ssh REMOTE_HOST “cd DIR_NAME; tar xvf – “

Create compressed tarfile tar –cvzf orahome.tar/u01/app/oracle/product/12.2/dbOr for non GNU tar:tar -cvf - /u01/app/oracle/product/12.2/db | gzip>orahome.tar.gz

scp pull of the dgmenu directory scp -rp oracle@rac5501:/home/oracle/work/dgmenu .

scp push the dgmenu directory scp -rp dgmenu oracle@rac5502:/home/oracle/work

51

Rotate Listener Logs with Oracle CLI

Rotate Listener Logs in Oracle Database 12c

#!/bin/kshexport LISTENER_NAME=$1

. $HOME/.profile.CRONexport DATETIME=$(date +%y%m%d-%H%M)

cd $ORACLE_BASE/diag/tnslsnr/`hostname`/${LISTENER_NAME}/trace

lsnrctl << EOFset current_listener ${LISTENER_NAME}set log_status offmv ${LISTENER_NAME}.log ${LISTENER_NAME}.${DATETIME}.logset log_status onEOF

52

Rotate Listener Logs with Linux logrotate

Other use cases:1. Apache httpd.log, access_log, error_log• JVM logs, etc.#!/bin/ksh/usr/sbin/logrotate -f -s /apps/oracle/general/config/listener_HOSTNAME.status /apps/oracle/general/config/listener_HOSTNAME.conf

This logrotate config file looks like this:LISTENER_HOSTNAME.conf{weeklycopytruncaterotate 4compressmail charles@dbaexpert.com}