1

Managing Oracle Enterprise Manager Cloud Control 12c with

Oracle Clusterware

Leighton L. Nelson#EM12c

2

About Me• Oracle DBA for 10+ years• Oracle ACE• Oracle Certified Expert: RAC and Grid

Infrastructure 11gR2• Co-Author Expert Oracle Enterprise Manager

Cloud Control 12c – Apress 2013

#C1

4LV

See Me Speak at COLLABORATE 14 – IOUG Forum■ 5 days of training with more than 5,500 expert Oracle users

■ List speaking sessions

■ User-driven training in:▪ Big Data▪ BI▪ Cloud Computing▪ Database Performance▪ Database Development

■ Learn more at collaborate14.ioug.org

3

▪ Engineered Systems▪ High Availability▪ OEM▪ Security

#C1

4LV

COLLABORATE 14 – IOUG Forum

■ 5 days of training with more than 5,500 expert Oracle users

■ Case studies and sessions cover the latest in:▪ Big Data▪ BI▪ Cloud Computing▪ Database Performance▪ Database Development

■ Learn more at collaborate14.ioug.org

4

▪ Engineered Systems▪ High Availability▪ OEM▪ Security

5

Agenda• Oracle Clusterware Overview• OEM 12c Architecture Overview• OEM High Availability• Level 2 HA Configuration• Oracle Clusterware Setup• OEM Installation• OEM Failover • Management Repository Configuration

6

Oracle Clusterware

6

ASM + Clusterware = Grid Infrastructure

7

OEM 12c Architecture Overview

8

OEM High Availability Levels

Level Description Load Balancer Required Cost1 OMS and Repository on separate hosts.

No redundancy. No $

2 OMS installed on shared storage with VIP used in active/passive failover.Repository Database using local Data Guard.

No $$

3 Multiple OMSs in active/active configuration. Repository using RAC Database with local Data Guard

Yes, At primary site $$$

4 Primary OMS in active/active configuration with RAC Repository database. Standby OMS at DR site in active/active configuration. Standby RAC database at DR site

Yes. At pimary and standby sites

$$$$

9

Level 2 – Active/Passive HA• A minimum of 2 Servers Required• OMS binaries installed on shared filesystem

• NFS/OCFS2/DBFS/ACFS• OMS can run on one node in cluster at any given

time• Data Guard for Management Repository

10

Level 2 – Active/Passive OMS

11

Level 2 – Active/Passive OMS

12

Oracle Clusterware Setup• Clusterware can be used to create/manage VIP

• 11.2+ uses appvipcfg for VIP<GRID_HOME>/bin/appvipcfg create -network=1 \ -ip=192.168.1.0 \ -vipname=omsvip \ -user=root

• VIP can be created on non-default network• In Oracle 12c Flex Clusters app vips can be

created on leaf nodes• Allow Oracle Grid Infrastructure software owner

(e.g. grid) to run the script to start the VIP.<GRID_HOME>/bin/crsctl setperm resource omsvip -u user:grid:r-x

13

Oracle Clusterware Setup• Start the VIP as the GI owner e.g. grid<GRID_HOME>/bin/crsctl start resource omsvip

• Check the status of the VIP<GRID_HOME>/bin/crsctl status resource omsvipThe status of the output should be similar to the following:NAME=omsvipTYPE=app.appvip_net1.typeTARGET=ONLINESTATE=ONLINE on oms1

• View full configuration with -f<GRID_HOME>/bin/crsctl status resource omsvip –f

14

Oracle Clusterware Setup• Check if virtual hostname and VIP are resolvablenslookup <omsvip>• Also do a reverse lookup of the IP address.nslookup <virtual IP address>• Verify that the IP address returned from the

nslookup output is running on the OMS host.ifconfig –a | grep <virtual IP address>

15

OEM Installation• Create ORACLE_HOME for the OMS on the shared

storage on all nodes in the clustermkdir –p /u01/app/oms_share

• Create Oracle Inventory directory under ORACLE_HOME for the OMS on all nodes

mkdir /u01/app/oms_share/oraInventory

• Create the inventory pointer in the oraInventory directory

vi oraInst.locinventory loc=/u01/app/oracle/oms_share/oraInventory inst_group=oinstall

16

OEM Installation• Install OEM by setting ORACLE_HOSTNAME

environment variablerunInstaller -invPtrloc /u01/app/oms_share/oraInst.loc

ORACLE_HOSTNAME=omsvip.example.com –debug

17

OEM Failover• To   manually relocate the VIP to another host in

the cluster issue the following command.$crsctl relocate res omsvipCRS-2673: Attempting to stop 'omsvip' on 'oms1'CRS-2677: Stop of 'omsvip' on 'oms1' succeededCRS-2672: Attempting to start 'omsvip' on 'oms2'CRS-2676: Start of 'omsvip' on 'oms2' succeeded

• Check if the IP address associated with the VIP is running on the relocated host.

ifconfig –a|grep <vip>

18

OEM Failover• Establish IP on failover server (done through Clusterware)• Start listener (if part of same failover group)• Start database (if required)• Set the ORACLE_HOSTNAME environment variable to the

virtual hostname. Continuing with our example we use the command below.

export ORACLE_HOSTNAME=omsvip.example.com• Start the OMS on the new node$OMS_HOME/bin/emctl start oms

19

Add OEM Clusterware resource

• OEM can be added as Clusterware resource• Administrator managed (static, 2 – nodes)• Policy managed (dynamic, > 2 nodes)

• Handled through Agent Framework• C/C++

• Create Action Script• Store on shared storage• Specify START/STOP/CHECK/ABORT routines

20

Example $ crsctl add serverpool oem_sp -attr "PARENT_POOLS=Generic, SERVER_NAMES=oms1

oms2”

$ crsctl add resource oem -type cluster_resource -attr "ACTION_SCRIPT=/opt/cluster/scripts/oem12c.scr, PLACEMENT='restricted', SERVER_POOLS=oem_sp, CHECK_INTERVAL=‘60', RESTART_ATTEMPTS='2', START_DEPENDENCIES='hard(omsvip)', STOP_DEPENDENCIES='hard(omsvip)'"

21

Clusterware resource• Oracle 12c introduced a new generic_application resource

type

crsctl add resource oem -type generic_application -attr "START_PROGRAM='$OMS_HOME/bin/emctl start oms', STOP_PROGRAM='$OMS_HOME/bin/emctl stop oms -all', CLEAN_PROGRAM='$OMS_HOME/bin/emctl stop oms –all -

force', CHECK_PROGRAM='$OMS_HOME/bin/emctl status oms'

22

Use OEM to “dogfood” HA configuration

22

23

Adding OEM Resources using OEM

23

24

24

25

25

26

26

27

27

28

28

29

29

30

30

31

31

32

32

33

33

34

34

35

35

36

36

37

37

38

Management Repository• Use OEM to Create Standby Database!• Can be configured using Data Guard with Fast-

Start-Failover• Also can be configured with RAC/RAC One Node

• Storage becomes SPOF• Listeners should be reachable from all nodes in

cluster• Database should be started before starting OEM

39

Configure OEM with Data Guard Repo

Create database services for fast failover.• srvctl add service -d emrep -s oemsvc -l

PRIMARY -q TRUE -e SESSION -m BASIC -w 10 -z 6

• srvctl add service -d emrep2 -s oemsvc -l PRIMARY -q TRUE -e SESSION -m BASIC -w 10 -z 6

40

Configure OMS with Data Guard Repoemctl config oms -store_repos_details -

repos_conndesc '"(DESCRIPTION=(FAILOVER=ON)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=repo1)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=repo2)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=oemsvc))(FAILOVER_MODE=(TYPE=select)(METHOD=basic)))"' -repos_user sysman

41

Configure OMS with RAC/RAC One Node Repoemctl config oms -store_repos_details -

repos_conndesc " (DESCRIPTION= (ADDRESS=(PROTOCOL=TCP) (HOST=emrep-scan.example.com)(PORT=1521) ) (CONNECT_DATA=(SERVER=DEDICATED)

(SERVICE_NAME = emrep)))" -repos_user sysman

42

Licensing

42

43

Caveats

•Shared File System becomes SPOF for OMS•Incurred downtime between failover (minutes)•May require additional licenses if Data Guard is used

43

44

THANK YOUhttp://blogs.griddba.com

linkedin.com/pub/leighton-nelson/4/6b8/a7a

@leight0nn

leightonn@gmail.com

45

References• How to Configure Grid Control OMS in Active/Passive

CFC Environments failover / HA (Doc ID 405642.1)• How to Configure the OMS Connect String when

Repository is in a Dataguard setup [ID 1328768.1]• Oracle® Clusterware Administration and Deployment

Guide12c Release 1• How To Configure Grid Control Components for High

Availability Note 406014.1• Oracle Clusterware 11gR2 Whitepaper

http://www.oracle.com/technetwork/database/clusterware/overview/oracle-clusterware-11grel2-owp-1-129843.pdf