Under the Hoodof Oracle Clusterware 2.0:Grid InfrastructureAlex Gorbachev

29 August, 2012

© 2009/2010 Pythian

Agenda

• Place of Grid Infrastructure in Oracle RAC

• Node membership and evictions

• Clusterware architecture & startup sequence

• Resources Management and troubleshooting

2

© 2009/2010 Pythian

Agenda

2

Nee

d to

mem

oriz

e

Understanding

Low

Hig

h

Shallow In-depth

The more you understand,

the less you need to memorize

© 2012 Pythian

Single Instance Oracle Database

3

DatafilesControlfilesredo logsflashback logs, change tracking and etc...Database

SERVER

INSTANCE

Memory (SGA, PGA)Processes (PMON, SMON, LGWR and etc. + multiple shadow processes)

Read/write

APP

Query/DML/DDL

© 2012 Pythian

Single Instance Oracle Database

4

Database

SERVER

INSTANCE

APP

© 2012 Pythian

Oracle RAC Database

5

Database

SERVER 1

INSTANCE 1

APP

© 2012 Pythian

Oracle RAC Database

5

Database

SERVER 1

INSTANCE 1

APP

SERVER 2

INSTANCE 2

© 2012 Pythian

Oracle RAC Database

5

Database

SERVER 1

INSTANCE 1

APP

SERVER 2

INSTANCE 2

SERVER 3

INSTANCE 3

© 2012 Pythian

Oracle RAC Database

5

Database

SERVER 1

INSTANCE 1

APP

SERVER 2

INSTANCE 2

SERVER 3

INSTANCE 3

© 2012 Pythian

Oracle RAC Database

5

Database

SERVER 1

INSTANCE 1

APP

SERVER 2

INSTANCE 2

SERVER 3

INSTANCE 3

© 2012 Pythian6

RAC looks simple. Eh?

© 2009/2010 Pythian

Role of Grid Infrastructure

OS

Grid Infrastr.

Instance

ASM

VIPListener

Service

Shared storage

OCR Votingdisk

OS

Grid Infrastr.

Instance

ASM

VIPListener

Service

OS

Grid Infrastr.

Instance

ASM

VIPListener

Service

interconnectstorage access

7

© 2009/2010 Pythian

Role of Grid Infrastructure

OS

Grid Infrastr.

Instance

ASM

VIPListener

Service

Shared storage

OCR Votingdisk

OS

Grid Infrastr.

Instance

ASM

VIPListener

Service

OS

Grid Infrastr.

Instance

ASM

VIPListener

Service

interconnectstorage access

7

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

8

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

8

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

8

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

9

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

9

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

ShootTheOtherNodeInTheHead

9

© 2009/2010 Pythian

OS

Clusterware

CSSD

CRSDEVM

D

OPROCD

RACG

VIP

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

10

© 2009/2010 Pythian

OS

Clusterware

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

11

© 2009/2010 Pythian

OS

Clusterware

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

11

© 2009/2010 Pythian

OS

Clusterware

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

11

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

11

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

AskTheOtherNodeToRebootItself

(c) known quote

11

© 2009/2010 Pythian12

11gR2 Grid Infrastructure:CSSD attempts graceful

shutdown

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

Votingdisk

OS

Clusterware

OPROCD

CSSD

interconnect

13

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

Votingdisk

OS

Clusterware

OPROCD

CSSD

interconnect

CSSD Monitor/Agent

13

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

Votingdisk

interconnect

CSSD Monitor/Agent

13

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

14

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

14

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

14

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

OPROCD/CSSD Mon

14

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

OPROCD/CSSD Mon

14

© 2009/2010 Pythian

OS

Clusterware

OPROCD

CSSD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

15

© 2009/2010 Pythian

OS

Clusterware

OPROCD

CSSD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

15

© 2009/2010 Pythian

OS

Clusterware

OPROCD

CSSD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

15

© 2009/2010 Pythian

CSSD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

15

© 2009/2010 Pythian

OS

Clusterware

OPROCD

CSSD

OS

Clusterware

OPROCD

CSSDinterconnect

Votingdisk

16

© 2009/2010 Pythian

OS

Clusterware

OPROCD

CSSD

OS

Clusterware

OPROCD

CSSDinterconnect

16

© 2009/2010 Pythian

OS

Clusterware

OPROCD

CSSD

OS

Clusterware

OPROCD

CSSDinterconnect

16

© 2009/2010 Pythian

CSSD CSSDinterconnect

16

© 2009/2010 Pythian17

11gR2 Grid Infrastructure:CSSD attempts graceful

shutdown

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

18

InstanceLMON

Instance

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

18

InstanceLMON

Instance

member kill

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

18

InstanceLMON

Instance

member kill

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

18

InstanceLMON

Instance

member kill

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

18

InstanceLMON

Instance

member kill

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

18

InstanceLMON

Instance

member kill

Eviction by escalation of a member kill

© 2009/2010 Pythian

OS

Clusterware

CSSD

OPROCDinterconnect

18

InstanceLMON

member kill

Eviction by escalation of a member kill

© 2009/2010 Pythian

OS

Clusterware

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

19

© 2009/2010 Pythian

OS

Clusterware

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

19

11gR2 Intelligent Platform Management Interface

© 2009/2010 Pythian

OS

Clusterware

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

19

11gR2 Intelligent Platform Management Interface

© 2009/2010 Pythian

OS

Clusterware

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

19

11gR2 Intelligent Platform Management Interface

© 2009/2010 Pythian

OS

Clusterware

OPROCD

OS

Clusterware

CSSD

OPROCDinterconnect

Votingdisk

CSSD

20

Exadata Fencing

© 2009/2010 Pythian

10g / 11gR1• Linux & UNIX inittab

init.cssd

init.evmd

init.crsd

• Linux & UNIX init.dinit.crs

• Windows Services

Grid Infrastructure Startup

21

11gR2• Linux & UNIX inittab

init.ohasd run

• Linux & UNIX init.dohasd start

• Windows Services

© 2009/2010 Pythian

Startup in Linux & Unix

[root@cheese2 ~]# ps -fe | grep 'init\.' | grep -v grep

root 4283 1 0 02:52 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run

[root@cheese2 ~]# tail -1 /etc/inittab

h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

[root@cheese2 ~]# ls -l /etc/rc3.d/*ohasd*

lrwxrwxrwx 1 root root 17 Sep 15 02:02 /etc/rc3.d/K15ohasd -> /etc/init.d/ohasd

lrwxrwxrwx 1 root root 17 Sep 15 02:02 /etc/rc3.d/S96ohasd -> /etc/init.d/ohasd

22

© 2009/2010 Pythian

init.crsd run

init.evmd run

init.cssd fatal

init.crs start

init.cssd autostart

init.cssd oprodc

init.cssd oclsomon

init.cssd daemon

init.cssd oclsvmon

oprocd

oclsomon.bin

ocssd.bin

oclsvmon.bin

evmd.bin

t

/etc/oracle/scls_scr/{host}/root/cssrun

/etc/oracle/scls_scr/{host}/root/crsstart• enable• disable

crsd.bin

Pre 11gR2 Clusterware Startup

23

© 2009/2010 Pythian

t

24

Grid Infrastructure Startup

© 2009/2010 Pythian

t

24

init.ohasd run

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

/etc/init.d/ohasd

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

/etc/oracle/scls_scr/{host}/root/ohasdstr• enable• disable

24

init.ohasd run

/etc/init.d/ohasd

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

/etc/oracle/scls_scr/{host}/root/ohasdstr• enable• disable

24

init.ohasd run

/etc/init.d/ohasd

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

OHAS

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

OHAS

Ora Agent

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

OHAS Root Agent

Ora Agent

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

OHAS Root Agent

Ora Agent

CSSD Agent

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

OHAS Root Agent

Ora Agent

CSSD Agent

CSSD Monitor

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

OHAS Root Agent

Ora Agent

CSSD Agent

CSSD Monitor

CSS

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

OHAS Root Agent

Ora Agent

CSSD Agent

CSSD Monitor

CSS

CRS

Disk Monitor

CTSS

ACFS Drivers

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

OHAS Root Agent

Ora Agent

CSSD Agent

CSSD Monitor

CSS

CRS

Disk Monitor

CTSS

ACFS Drivers

MDNS

GIPC

GPnP

EVM

ASM

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

OHAS Root Agent

Ora Agent

CSSD Agent

CSSD Monitor

CSS

CRS

Disk Monitor

CTSS

ACFS Drivers

MDNS

GIPC

GPnP

EVM

CRS Root Agent

•VIP•SCAN IP•Network•GNS•ACFS Registry

ASM

Grid Infrastructure Startup

© 2009/2010 Pythian

t

/etc/oracle/scls_scr/{host}/root/ohasdrun

24

init.ohasd run

OHAS Root Agent

Ora Agent

CSSD Agent

CSSD Monitor

CSS

CRS

Disk Monitor

CTSS

ACFS Drivers

MDNS

GIPC

GPnP

EVM

CRS Root Agent

CRS Ora Agent

•VIP•SCAN IP•Network•GNS•ACFS Registry

•Database•Instance•Listener•Services•Diskgroups•ONS•eONS•SCAN Listener

ASM

Grid Infrastructure Startup

© 2009/2010 Pythian

t

25

init.ohasd run

OHAS Root Agent

Ora Agent

CSSD Agent

CSSD Monitor

CSS

CRS

Disk Monitor

CTSS

ACFS Drivers

MDNS

GIPC

GPnP

EVM

CRS Root Agent

CRS Ora Agent

•VIP•SCAN IP•Network•GNS•ACFS Registry

•Database•Instance•Listener•Services•Diskgroups•ONS•eONS•SCAN Listener

ASM

Grid Infrastructure Startup (static slide)

© 2009/2010 Pythian

Grid Infrastructure Log Files

• alert<host>.log• ohasd• crsd• cssd• agent/ohasd/oraagent_oracle• agent/ohasd/oracssdagent_root• agent/ohasd/oracssdmonitor_root• agent/ohasd/orarootagent_root• agent/crsd/oraagent_oracle• agent/crsd/orarootagent_root

• ctssd• diskmon• gipcd• gnsd• gpnpd• mdnsd• racg

26

$GRID_HOME/log/{hostname}/

© 2009/2010 Pythian

Oracle Cluster Registry

• Repository for all shared configuration data

• Except OCR location itself

• OCR is accessed mostly read-only• Every component reads OCR

• OCR is written only by CRS• only from a single OCR master node

• 11gR2 - Oracle Local Registry (OLR)• managed by ohasd

27

© 2009/2010 Pythian

DEMOInterconnect Failure

• Simulate with “ifconfig eth1 down” on node 2• Both nodes notice the loss• Racing to evict each other

• from voting disk => 2 equal sub-clusters (cohorts)

• survives the one with the lowest leader #

• leader is the node with lowest # in sub-cluster

• Winner evicts another node• Setting kill-block in voting disk

• 11gR2 new feature: CSSD does clean restart

28

© 2009/2010 Pythian

DEMOVoting Disk Failure

• Simulate with storage interface down or on NFS server

• CSSD detects voting disk IO stale• disktimeout setting - 200 seconds default

• CSSD starts eviction • 11gR2 new feature: CSSD does clean restart

29

© 2009/2010 Pythian

DEMOCSSD is not healthy

• Simulate using kill -STOP <cssd.bin pid> (and try kill -9)• Another node observes NHB loss

• After misscount seconds => attempt eviction

• but CSSD is frozen and can’t commit suicide

• CSSD Monitor detects CSSD timeout• Commit suicide

30

© 2009/2010 Pythian

HomeworkHost sick - CPU stalled

• Used to simulate by pausing OPROCD• kill -STOP <oprocd pid>

• sleep 1 or 2

• kill -CONT <oprocd pid>

• OPROCD is now a thread in CSSD Monitor• kill -STOP {cssdmonitor.bin} ; sleep 1 ; kill -CONT {cssdmonitor.bin}

• Doesn’t produce any visible results! Is there still OPROCD?

31

© 2009/2010 Pythian

DEMOStartup troubleshooting

• Break before starting up• Interconnect, voting disk, Grid Home missing

• Check processes using “ps -fe | grep init”• Check syslog (/var/log/messages)• boot sequence• Clusterware log files

• if *.bin processes are running already

• crsctl check crs• crsctl status resource -t -init

32

© 2009/2010 Pythian

11gR2 Cluster Resources

• Introduced resource type• APPLICATION was the only supported resource type in 11gR1

• Resource types using resource agents

• Cluster and local resources• Sophisticated resources dependencies (stop & start)

• hard

• weak

• attraction

• pullup

• dispersion

33

© 2009/2010 Pythian

t

34

init.ohasd run

OHAS Root Agent

Ora Agent

CSSD Agent

CSSD Monitor

CSS

CRS

Disk Monitor

CTSS

ACFS Drivers

MDNS

GIPC

GPnP

EVM

CRS Root Agent

CRS Ora Agent

• VIP• SCAN IP• Network• GNS• ACFS Registry

• Database• Instance• Listener• Services• Diskgroups• ONS• eONS• SCAN Listener

ASM

Grid Infrastructure Processes• ora.cluster_interconnect.haip

OHAS Internal Resources

CRS Managed Resources

© 2009/2010 Pythian

Troubleshooting Something Down

• OHASD up?• OHASD Agents up? • Internal resources up?• CRSD Agents up?• Managed resources up?

• Listener & VIP

• Database & ASM instance

• Services

• Have the nodes rebooted?• Have resources re-started?

35

© 2009/2010 Pythian

11gR2 Grid Infrastructure References

• Oracle Clusterware Administration and Deployment Guide

• MOS 1053147.1

• 11gR2 Clusterware and Grid Home - What You Need to Know

• MOS 1050908.1

• How to Troubleshoot Grid Infrastructure Startup Issues

• MOS 1053970.1

• Troubleshooting 11.2 Grid Infastructure Installation Root.sh Issues

• MOS 1050693.1

• Troubleshooting 11.2 Clusterware Node Evictions (Reboots)

• MOS 942166.1

• How to Proceed from Failed 11gR2 Grid Infrastructure Installation

36