4RAC

ORACLE 10g RAC OVERVIEW

AGENDA

What is Real Application Cluster (RAC) ? Why to use RAC ? Single instance vs RAC RAC Architecture Sharing of Resources Background processes Internal structures and services RAC Administration ORACLE Clusterware RAC Mechanism Failover Load Balancing

WHAT IS RAC ?

Real Application Clusters (RAC) introduced in oracle 9i, is a technology that enables a single database to work on multiple instances simultaneously on different nodes.A RAC database requires three components: cluster nodes Shared storage Oracle Clusterware

To identify RAC instance in a database the following can be used: Use the DBMS_UTILITY.IS_CLUSTER_DATABASE function Show parameter CLUSTER_DATABASE

WHY USE RAC ?

High AvailabilityFailoverReliabilityScalabilityManageabilityRecoverabilityTransparencyRow LockingError DetectionBuffer Cache ManagementContinuous OperationsLoad Balancing/SharingReduction in total cost of ownership

SINGLE INSTANCE ~ RAC

Difference between Oracle Single instance and Oracle RAC

SingleInstanceOracle RAC

Single instance accessing databaseMultiple instances accessing same database

Database can be local or sharedDatabase must be on shared.

Minimum one node is RequiredMinimum two node required

Clusterware software is not requiredClusterware Software is required.

Consists of one SGA and one set of backgroundprocessesEach instance having ownone SGA and one set of backgroundprocesses

One set of redo logsMultiple set redo logs depending on number of instances

No Cache fusion used un Single instanceCache fusion is used to transfer or access block from remote instance

V$ views to monitor and manage instanceFor monitoring at the cluster level Oracle provides GV$ views.

Meeting some of the business requirements (availability, scalability) is limited to a single instance configuration.Modern business requirements of high availability and linear scalability are provided by multiple instances sharing a common physical database

RAC ARCHITECTURE:

Public Network:It is the public IP on which listeners would be listening and clients would contact the listener on this public IP.Private Network Interconnect:It is a network path i.e exclusively used for inter-instance communication used by the cluster and dedicated to the server nodes of a cluster. It is used for the synchronization ofresources and in some cases for the transfer of data. It has a high bandwidth and low latency.Storage Network:It is the network which connects the instances to the database in RAC.WHATS SHARED, WHATS NOTSHAREDDisk accessResources that manage dataAll instances have common data & controls filesNOT SHARED1) Each node has its own dedicated: System memory Operating system Database instance Application software2) Each instance has individual Log files and Rollback segments

BACKGROUND PROCESSES:

1. Global Cache Service Processes (LMSn)LMSn handles block transfers between the holding instances buffer cache and requesting foreground process on the requesting instance.LMS maintains read consistency by rolling back any uncommitted transactions for blocks that are being requested by any remote instance.Even if n value(0-9) varies depending on the amount of messaging traffic amongst nodes in the cluster, there is default, one LMS process per pair of CPUs.2. Global Enqueue Service Monitor (LMON)It constantly handles reconfiguration of locks and global resources when a node joins or leaves the cluster. Its services are also known as Cluster Group Services (CGS).3. Global Enqueue Service Daemon (LMD)It manages lock manager service requests for GCS resources and sends them to a service queue to be handled by the LMSn process. The LMD process also handles global deadlock detection and remote resource requests (remote resource requests are requests originating from another instance).4. Lock Process (LCK)LCK manages non-cache fusion resource requests such as library and row cache requests and lock requests that are local to the server. Because the LMS process handles the primary function of lock management, only a single LCK process exists in each instance.5. Diagnosability Daemon (DIAG)This background process monitors the health of the instance and captures diagnostic data about process failures within instances. The operation of this daemon is automated and updates an alert log file to record the activity that it performs.

NOTE:In RAC environment, all instances have their separate alert logs.Global Service Daemon (GSD)This is a component in RAC that receives requests from the SRVCTL control utility to execute administrative tasks like startup or shutdown. The command is executed locally on each node and the results are returned to SRVCTL. The GSD is installed on the nodes by default.INTERNAL STRUCTURES AND SERVICES

Global Resource Directory (GRD)Records current state and owner of each resourceContains convert and write queuesDistributed across all instances in clusterMaintained by GCS and GES

Global Cache Services (GCS)Implements cache coherency for databaseCoordinates access to database blocks for instancesGlobal Enqueue Services (GES)Controls access to other resources (locks) including library cache and dictionary cachePerforms deadlock detection

RAC ADMINISTRATION:

Parameter File Using a shared SPFILE is recommended while each instance can have its own dedicated parameter file as well. SPFILE should be placed on a shareable disk subsystem like a rawdevice, a clustered file system, or Automatic Storage Management (ASM). Oracle instance parameters for the RAC environment can be grouped into three major categories i.e Unique, Identical,Instance-Specific Parameters. If a parameter appears more than once in a parameter file, the last specified value is the effective value, unless the values are on consecutive linesin that case values from consecutive lines are concatenated. To set the value for a parameter the following syntax is used. An asterisk (*) or no value in place of an instance_nameimplies that the parameter value is valid for all the instances. . =*.undo_management=auto To set the parameter in SPFILE, the following command can be used. alter system set =scope=comment=deferred sid= When DBCA is used to create an RAC database, by default it creates an SPFILE on the shared disk subsystem used. Else it can also be created manually.

Undo Management

Oracle stores original values of the data called Before imagein undo segments to provide read consistency and to roll back uncommitted transactions. In the Oracle RAC environment, each instance stores transaction undo data in its dedicated undo tablespace. For this we must set undo_tablespace for individual instances as follows. prod1.undo_tablespace= undo_tbs1 prod2.undo_tablespace=undo_tbs2 Instances in RAC can use either automatic or manual undo management, but the parameter undo_managementhas to be the same across all the instances.To increase the size of an undo tablespace, either of the following can be used.1. Add another datafile to undo tablespace.2. Increase the size of the existing datafile(s) belonging to the undo tablespace. While using Manual Undo Management following considerations aremade.1. Use manual undo management only if you have very good reason for not using automatic undo management.2.Do not create other objects such as tables, indexes, and so on in the tablespace used for rollback segments.3.Create one rollback segment for every four concurrent transactions.Temporary Tablespace

In an RAC environment, all instance share the same temporary tablespace with each instance creating a temporary segment in the tablespace it is using. The size should be at least equal to the concurrent maximum requirement of all the instances. The default temporary tablespace cannot be dropped or taken offline; however it can be changed followed by dropping or taking offline the original default temporary tablespace. To determine the temporary tablespace used by each instance GV$SORT_SEGMENTand GV$TEMPSEG_USAGE views can be queried on INST_IDcolumn basis. In a temporary tablespace group, an user will always use the same assigned temporary tablespace irrespective of the instance being used. Aninstance can reclaim the space used by other instancestemporary segments in that tablespace if required for large sorts.

Online Redologs

In an RAC environment, each instance has its own set of online redologfiles and redologscalled Thread. The size of an online redologis independent of other instancesredologsizes and is determined by the local instances workload and backup and recovery considerations. Each instance has exclusive write access to its own online redologfiles, however it can read another instances current online redologfile to perform instance recovery if needed. Thus, an online redologneeds to be located on a shared storage device and cannot be on a local disk. Similar to single instance database, views V$LOG and V$LOGFILE can be used in RAC for getting information about redo logs.

Archive Logs

Steps to enable or disable log mode1.Set cluster_database=false for the instance:SQL>alter system set cluster_database=false scope=spfile sid= prod1;2. Shut down all the instances accessing the database:$srvctl stop database -d prod3. Mount the database using the local instance:SQL>startup mount4. Enable archiving:SQL>alter database archivelog/ noarchivelog;5. Change the parameter cluster_database=true for the instance prod1:SQL>alter system set cluster_database=true scope=spfile sid=prod1;6. Shut down the local instance:SQL>shutdown ;7. Bring up all the instances:$srvctl start database -d prodFlashback Area (Enable / Disable )1. Set cluster_database=false for the instance to perform this operation.SQL>alter system set cluster_database=false scope=spfile sid= prod1;2. Set DB_RECOVERY_FILE_DEST_SIZE and DB_RECOVERY_FILE_DEST.The DB_RECOVERY_FILE_DEST parameter should point to a shareable disk subsystem.SQL>alter system set db_recovery_file_dest_size=200m scope=spfile;SQL>alter system set db_recovery_file_dest=/ocfs2/flashback scope=spfile;3. Shut down all instances accessing the database:$srvctl stop database -d prod4. Mount the database using the local instance:SQL>startup mount5. Enable the flashback by issuing the following command:SQL>alter database flashback on/off;6. Change back the parameter to cluster_database=true for the instance prod1:SQL>alter system set cluster_database=true scope=spfile sid=prod17. Shut down the instance:SQL>shutdown;8. Start all the instances:$srvctl start database -d prod

SRVCTL Utility To start all instances associated with a database the following command can be executed from any of the nodes. The command also starts listeners on each node if not already running. $srvctl start database -d Similarly, to shut down all instances associated with the database stop command can be used which does not stop listeners, as they mightbe serving other database instances running on the same machine. $srvctl stop database d Options specified by -o are directly passed on to SQL *Plus as command-line options for start/stop commands.$srvctl stop database -d -o immediate$srvctl start database -d -o force To perform operation on individual instance level, -I option can be used. $srvctl stop database -d -i instance

ORACLE CLUSTERWARE

It is a cluster manager integrated in the Database to handle the cluster including node membership, group services, global resource management, and high availability functions. It can also be used with non-cluster database. Names as it has evolved - 1.Oracle Cluster Management Services (OCMS) 9.0.1 and 9.22.Cluster Ready Services (CRS) ( Generic, portable cluster manager) 10.1 3.Oracle Clusterware (CRS has been renamed) 10.2 Background processes - 1. Cluster Synchronization Service (CSS) 2. Cluster Ready Services (CRS) 3. Event Manager (EVM) To administer Clusterware, CRSCTL utility can be used i.e present in $ORA_CRS_HOME/bin. Oracle Clusterware must be installed prior to installing ORACLE database. We need ROOT user during the installation process to perform various tasks requiring super user privileges. Being the first Oracle software to be installed on the system, it is susceptible to configuration errors. So it is recommended to useCluster Verification Utility (CLUVFY) on all nodes i.e introduced in Oracle 10.2 but backward compatible with 10.1. Oracle Clusterware requires two files that must be located on shared storage for its operation.1. Oracle Cluster Registry (OCR)2. Voting DiskOracle Cluster Registry (OCR)1) Located on shared storage and in Oracle 10.2 and above can be mirrored to maximum two copies.2) Defines cluster resources including Databases and Instances ( RDBMS and ASM) Services and Node Applications (VIP,ONS,GSD) Listener ProcessVoting Disk (Quorum Disk / File in Oracle 9i)1) Used to determine RAC instance membership and is located on shared storage accessible to all instances.2) used to determine which instance takes control of cluster in case of node failure to avoid split brain3) In Oracle 10.2 and above can be mirrored to only Odd number of copies (1, 3, 5 etc)

VIRTUAL IP (VIP)

To make the applications highly available and to eliminate SPOF,Oracle 10g introduced a new feature called CLUSTER VIPs i.e a virtual IP address different from the set of in cluster IP addresses that is used by the outside world to connect to the database. A VIP name and address must be registered in the DNS along with standard static IP information. Listeners would be configured to listen on VIPs instead of the public IP. When a node is down, the VIP is automatically failed over to oneof the other nodes. The node that gets the VIP will re-ARPto the world, indicating the new MAC address of the VIP. Clients are sent error message immediately rather than waiting for the TCP timeout value.

CACHE FUSION:

1) Underlying technology that enables RAC (starting with 9i and improved performance with 10g)2) Protocol that allows instances to combine their data caches intoa shared global cache.3) Allows any node to get the most up-to-date data information from the cache of any other node in the cluster without having to access the disk drives again.4) Needed when Dirty Blockof data is created Data from disk is read into memory on a node Data is updated on that node Data hasnt been written to disk yet Another node requests the data

WHAT IS FAILOVER?

1) If a node in the shared disk cluster fails, the system dynamically redistributes the workload among the surviving cluster nodes.2) RAC checks to detect node and network failures. A disk-based heartbeat mechanism uses the control file to monitor node membership and the cluster interconnect is regularly checked to determine correct operation.

3) Enhanced failover reliability in 10g with the use of Virtual IP addresses (VIPs)4) If one node or instance fails, node detecting failure does the following. Read redo log of failed instance from last checkpoint Apply redo to datafiles including undo segments (roll forward) Rollback uncommitted transactions Cluster is frozen during part of this process

FAST APPLICATION NOTIFICATION (FAN)

1) FAN is a method introduced in Oracle 10.1, by which applicationscan be informed of changes in cluster status for - Fast node failure detection Workload balancing2) Advantageous by preventing applications from - Waiting for TCP/IP timeouts when a node fails Trying to connect to currently down database service Processing data received from failed node3) Can be notified using - Server side callouts Fast Connection Failover (FCF) ONS API

ORACLE NOTIFICATION SERVICE (ONS)

1) ONS, introduced in Oracle 10.1 is a subscribe serviceused by Oracle Clusterware to propagate messages to : Nodes in cluster Middle-tier application servers Clients2) ONS is automatically launched for RAC on each node as part of the Oracle Clusterware installation process.However, it can also be configured to run on nodes hosting client or mid-tier applications.3) It is an underlying mechanism for Fast Application Notification (FAN).4) ONSruns independent of Transparent Application Failover (TAF).

TRANSPARENT APPLICATION FAILOVER (TAF)

1) TAF is a client-side feature that allows for clients to reconnect to surviving databases in the event of a failure of a database instance.2) Masks failures to end users; they dont need to log back into the system3) Applications and users are transparently reconnected to another node4) Applications and queries continue uninterrupted5) Transactions can failover and replay6) Login context maintained7) DML transactions are rolled back8) Requires configuration in TNSNAMES.ORARAC_FAILOVER = (DESCRIPTION = (ADDRESS_LIST = (FAILOVER = ON)(ADDRESS = (PROTOCOL = TCP)(HOST = node1)(PORT = 1521))(ADDRESS = (PROTOCOL = TCP)(HOST = node2)(PORT = 1521))) (CONNECT_DATA = (SERVICE_NAME = RAC) (SERVER = DEDICATED) (FAILOVER_MODE =(TYPE=SELECT)(METHOD=BASIC)(RETRIES=30)(DELAY=5))) )

What is FAN?Fast application Notificationas it abbreviates to FAN relates to the events related toinstances, services and nodes. This is a notification mechanism that Oracle RAC usesto notify other processes about the configuration and service level information thatincludes service status changes such as, UP or DOWN events. Applications canrespond to FAN events and take immediate action.Where can we apply FAN UP and DOWN events?FAN UP and FAN DOWN events can be applied to instances, services and nodes.State the use of FAN events in case of a cluster configuration change?During times of cluster configuration changes,Oracle RAC high availability frameworkpublishes a FAN event immediately when a state change occurs in the cluster.Soapplications can receive FAN events and react immediately.This prevents applicationsfrom polling database and detecting a problem after such a state change.Why should we have seperate homes for ASm instance?It is a good practice to have ASM home seperate from the databasehome(ORACLE_HOME).This helps in upgrading and patching ASM and the Oracledatabase software independent of each other.Also,we can deinstall the Oracle databasesoftware independent of the ASM instance.What is the advantage of using ASM?Having ASM is the Oracle recommended storage option for RAC databases as the ASMmaximizes performance by managing the storage configuration across the disks.ASMdoes this by distributing the database file across all of the available storage within ourcluster database environment.What is rolling upgrade?It is a new ASM feature from Database 11g.ASM instances in Oracle database 11grelease(from 11.1) can be upgraded or patched using rolling upgrade feature. Thisenables us to patch or upgrade ASM nodes in a clustered environment without affectingdatabase availability.During a rolling upgrade we can maintain a functional cluster whileone or more of the nodes in the cluster are running in differentsoftware versions.Can rolling upgrade be used to upgrade from 10g to 11g database?No, it can be used only for Oracle database 11g releases(from 11.1).State the initialization parameters that must have same value for every instance inan Oracle RAC database:-Some initialization parameters are critical at the database creation time and must havesame values. Their value must be specified in SPFILE or PFILE for every instance. Thelist of parameters that must be identical on every instance are given below:ACTIVE_INSTANCE_COUNTARCHIVE_LAG_TARGETCOMPATIBLECLUSTER_DATABASECLUSTER_DATABASE_INSTANCECONTROL_FILESDB_BLOCK_SIZEDB_DOMAINDB_FILESDB_NAMEDB_RECOVERY_FILE_DESTDB_RECOVERY_FILE_DEST_SIZEDB_UNIQUE_NAMEINSTANCE_TYPE (RDBMS or ASM)PARALLEL_MAX_SERVERSREMOTE_LOGIN_PASSWORD_FILEUNDO_MANAGEMENTCan the DML_LOCKS and RESULT_CACHE_MAX_SIZE be identical on allinstances?These parameters can be identical on all instances only if these parameter values are set to zero.What two parameters must be set at the time of starting up an ASM instance in a RAC environment?The parameters CLUSTER_DATABASE and INSTANCE_TYPE must be set.Mention the components of Oracle clusterware:-Oracle clusterware is made up of components likevoting diskandOracle ClusterRegistry(OCR).What is a CRS resource?Oracle clusterware is used to manage high-availability operations in a cluster. Anythingthat Oracle Clusterware manages is known as a CRS resource.Some examples of CRSresources are database,an instance,a service,a listener,a VIP address,an applicationprocess etc.What is the use of OCR?Oracle clusterware manages CRS resources based on the configuration information ofCRS resources stored in OCR(Oracle Cluster Registry).How does a Oracle Clusterware manage CRS resources?Oracle clusterware manages CRS resources based on the configuration information ofCRS resources stored in OCR(Oracle Cluster Registry).Name some Oracle clusterware tools and their uses?OIFCFG allocating and deallocating network interfacesOCRCONFIG Command-line tool for managing Oracle Cluster RegistryOCRDUMP Identify the interconnect being usedCVU Cluster verification utility to get status of CRS resourcesWhat are the modes of deleting instances from ORacle Real Application clusterDatabases?We can delete instances using silent mode or interactive mode using DBCA(DatabaseConfiguration Assistant).How do we remove ASM from a Oracle RAC environment?We need to stop and delete the instance in the node first in interactive or silentmode.After that asm can be removed using srvctl tool as follows:srvctl stop asm -n node_namesrvctl remove asm -n node_nameWe can verify if ASM has been removed by issuing the following command:srvctl config asm -n node_nameHow do we verify that an instance has been removed from OCR after deleting aninstance?Issue the following srvctl command:srvctl config database -d database_namecd CRS_HOME/bin./crs_statHow do we verify an existing current backup of OCR?We can verify the current backup of OCR using the following command :Ocrconfig showbackupWhat are the performance views in an Oracle RAC environment?We have v$ views that are instance specific. In addition we have GV$ views called asglobal views that has an INST_ID column of numeric data type.GV$ views obtaininformation from individual V$ views.What are the types of connection load-balancing?There are two types of connection load-balancing:server-side load balancing and clientside load balancing.What is the differnece between server-side and client-side connection loadbalancing?Client-side balancing happens at client side where load balancing is done usinglistener.In case of server-side load balancing listener uses a load-balancing advisory toredirect connections to the instance providing best service.Give the usage of srvctl:-srvctl start instance -d db_name -i inst_name_list [-o start_options]srvctl stop instance-d name -i inst_name_list [-o stop_options]srvctl stop instance -d orcl -i orcl3,orcl4-o immediatesrvctl start database -d name [-o start_options]srvctl stop database -dname[-o stop_options]srvctl start database -d orcl -o mount

SINGLE CLIENT ACCESS NAME (SCAN) ARCHITECTURESingle Client Access Name (SCAN) iss a new Oracle Real Application Clusters (RAC) 11g Release 2 feature that provides a single name for clients to access Oracle Databases running in a cluster. Thebenefitis that the clients connect information does not need to change if you add or remove nodes in the cluster. Having a single name to access the cluster allows clients to use the EZConnect client and the simple JDBC thin URL to access any database running in the cluster, independently of which server(s) in the cluster the database is active. SCAN provides load balancing and failover for client connections to the database.The SCAN works as a cluster alias for databases in the cluster.NETWORKREQUIREMENTS FORUSINGSCANThe SCAN is configured during the installation of Oracle Grid Infrastructure that is distributed with Oracle Database 11g Release2. Oracle Grid Infrastructure is a single Oracle Home that contains Oracle Clusterware and Oracle Automatic Storage Management. You must install Oracle Grid Infrastructure first in order to use Oracle RAC 11g Release 2. During the interview phase of the Oracle Grid Infrastructure installation, you will be prompted to provide a SCAN name. There are 2 options for defining the SCAN:1. Define the SCAN in your corporate DNS (Domain Name Service)2. Use the Grid Naming Service (GNS)USINGOPTION1 DEFINE THESCANIN YOUR CORPORATEDNSIf you choose Option 1, you must ask your network administrator to create a single name that resolves to 3 IP addresses using a round-robin algorithm. Three IP addresses are recommended considering load balancing and high availability requirements regardless of the number of servers in the cluster. The IP addresses must be on the same subnet as your public network in the cluster. The name must be 15 characters or less in length, not including the domain, and must be resolvable without the domain suffix (for example: sales1-scan must be resolvable as opposed to scan1-scan.example.com). The IPs must not be assigned to a network interface (on the cluster), since Oracle Clusterware will take care of it.You can check the SCAN configuration in DNS using nslookup. If your DNS is set up to provide round-robin access to the IPs resolved by the SCAN entry, then run the nslookup command at least twice to see the round-robin algorithm work. The result should be that each time, the nslookup would return a set of 3 IPs in a different order.Note:If your DNS server does not return a set of 3 IPs as shown in figure 3 or does not round-robin, ask your network administrator to enable such a setup. DNS using a round-robin algorithm on its own does not ensure failover of connections. However, the Oracle Client typically handles this. It is thereforerecommendedthat the minimum version of the client used is the Oracle Database 11g Release 2 client.USINGOPTION2 THEGRIDNAMINGSERVICE(GNS)If you choose option 2, you only need to enter the SCAN during the interview. During the cluster configuration, three IP addresses will be acquired from a DHCP service (using GNS assumes you have a DHCP service available on your public network) to create the SCAN and name resolution for the SCAN will be provided by the GNS1.IF YOU DO NOT HAVE ADNSSERVER AVAILABLE AT INSTALLATION TIMEOracle Universal Installer (OUI) enforces providing a SCAN resolution during the Oracle Grid Infrastructure installation, since the SCAN concept is an essential part during the creation of Oracle RAC 11g Release 2 databases in the cluster. All Oracle Database 11g Release 2 tools used to create a database (e.g. the Database Configuration Assistant (DBCA), or the Network Configuration Assistant (NetCA)) would assume its presence. Hence, OUI will not let you continue with the installation until you have provided a suitable SCAN resolution.However, in order to overcome the installation requirement without setting up a DNS-based SCAN resolution, you can use a hosts-file based workaround. In this case, you would use a typical hosts-file entry to resolve the SCAN to only 1 IP address and one IP address only. It is not possible to simulate the round-robin resolution that the DNS server does using a local host file. The host file look-up the OS performs will only return the first IP address that matches the name. Neither will you be able to do so in one entry (one line in the hosts-file). Thus, you will create only 1 SCAN for the cluster. (Note that you will have to change the hosts-file on all nodes in the cluster for this purpose.) This workaround might also be used when performing an upgrade from former (pre-Oracle Database 11g Release 2) releases. However, it is strongly recommended to enable the SCAN configuration as described under Option 1 or Option 2 above shortly after the upgrade or the initial installation. In order to make the cluster aware of the modified SCAN configuration, delete the entry in the hosts-file and then issue: srvctl modify scan -n as the root user on one node in the cluster. The scan_name provided can be the existing fully qualified name (or a new name), but should be resolved through DNS, having 3 IPs associated with it, as discussed. The remaining reconfiguration is then performed automatically.SCAN CONFIGURATION IN THECLUSTERDuring cluster configuration, several resources are created in the cluster for SCAN. For each of the 3 IP addresses that the SCAN resolves to, a SCAN VIP resource is created and a SCAN Listener is created. The SCAN Listener is dependent on the SCAN VIP and the 3 SCAN VIPs (along with their associated listeners) will be dispersed across the cluster. This means, each pair of resources (SCAN VIP + Listener) will be started on a different server in the cluster, assuming the cluster consists of three or more nodes.In case, a 2-node-cluster is used (for which 3 IPs are still recommended for simplification reasons), one server in the cluster will host two sets of SCAN resources under normal operations. If the node where a SCAN VIP is running fails, the SCAN VIP and its associated listener will failover to another node in the cluster. If by means of such a failure the number of available servers in the cluster becomes less than three, one server would again host two sets of SCAN resources. If a node becomes available in the cluster again, the formerly mentioned dispersion will take effect and relocate one set accordingly.DATABASECONFIGURATION USINGSCANFor Oracle Database 11g Release 2, SCAN is an essential part of the configuration and therefore the REMOTE_LISTENER parameter is set to the SCAN per default, assuming that the database is created using standard Oracle tools (e.g. the formerly mentioned DBCA). This allows the instances to registerwith the SCAN Listeners as remote listeners to provide information on what services are being provided by the instance, the current load, and a recommendation on how many incoming connections should be directed to the instance. In this context, the LOCAL_LISTENER parameter must be considered. The LOCAL_LISTENER parameter should be set to the node-VIP. If you need fully qualified domain names, ensure that LOCAL_LISTENER is set to the fully qualified domain name (e.g. node-VIP.example.com). By default, a node listener is created on each node in the cluster during cluster configuration. With Oracle Grid Infrastructure 11g Release 2 the node listener run out of the Oracle Grid Infrastructure home and listen on the node-VIP using the specified port (default port is 1521). Unlike in former database versions, it is not recommended to set your REMOTE_LISTENER parameter to a server side TNSNAMES alias that resolves the host to the SCAN (HOST=sales1-scan for example) in the address list entry, but use the simplified SCAN:port syntax as shown in figure 5.[oracle@mynode]srvctl config scan_listenerSCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521[oracle@mynode]srvctl config scanSCAN name: sales1-scan, Network: 1/133.22.67.0/255.255.255.0/SCAN VIP name: scan1, IP: /sales1-scan.example.com/133.22.67.192SCAN VIP name: scan2, IP: /sales1-scan.example.com/133.22.67.193SCAN VIP name: scan3, IP: /sales1-scan.example.com/133.22.67.194NAME TYPE VALUE local_listener string (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=133.22.67.111)(PORT=1521))))remote_listener string sales1-scan.example.com:1521Note:if you are using the easy connect naming method, you may need to modify your SQLNET.ORA to ensure that EZCONNECT is in the list when specifying the order of the naming methods used for the client name resolution lookups (the Oracle 11g Release 2 default is NAMES.DIRECTORY_PATH=(tnsnames, ldap, ezconnect))HOW CONNECTION LOAD BALANCING WORKS USINGSCANFor clients connecting using Oracle SQL*Net 11g Release 2, three IP addresses will be received by the client by resolving the SCAN name through DNS as discussed. The client will then go through the list it receives from the DNS and try connecting through one of the IPs received. If the client receives an error, it will try the other addresses before returning an error to the user or application. This is similar to how client connection failover works in previous releases when an address list is provided in the client connection string.When a SCAN Listener receives a connection request, the SCAN Listener will check for the least loaded instance providing the requested service. It will then re-direct the connection request to the local listener on the node where the least loaded instance is running. Subsequently, the client will be given the address of the local listener. The local listener will finally create the connection to the database instance.Note:This example assumes an Oracle 11g R2 client using a default TNSNAMES. ORA:ORCLservice =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = sales1-scan.example.com)(PORT = 1521))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = MyORCLservice)))VERSION AND BACKWARD COMPATIBILITYThe successful use of SCAN to connect to an Oracle RAC database in the cluster depends on the ability of the client to understand and use the SCAN as well as on the correct configuration of the REMOTE_LISTENER parameter setting in the database. If the version of the Oracle Client connecting to the database as well as the Oracle Database version used are both Oracle Database 11g Release 2 and the default configuration is used as described in this paper, no changes to the system are typically required.The same holds true, if the Oracle Client version and the version of the Oracle Database that this client is connecting to are both pre-11g Release 2 version (e.g. Oracle Database 11g Release 1 or Oracle Database 10g Release 2, or older). In this case, the pre-11g Release 2 client would use a TNS connect descriptor that resolves to the node-VIPs of the cluster, while the Oracle pre-11g Release 2 database would still use a REMOTE_LISTENER entry pointing to the node-VIPs. The disadvantage of this configuration is that SCAN would not be used and hence the clients are still exposed to changes every time the cluster changes in the backend. Similarly, if an Oracle Database 11g Release 2 is used, but the clients remain on a former version. The solution is to change the Oracle client and / or Oracle Database REMOTE_LISTENER settings accordingly. The following cases need to be considered:Note:If using a pre-11g Release 2 client (Oracle Database 11g Release or Oracle Database 10g Release 2, or older) you will not fully benefit from the advantages of SCAN. Reason: The Oracle Client will not be able to handle a set of three IPs returned by the DNS for SCAN. Hence, it will try to connect to only the first address returned in the list and will more or less ignore the others. If the SCAN Listener listening on this specific IP is not available or the IP itself is not available, the connection will fail. In order to ensure load balancing and connection failover with pre-11g Release 2 clients, you will need to change the TNSNAMES.ora of the client so that it would use 3 address lines, where each address line resolves to one of the SCAN VIPs.Sample TNSNAMES.ora for Oracle Database pre- 11g Release 2 Clientssales.example.com =(DESCRIPTION=(ADDRESS_LIST= (LOAD_BALANCE=on)(FAILOVER=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=133.22.67.192)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=133.22.67.193)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=133.22.67.194)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME= salesservice.example.com)))USINGSCANIN AMAXIMUMAVAILABILITYARCHITECTUREENVIRONMENTIf you have implemented a Maximum Availability Architecture (MAA) environment, in which you use Oracle RAC for both your primary and standby database (in both, your primary and standby site), which are synchronized using Oracle Data Guard, using SCAN provides a simplified TNSNAMES configuration that a client can use to connect to the database independently of whether the primary or standby database is the currently active (primary) database. In order to use this simplified configuration, Oracle Database 11g Release 2 introduces two new SQL*Net parameters that can be used on for connection strings of individual clients. The first parameter isCONNECT_TIMEOUT. It specifies the timeout duration (in seconds) for a client to establish an Oracle Net connection to an Oracle database. This parameter overridesSQLNET.OUTBOUT_CONNECT_TIMEOUTin the SQLNET.ORA. The second parameter isRETRY_COUNTand it specifies the number of times an ADDRESS_LIST is traversed before theconnection attempt is terminated. Using these two parameters, both, the SCAN on the primary site and the standby site, can be used in the client connection strings. Even, if the randomly selected address points to the site that is not currently active, the timeout will allow the connection request to failover before the client has waited unreasonably long (the default timeout depending on the operating system can be as long as 10 minutes).sales.example.com =(DESCRIPTION=(CONNECT_TIMEOUT=10)(RETRY_COUNT=3)(ADDRESS_LIST= (LOAD_BALANCE=on)(FAILOVER=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=sales1-scan)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=sales2-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME= salesservice.example.com)))USINGSCANWITHORACLECONNECTIONMANAGERIf you use Oracle Connection Manager (CMAN) with your Oracle RAC Database, the REMOTE_LISTENER parameter for the Oracle RAC instances should include the CMAN server so that the CMAN server will receive load balancing related information and can therefore load balance connections across the available instances. The easiest way to achieve this would be to add the CMAN-server as an entry to the REMOTE_LISTENER of the databases that clients want to connect to via CMAN as shown in figure 10. Note also that you will have to remove the SCAN from the TNSNAMES connect descriptor of the clients and further configurations will be required for the CMAN server. See the CMAN documentation for more details.SQL>show parameters listenerNAME TYPE VALUE- listener_networks string local_listener string (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=148.87.58.109)(PORT=1521))))remote_listener string stscan3.oracle.com:1521,(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=CMANserver)(PORT=1521))))

Virtual IP(VIP) in RACHow new connection establish in Oracle RAC?For failover configuration we should need to configure our physical ip of host name in listener configuration. Listener process is accepting new connection request and handover user process to server process or dispatcher process in Oracle.Means using listener new connection is being established by Oracle. Once connection gets established there is no need of listener process. If new connection is trying to get session in database and listener is down then what will be happening. User process gets error message and connection fails. Because listener is down in same host or something else problem. But in Oracle RAC database environment database is in sharing mode. Oracle RAC database is shared by all connected nodes. Means more than 1 listeners are running in various nodes.In Oracle RAC database if user process is trying to get connection with some listener and found listener is down or node is down then Oracle RAC automatically transfer this request to another listener on another node. Up to Oracle 9i we use physical IP address in listener configuration. Means if requested connection gets failed then it will be diverting to another node using physical IP address of another surviving node. But during this automatically transfer, connection should need to wait up to get error message of node down or listener down using TCP/IP connection timeout. Means session should need to wait up to getting TCP/IP timeout error dictation. Once error message is received oracle RAC automatically divert this new connection request to another surviving node.Using physical IP address there is biggest gap to get TCP/IP timeout for failover suggestion. Session should need to wait for same timeout. High availability of Oracle RAC depends on this time wasting error message.Why VIP (Virtual IP) needs in Oracle RAC?From Oracle 10g, virtual IP considers to configure listener. Using virtual IP we can save our TCP/IP timeout problem because Oracle notification service maintains communication between each nodes and listeners. Once ONS found any listener down or node down, it will notify another nodes and listeners with same situation. While new connection is trying to establish connection to failure node or listener, virtual IP of failure node automatically divert to surviving node and session will be establishing in another surviving node. This process doesnt wait for TCP/IP timeout event. Due to this new connection gets faster session establishment to another surviving nodes/listener.Characteristic of Virtual IP in Oracle RAC:Virtual IP (VIP) is for fast connection establishment in failover dictation. Still we can use physical IP address in Oracle 10g in listener if we have no worry for failover timing. We can change default TCP/IP timeout using operating system utilities or commands and kept smaller. But taking advantage of VIP (Virtual IP address) in Oracle 10g RAC database is advisable. There is utility also provided to configure virtual IP (vip) with RAC environment called VIPCA. Default path is $ORA_CRS_HOME/bin. During installation of Oracle RAC, it is executed.Advantage of Virtual IP deployment in Oracle RAC:Using VIP configuration, client can be able to get connection fast even fail over of connection request to node. Because vip automatically assign to another surviving node faster and it cant wait for TNS timeout old fashion.Disadvantage of Virtual IP deployment in Oracle RAC:Some more configurations is needed in system for assign virtual IP address to nodes like in /etc/hosts and others. Some misunderstanding or confusion may occur due to multiple IP assigns in same node.Important for VIP configuration:The VIPs should be registered in the DNS. The VIP addresses must be on the same subnet as the public host network addresses. Each Virtual IP (VIP) configured requires an unused and resolvable IP address.Split Brain Syndrome and I/O Fencing (RAC).

Split brain syndrome occurs when the instances in a RAC fails to connect or pingto each other via the private interconnect, Although the servers are physically upand running and the database instances on these servers is also running.

The individual nodes are running fine and can accept user connections and workindependently.

So, in a two node situation both the instances will think that the otherinstance is down because of lack of connection.

The problem which could arise out of this situation is that the sane blockmight get read, updated in these individual instances which cause data integrityissues, because the block changed in one instance will not be lockedand could be overwritten by another instance.

So, when a node fails, the failed node is prevented from accessing allthe shared disk devices and groups. This methodology is called I/O Fencing,Disk Fencing or Failure Fencing.

In a RAC environment the node which first detects the other unreachable nodewill evict it from the cluster to avoid data corruption.

Oracle RAC Q&A -I

1. Why Node Eviction happens on Oracle RAC ?

Oracle Clusterware evicts the node when following condition occur:- Node is not pinging via the network hearbeat- Node is not pinging the Voting Disk- Node is hung or busy and is unable to perform the above two tasks

Most cases the error cause is written to disk. If no error following the Metalink note:ID 559365.1 touse Diagwait option which will gives 10 seconds for the node to write logs to error log file.

#crsctl set css diagwait 13 -force#crsctl get css diagwait#crsctl check crs#crsctl unset css diagwait -f

1a. What is Miscount(MC) in Oracle RAC ?

The Cluster Synchronization Service (CSS) on RAC has Miscount parameter. This value represents maximum time, in seconds, that a network heartbeat can be missed before entering into a cluster reconfiguration to evict the node. The default value is 30 seconds (Linux 60 seconds in 10g, 30 sec in 11g).

2. What is the use of CSS Heartbeat Mechanism in Oracle RAC ?

The CSS of the Oracle Clusterware maintains two heartbeat mechanisms1. The disk heartbeat to the voting device and2. The network heartbeat across the interconnect (This establish and confirm valid node membership in the cluster).

Both of these heartbeat mechanisms have an associated timeout value. The disk heartbeat has an internal i/o timeout interval (DTO Disk TimeOut), in seconds, where an i/o to the voting disk must complete. The misscount parameter (MC), as stated above, is the maximum time, in seconds, that a network heartbeat can be missed. The disk heartbeat i/o timeout interval is directly related to the misscount parameter setting. The Disk TimeOut(DTO) = Miscount(MC) - 15 secconds (some versions are different).

3. What happens if latencies to voting disks are longer ?

If I/O latencies to the voting disk are greater than the default Disk TimeOut (DTO), then the cluster may experince CSS node evictions.

4. What is CSS miscount ?

The CSS miscount represents the maximum seconds the network hearbeat can be missed before entering into cluster reconfiguration and evict the node. The default CSS miscount is 30 seconds. (only for 10g Linux it is 60 secods).

4a. How to change the CSS miscount default value ?

1) Shut down CRS on all but one node. For exact steps use Note 309542.12) Execute crsctl as root to modify the misscount:$ORA_CRS_HOME/bin/crsctl set css misscountwhereis the maximum i/o latency to the voting disk +1 second3) Reboot the node where adjustment was made4) Start all other nodes shutdown in step 1

5. How to start and stop CRS ?

Note: Typically the Oracle clusterware starts up automatically during startup.

10gR1 and R2------------cd /etc/init.dinit.crs stopinit.crs start

To disable crs to start during next reboot. It will not bring down running crs.init.crs enableinit.crs disable

10gR2 and higher versions Only------------------------------Start Oracle Clusterwarecrsctl start crs

Stop Oracle Clusterwarecrsctl stop crs

6. How to move regular DB to an ASM disk group ?

The following are the steps involved in moving regular db files to ASM disk

group.

Assume:1. Oracle RAC instance is up already2. DB name to be moved PROD3. RAC db and normal DB both are in same instance.

1. Install and bring up Oracle RAC instance and ASM disk group.2. Comment control file location in the DB you want to move and add ASM

disk name for control_file.ex. control_file="+DATA_GRP"

3. SQL> startup nomountSQL> Show parameter => Control_files will show new disk grp

4. Use RMAN to move control file from regular disk to ASM using restore

command.rmanrman> connect targetrman> restore controlfile from '/u01/oracle/PROD/cntrl01.ctl';

5. Verify using asmcmdasmcmd> cd DSK_GRP/DATA_GRP/PRODasmcmd> ls => you can see new controlfile under PROD directory.

6. Now mount the DBsqlplus "/as sysdba"sql> alter database mount;

7. Now use RMAN to move the data files.rmanconnect target (connected to PROD)rman> backup as copy database format '+DATA_GRP';Note: you can use asmcmd to monitor the data file movements to ASM.

8. rman> swith database to copy;

9. sqlplus "/as sysdba" ; alter database open;

10. select * from v$datafile;select * from v$tempfile;select * from v$controlfile;select * from v$logfile;

11. sql> alter database drop logfile '/u01/.../redo01.log';alter database add logfile '+DATA_GRP';

Note: Repeat same step for all log files except current used logfile.select * from v$log to find which one is current

12. alter system switch logfile;drop the first one which was being used.

13. Now vi init.ora and put full path for controlfile for DB to start

properly.*.control_file="+DATA_GRP/PROD/controlfile/current.333.433.3333"

13. vi init.ora => Change location of arc to ASM*.log_archive_dest_1='LOCATION=+DATA_GRP/PROD' => if you omit PROD it

will not work properly.

14. alter system switch logfile; => now the new arc will go to ASM.

15. END

------------------

3. What is a NIC card and HBA card.

Oracle RAC requires a NIC or HBA card which enables the computer to talk to networkor to a storage subsystem.

There are diffrent speeds of BHA card: 1Gbit/S, 2GBit/S, 4, 8, 10, 20 GBits/s

HBA has a unique World Wide Name (WWN),which is similar to an Ethernet MAC address in that it uses an OrganizationallyUnique Identifier (OUI) assigned by the IEEE.

4. What is a TPS.

http://www.dba-oracle.com/m_transactions_per_second.htm

==

5. What is the use of crs_getperm command ?Used to get permission information.

crs_getpermUsage: crs_getperm resource_name [-u user|-g group] [-q]

crs_getperm ora.dudb.dudb1.instName: ora.dudb.dudb1.instowner:oracle:rwx,pgrp:oinstall:rwx,other::r--,

==

6. what is the use of crs_profile ?

Used to create, validate, delete and update a profile for RAC.

==

7. Where will you check for RAC log files?

==

8. What is OCFS ?

===

8a. What is OCR ?

- Is a binary file used to store configuration information ans statusinformation. Its like windows registry.- Maintained by CRS Daemon.- Can be mirrered in 10R2.- Include config information of DB, ASM, Services, VIP, Listener and etc

8b. What is Voting Disk?-Used stores node membership information.-Used by CSS during split-brain synarios. (two nodes trying to do same task).-Used to determine RAC instance membership.-

8c. What is VIP?- All application connect using VIP-

==

9. What is Oracle ClusterWare ?

a. It is franework which contains application modeling logic.Invokes application aware agents.Performs resource recovery. Whan a node goes down, Clusterware frameworkrecovers the application by relocationg the resources to a live node.

This can be done for non Oracle applications as well. For ex. xclock.

b. Clusterware also hosts OCR cache.

The Oracle Clusterware requires two clusterware components:a voting disk to record node membership information and theOracle Cluster Registry/Repository (OCR) to record cluster configuration information.The voting disk and the OCR must reside on shared storage.

==

10. What is a resource ?

A resource is a Oracle Clusterware manager application.'Profile attributes' for a resource is stored in Oracle Cluster Registry.

11. What is OCR?Oracle Cluster Registry or OCR is a component of Oracle Clusterware Framework.It stores profile attibute information.Oracle RAC consists of series of resources.

Other applications can also be treated as a resource.OCR contains information pertaining to instance-to-node mappingYou cant have more than two OCRs.

11. How to register a resource ?

a. Use crs_profile to create .CAP file with configuration details.b. use crs_register to read .CAP file and update the OCR.c. Resources can have dependencies. It will start in order and failover as a single unit.

12. What does crs_start / crs_stop does ?

Reads config info from OCR and calls agent with command 'start'.The agents (can be user written) actully stops the resource.

crs_start => read OCR config info => calls 'Control Agent' with command start. => Control agent stops the resource.

crs_stop => read OCR config info => call 'Control agent' with 'stop' => control agent stops app.

==

13. Question: Using the crs_start command to start/stop services.

As per Oracle documentation.....2) Oracle Database Oracle Clusterware and Oracle Real ApplicationClusters Administration and Deployment Guide10g Release 2 (10.2)Part Number B14197-03Page 260 says

"Note: Do not use the Oracle Clusterware commands crs_register, crs_profile,crs_start or crs_stop on resources with names beginning with the prefix "ora"unless either Oracle Support asks you to, or unless Oracle has certified you asdescribed in http://metalink.oracle.com. Server Control (SRVCTL) is the correctutility to use on Oracle resources. You can create resources that depend onresources that Oracle has defined. You can also use the Oracle Clusterware commands toinspect the configuration and status."

==

14. What is the difference between Oracle Clusterware and CRS ?

Oracle Clusterware is formerly known as Cluster Ready Services (CRS). It is an integrated cluster management solution that enables you to link multiple servers so that they function as a single system or cluster. The Oracle Clusterware simplifies the infrastructure required for RAC because it is integrated with the Oracle Database. In addition, Oracle Clusterware is also available for use with single-instance databases and applications that you deploy on clusters

Note: The commands stating with crs_ are still valid and same.

==

15. What is 'Split brain Syndrome' ?

The Oracle Clusterware manages node membership and 'prevents' 'split brain syndrome' in which two or more instances attempt to control the database. This can occur in cases where there is a break in communication between nodes through the interconnect.

16. What is Oracle recomendation for interconnect ?

Oracle recommends that you configure a redundant interconnect to prevent the interconnect from being a single point of failure.

Oracle also recommends that you use User Datagram Protocol (UDP) on a Gigabit Ethernet for your cluster interconnect.Crossover cables are not supported for use with Oracle Clusterware or RAC databases.

17. List the commands used to manage RAC ?

crs_profile

crs_register

crs_relocate

crs_getperm

crs_setperm

crs_stat

srvctl

--crsctl

$crsctl check crsCSS appears healthyCRS appears healthyEVM appears healthy

$crsctl check cssdCSS appears healthy

$crsctl check evmdEVM appears healthy

crsctl add css votedisk- adds a new voting diskcrsctl delete css votedisk- removes a voting diskcrsctl enable crs - enables startup for all CRS daemonscrsctl disable crs - disables startup for all CRS daemonscrsctl start crs - starts all CRS daemons.crsctl stop crs - stops all CRS daemons. Stops CRS resources

$ crsctl query crs activeversionCRS active version on the cluster is [10.2.0.1.0]

--ocrdump

--ocrconfig

dsudsbs1:oracle$ ocrconfig -showbackupdsudsbs1 2009/11/25 19:42:50 /opt/crs/oracle/product/10.2/app/cdata/crsdsudsbs1 2009/11/25 15:42:49 /opt/crs/oracle/product/10.2/app/cdata/crsdsudsbs1 2009/11/25 11:42:49 /opt/crs/oracle/product/10.2/app/cdata/crsdsudsbs1 2009/11/24 19:42:47 /opt/crs/oracle/product/10.2/app/cdata/crsdsudsbs1 2009/11/12 19:42:12 /opt/crs/oracle/product/10.2/app/cdata/crs

ocrconfig -repair ocrocrconfig -replaceocrconfig -export/-importocrconfig -upgrade--ocrcheck - no param needed.

$ocrcheckStatus of Oracle Cluster Registry is as follows :Version : 2Total space (kbytes) : 0Used space (kbytes) : 4588Available space (kbytes) : 4294962708ID : 1014742862Device/File Name : /dev/rdsk/c5t0d3Device/File integrity check succeededDevice/File Name : /dev/rdsk/c5t0d4Device/File integrity check succeeded

Cluster registry integrity check succeeded--

18. How to take backup of Voting disk ?.

Use dd command to backup.

dd if=voting_disk_file of=backup_vt_file

In windoes use ocopy.

To add and remove voting disks use crsctl:

crsctl add css voting_disk_path

crsctl delete css voting_disk_path

if your cluster is down use force option

crsctl add css voting_disk_path -force

==

19. How to find location of voting disk ?

option 1:

crsctl query css votedisk0. 0 /dev/rdsk/c5t0d51. 0 /dev/rdsk/c5t0d62. 0 /dev/rdsk/c5t0d7

located 3 votedisk(s).

option 2:take a ocrdump.ocrdump -stdout -keyname SYSTEM.css.diskfile

20. What is CRS?

21. What are the log file locations for RAC ?

cd $ORACLE_HOME/log//client-- when you execute command like oifcfg, ocrconfig and etc-- a log file will be created here.

cd $ORACLE_HOME/log//crsd

cd $ORACLE_HOME/log//racg

==22. How to backup OCR ?

Oracle Cluster Registry (OCR) and recovering it. Oracle Clusterware automatically creates OCR backups every four hours and it always retains the last three backup copies of the OCR. The CRSD process that creates the backups also creates and retains an OCR backup for each full day and then at the end of a week a complete backup for the week. So there is a robust backup taking place in the background. And you guessed it right; you cannot alter the backup frequencies. This is meant to protect you, the DBA, so that you can copy these generated backup files at least once daily to a different device from where the primary OCR resides. These files are located at %CRS_home/cdata/my_cluster.

==23. How to find location of OCR?

ocrcheckStatus of Oracle Cluster Registry is as follows :Version : 2Total space (kbytes) : 0Used space (kbytes) : 4588Available space (kbytes) : 4294962708ID : 1014742862Device/File Name : /dev/rdsk/c5t0d3Device/File integrity check succeededDevice/File Name : /dev/rdsk/c5t0d4Device/File integrity check succeeded

Cluster registry integrity check succeeded

24. How to restore OCR file if currupted ?

Do the following to restore our OCR on Unix/Linux Systems.

To show the backups, type the commands ocrconfigshowbackupCheck the contents by doing ocrdump -backupfile my_fileGo to bin and stop the CRS. crs stop on all nodes.Perform the restore ocrconfigrestore my_fileRestart the nodes crs startWe have spoken and seen the CVU (Cluster Verification Utility) play a crucial role during installation in our RAC on VMware Series. Check the OCRs integrity. Get a verbose output of all of the nodes by doing this: cluvfy comp ocr n all -verbose

==

25. How to compare all nodes with cluvfy?

cluvfy comp ocr -n all [-verbose]

oracle$ cluvfy comp ocr -n all

Verifying OCR integrityChecking OCR integrity...

Checking the absence of a non-clustered configuration...All nodes free of non-clustered, local-only configurations.

Uniqueness check for OCR device passed.

Checking the version of OCR...OCR of correct Version "2" exists.

Checking data integrity of OCR...Data integrity check for OCR passed.

OCR integrity check passed.

Verification of OCR integrity was successful.dsudsbs1:oracle$==

26. How to manage ASM?

Administering ASM Instances with SRVCTL in RACUse the following command to add configuration information to an existing ASM instance:

srvctl add asm -n mynode_name -i myasm_instance_name -o myoracle_homeIf, however, you choose not to add the I option, then the changes are propogated throughout the entire ASM instance pool.

To remove an ASM instance, use the following syntax:

srvctl remove asm -n mynode_name [-i myasm_instance_name]In order to enable an ASM instance, use the following syntax:

srvctl enable asm -n mynode_name [-i ] myasm_instance_nameIn order to disable an ASM instance use the following syntax:

srvctl disable asm -n mynode_name [-i myasm_instance_name]Note that you can also use the SRVCTL utility to start, stop, and get the status of an ASM instance. See the examples below.

To start an ASM instance, do the following:

srvctl start asm -n mynode_name [-i myasm_instance_name] [-o start_options] [-c| -q]To stop an ASM instance, type the following syntax:

srvctl stop asm -n mynode_name [-i myasm_instance_name] [-o stop_options] [-c| -q]To list the configuration of an ASM instance do the following:

srvctl config asm -n mynode_nameTo get the status of an ASM instance, see the following syntax:

srvctl status asm -n mynode_name

==

27. How to start and stop RAC ?

Starting Up and Shutting Down with SRVCTLWe have covered SRVCTL before, so we'll do a quick syntax check here, to start an instance:

srvctl start instance -d mydb -i "myinstance_list" [-o start_options] [-c connect_str | -q]

To stop, do the following:

srvctl stop instance -d mydb -i " myinstance_list" [-o stop_options] [-c connect_str | -q]

To start and stop the entire RAC cluster database, meaning all of the instances, you will do the following from your SRVCTL in the command line:

srvctl start database -d mydb [-o start_options] [-c connect_str | -q]

srvctl stop database -d mydb [-o stop_options] [-c connect_str | -q]

There are several options and we will look at all of them in upcoming articles in RAC administration

==

28 . How to take ocrdump?

login as root.type ocrdump. It create a file as OCRDUMPFILE

vi to see the ocrdump.

if you type again you will get error:# ocrdumpPROT-303: Dump file already exists [OCRDUMPFILE]

==

Rac ArchitectureA cluster is a set of 2 or more machines (nodes) that share or coordinate resources to performthe same task.A RAC database is 2 or more instances running on a set of clustered nodes, with all instancesaccessing a shared set of database files.Depending on the O/S platform, a RAC database may be deployed on a cluster that uses vendor clusterware plus Oracle's own clusterware (Cluster Ready Services), or on a cluster that solely uses Oracle's own clusterware.Thus, every RAC sits on a cluster that is running Cluster Ready Services. srvctl is the primary tool DBAs use to configure CRS for their RAC database and processes.

Cluster Ready Services and the OCR

Cluster Ready Services, or CRS, is a new feature for 10g RAC. Essentially, it is Oracle's own clusterware. On most platforms, Oracle supports vendor clusterware; in these cases, CRS interoperates with the vendor clusterware, providing high availability support and service and workload management. On Linux and Windows clusters, CRS serves as the sole clusterware. In all cases, CRS provides a standard cluster interface that is consistent across all platforms.

CRS consists of four processes (crsd, occsd, evmd, and evmlogger) and two disks: the Oracle Cluster Registry (OCR), and the voting disk.

CRS manages the following resources:

The ASM instances on each nodeDatabasesThe instances on each nodeOracle Services on each nodeThe cluster nodes themselves, including the following processes, or "nodeapps":VIPGSDThe listenerThe ONS daemonCRS stores information about these resources in the OCR. If the information in the OCR for oneof these resources becomes damaged or inconsistent, then CRS is no longer able to manage that resource. Fortunately, the OCR automatically backs itself up regularly and frequentlyInteracting with CRS and the OCR: srvctl

srvctl is the tool Oracle recommends that DBAs use to interact with CRS and the cluster registry.Oracle does provide several tools to interface with the cluster registry and CRS more directly,at a lower level, but these tools are deliberately undocumented and intended only for use by Oracle Support. srvctl, in contrast, is well documented and easy to use. Using other tools to modifythe OCR or manage CRS without the assistance of Oracle Support runs the risk of damaging the OCR.

Oracle10g RAC Service Architecture

Overview of Real Application Cluster Ready Services, Nodeapps, and User Defined ServicesOverview

Service ArchitectureCluster Ready Services (CRS)NASM ServodeappsUser defined Services icesInternally Managed ServicesMonitoring Services

RAC Service Architecture

Oracle 10g RAC features a service based architectureThis is an improvement over 9i RAC in several waysIncreased flexibilityIncreased manageabilityImprovements in High AvailabilityEnables 10g Grid Deployment

RAC Services and High Availability

Oracle Services facilitate high availability of databases and related applicationsIf key database resources become unavailable (network, storage, etc.):Instances and Services will be relocated to another nodeThe failed node will be rebootedBy default, after any server boot-up, Oracle attempts to restart all services on the node

Cluster Ready Services

Manage the RAC ClusterSeveral Different ServicesOracleCRSServiceOracle CSServiceOracleEVMServiceOraFenceServiceRequired for RAC installationInstalled in its own CRS_HOME

CRS Basics

Used to manage RACOnly one set of CRS Daemons per systemMultiple instances share the same CRSCRS runs as both root and Oracle usersCRS must be running before RAC can start

CRS Management

Started automaticallyCan stop and start manuallyStart the OracleCRSServiceStop the OracleCRSServiceUses the voting disk and OCR (Oracle Cluster Repository)Requires 3 network addressesPublicPrivateVirtual Public

CRS Services

OracleCRSServiceCluster Ready Services DaemonOracleCSServiceOracle Cluster Synchronization Service DaemonOracleEVMServiceEvent Manager DaemonOraFenceServiceProcess MonitorCluster Ready Services Daemon

OracleCRSService

Runs as Administrator userAutomatically restartedManages Application ResourcesStarts, stops and fails-over application resourcesMaintains the OCR (Oracle Cluster Repository)Keeps state information in the OCR

Oracle Cluster Synchronization Service DaemonOracleCSService

Runs as Administrator userMaintains the heartbeat (failure causes system reboot)Provides Node MembershipGroup AccessBasic Cluster LockingCan integrate with 3rd party clustering products or run standaloneOracleCSService also works with non-RAC systems

Event Manager DaemonOracleEVMService

Runs as Administrator userRestarts on failureGenerates EventsStarts the racgevt thread to invokes Server Callouts

Process MonitorOraFenceService

Runs as Administrator userLocked in memory to monitor the clusterProvides I/O fencingOraFenceService periodically monitors cluster status, and can reboot the node if aproblem is detected An OraFenceService failure results inOracle Clusterware restarting the node

RACG

RACG is a behind-the-scenes process (or thread) that extends clusterware tosupport Oracle-specific requirements and complex resources.Runs server callout scripts when FAN events occur.Runs as processes (or threads), not as a service (racgmain.exe, racgimon.exe)

Cluster Ready Services Management

Log FilesOracleCRSService%ORA_CRS_HOME%\log\hostname\crsdOracle Cluster Registry (OCR)%ORA_CRS_HOME%log\hostname\ocrOracleEVMService%ORA_CRS_HOME%\log\hostname\evmdOracleCSService%ORA_CRS_HOME%log\hostname\cssdRACG%ORA_CRS_HOME%log\hostname\racg

Nodeapp Services

Nodeapps are a standard set of Oracle application services that are automaticallylaunched for RACVirtual IP (VIP)Oracle Net ListenerGlobal Services Daemon (GSD)Oracle Notification Service (ONS)Nodeapp services run on each nodeCan be relocated to other nodes through the virtual IP

VIP (Virtual IP)

Creates a virtual IP address used by the ListenerThe virtual IP address fails over between nodesMultiple virtual IP addresses can exist on the same system (during failover)Independent of the Oracle InstancePotential Problem if more than one database per node

Global Services Daemon (GSD)

The daemon which executes SRVCTL commandsGSD receives requests from SRVCTL to execute administrative tasks, such as startup or shutdownThe command is executed locally on each node, and the results are sent back to SRVCTL.The daemon is installed on the nodes by default. It is important that you do not kill this process and it should not be deleted.

Listener

Server-side component of Oracle NetListens for incoming client connection requestsManages the traffic to the server; when a client requests a network session with a server,the listener actually receives the request and brokers the client requestIf the client's information matches the listener's information, then the listener grantsa connection to the server.

Oracle Notification Service (ONS)

The Oracle Notification Service is installed automatically on each RAC node as aNode ApplicationONS starts automatically with each bootONS uses a simple push/subscribe method to publish event messages to all RACnodes with active ONS daemons

ONS and Fast Application Notification

ONS can be configured to run on nodes hosting client or mid-tier applicationsONS is the key component of Fast Application Notification (FAN)Can be utilized to extend RAC high availability and load balancing to mid-tierapplicationsIndependent of True Application FailoverLess reliance on network configuration

User Defined Services

User defined, named services may be created to manage database resources that are associated with application workloadsOne or more database instances may be mapped to a single serviceA database instance may be assigned to one or more servicesThe Automated Workload Repository may be used to monitor Service metrics

User Defined Services and Failover

Services can be defined with preferred and alternate instancesA service may be assigned to start on preferred instancesThe same service may have alternate instances assigned for failoverIf multiple services are assigned for the same database, the preferred and alternate instance assignments may be different for each service

Automatic Storage Management Services

Automatic Storage Management (ASM) is a storage option for creating and managing databasesASM operates like a Logical Volume Manager between the physical storage and the database.A small, automatically managed Oracle database instance is created on each node (if ASM is chosenas a storage option)ASM instances start automatically as Oracle services

Internally Managed Services

When the Global Services Daemon is started as a part of the Node Applications,it in turn launches key internally managed servicesThe Global Cache Service manages Cache Fusion and in-memory data buffersThe Global Enqueue Service manages inter-instance locking and RAC recoveryGCS and GES show up as OS processes or threads,but GSD is the only service that can be externally controlledGCS and GES together manage a set of virtual tables in memory,called the Global Resouce DirectoryGlobal Cache Service (GCS)

The controlling process that implements Cache Fusion.Manages the status and transfer of data blocks across the buffer caches of all instances.Tightly integrated with the buffer cache manager to enable fast lookup of resourceinformation in the Global Resource Directory.Maintains the block mode for blocks in the global role.Employs various background processes (or threads) such as theGlobal Cache Service Processes (LMSn) and Global Enqueue Service Daemon (LMD).

Global Enqueue Service Monitor (LMON)

Background process that monitors the entire cluster to manage global resources.Manages instance and process expirations and recovery for GCS.Handles the part of recovery associated with global resources

Global Resource Directory

The data structures associated with global resources. It is distributed across allinstances in a cluster.Global Cache Service and Global Enqueue Service maintain the Global ResourceDirectory to record information about resources and enqueues held globally.The Global Resource Directory resides in memory and is distributed throughout thecluster to all nodes. In this distributed architecture, each node participates inmanaging global resources and manages a portion of the Global Resource Directory.

Monitoring RAC Services

%ORA_CRS_HOME%\bin\crs_statNAME=ora.rac1.gsdTYPE=applicationTARGET=ONLINESTATE=ONLINE

NAME=ora.rac1.oemTYPE=applicationTARGET=ONLINESTATE=ONLINE

NAME=ora.rac1.onsTYPE=applicationTARGET=ONLINESTATE=ONLINEMonitoring RAC Services

Creating a tabular report:%ORA_CRS_HOME%\bin\crs_stat -tName Type Target State Host

--------------------------------------------------------------------------------------------------

ora.rac1.gsd application ONLINE ONLINE rac1

ora.rac1.oem application ONLINE ONLINE rac1

ora.rac1.ons application ONLINE ONLINE rac1

ora.rac1.vip application ONLINE ONLINE rac1

ora.rac2.gsd application ONLINE ONLINE rac2

ora.rac2.oem application ONLINE ONLINE rac2

ora.rac2.ons application ONLINE ONLINE rac2

ora.rac2.vip application ONLINE ONLINE rac2

Review

What advantages does a service based architecture offer?What four services comprise Cluster Ready Services?Nodeapps consists of which four applications?True or False: a database instance may be assigned to multiple services

Summary

Service ArchitectureCluster Ready Services (CRS)OracleCRSServiceOracleCSServiceOracleEVMServiceOraFenceServiceNodeappsVIPListenerGSDONSUser defined ServicesASM ServicesInternally managed servicesGlobal Cache ServiceGlobal Enqueue ServiceGlobal Resource DirectoryMonitoring Services

Oracle RAC Interview Question With AnswerWhat is RAC?

RAC stands for Real Application cluster. It is a clustering solution from Oracle Corporation that ensures high availability of databases by providing instance failover, media failover features.

What is RAC and how is it different from non RAC databases?

RAC stands for Real Application Cluster, you have n number of instances running in their own separate nodes and based on the shared storage. Cluster is the key component and is a collection of servers operations as one unit. RAC is the best solution for high performance and high availably. Non RAC databases has single point of failure in case of hardware failure or server crash.

Give the usage of srvctl :

srvctl start instance -d db_name -i "inst_name_list" [-o start_options]

srvctl stop instance -d name -i "inst_name_list" [-o stop_options]

srvctl stop instance -d orcl -i "orcl3,orcl4" -o immediate

srvctl start database -d name [-o start_options]

srvctl stop database -d name [-o stop_options]

srvctl start database -d orcl -o mount

Mention the Oracle RAC software components :Oracle RAC is composed of two or more database instances. They are composed of Memory structures and background processes same as the single instance database.Oracle RAC instances use two processes GES(Global Enqueue Service), GCS(Global Cache Service) that enable cache fusion.Oracle RAC instances are composed of following background processes:ACMSAtomic Controlfile to Memory Service (ACMS)GTX0-jGlobal Transaction ProcessLMONGlobal Enqueue Service MonitorLMDGlobal Enqueue Service DaemonLMSGlobal Cache Service ProcessLCK0Instance Enqueue ProcessRMSnOracle RAC Management Processes (RMSn)RSMNRemote Slave Monitor

What is GRD?GRD stands for Global Resource Directory. The GES and GCS maintains records of the statuses of each datafile and each cahed block using global resource directory.This process is referred to as cache fusion and helps in data integrity.

What are the different network components are in 10g RAC?

public, private, and vip componentsPrivate interfaces is for intra node communication. VIP is all about availability of application. When a node fails then the VIP component fail over to some other node, this is the reason that all applications should based on vip components means tns entries should have vip entry in the host list

Give Details on ACMS:

ACMS stands for Atomic Controlfile Memory Service.In an Oracle RAC environment ACMS is an agent that ensures a distributed SGA memory update(ie)SGA updates are globally committed on success or globally aborted in event of a failure.

What is Cache Fusion?

Cache fusion is the mechanism to transfer the data block from memory to memory of one node to the other.If two nodes require the same block for query or update, the block must be transfered from the cache of one node to the other. RAC system must equipped with low-latency and high speed inter-connect to make it happen.

Give Details on Cache Fusion:Oracle RAC is composed of two or more instances. When a block of data is read from datafile by an instance within the cluster and another instance is in need of the same block,it is easy to get the block image from the insatnce which has the block in its SGA rather than reading from the disk. To enable inter instance communication Oracle RAC makes use of interconnects. The Global Enqueue Service(GES) monitors and Instance enqueue process manages the cahce fusion.

Cache Fusion is essentially a memory-to-memory transfer of data between the nodes in the RAC environment. Before Cache Fusion, a node was required to write some of the data to disk before it could be transferred to the next node in the cluster. Cache Fusion does a straight memory-to-memory transfer. In addition, each node's SGA has a map of what data is contained in the other node's data caches.

The performance improvement is phenomenal. Oracle leverages the vendor's high speed interconnects between the nodes to achieve the cache-to-cache data transfers. Before Cache Fusion, when you added a node to the cluster to increase performance of the application, it didn't always provide you with the performance improvement that you hoped for. With Cache Fusion, you can easily cost justify the addition of another node into a RAC cluster to increase the performance of the application running on it. Oracle sales pitches describe it as 'near linear horizontal scalability'.

What are the major RAC wait events?

In a RAC environment the buffer cache is global across all instances in the cluster and hence the processing differs.The most common wait events related to this are gc cr request and gc buffer busy

GC CR request :the time it takes to retrieve the data from the remote cache

Reason: RAC Traffic Using Slow Connection or Inefficient queries (poorly tuned queries will increase the amount of data blocks requested by an Oracle session. The more blocks requested typically means the more often a block will need to be read from a remote instance via the interconnect.)

GC BUFFER BUSY: It is the time the remote instance locally spends accessing the requested data block.

Give details on GTX0-j :The process provides transparent support for XA global transactions in a RAC environment.The database autotunes the number of these processes based on the workload of XA global transactions.

Give details on LMON:This process monitors global enques and resources across the cluster and performs global enqueue recovery operations.This is called as Global Enqueue Service Monitor.

Give details on LMD:This process is called as global enqueue service daemon. This process manages incoming remote resource requests within each instance.

Give details on LMS:This process is called as Global Cache service process.This process maintains statuses of datafiles and each cahed block by recording information in a Global Resource Dectory(GRD).This process also controls the flow of messages to remote instances and manages global data block access and transmits block images between the buffer caches of different instances.This processing is a part of cache fusion feature.

Give details on LCK0:This process is called as Instance enqueue process.This process manages non-cache fusion resource requests such as libry and row cache requests.

Give details on RMSn:This process is called as Oracle RAC management process.These pocesses perform managability tasks for Oracle RAC.Tasks include creation of resources related Oracle RAC when new instances are added to the cluster.

Give details on RSMN:This process is called as Remote Slave Monitor.This process manages background slave process creation andd communication on remote instances. This is a background slave process.This process performs tasks on behalf of a co-ordinating process running in another instance.

What components in RAC must reside in shared storage?All datafiles, controlfiles, SPFIles, redo log files must reside on cluster-aware shred storage.

What is the significance of using cluster-aware shared storage in an Oracle RAC environment?All instances of an Oracle RAC can access all the datafiles,control files, SPFILE's, redolog files when these files are hosted out of cluster-aware shared storage which are group of shared disks.

Give few examples for solutions that support cluster storage:ASM(automatic storage management),raw disk devices,network file system(NFS), OCFS2 and OCFS(Oracle Cluster Fie systems).

What is an interconnect network?An interconnect network is a private network that connects all of the servers in a cluster. The interconnect network uses a switch/multiple switches that only the nodes in the cluster can access.

How can we configure the cluster interconnect?Configure User Datagram Protocol(UDP) on Gigabit ethernet for cluster interconnect.On unix and linux systems we use UDP and RDS(Reliable data socket) protocols to be used by Oracle Clusterware.Windows clusters use the TCP protocol.

Can we use crossover cables with Oracle Clusterware interconnects?No, crossover cables are not supported with Oracle Clusterware intercnects.

What is the use of cluster interconnect?Cluster interconnect is used by the Cache fusion for inter instance communication.

How do users connect to database in an Oracle RAC environment?Users can access a RAC database using a client/server configuration or through one or more middle tiers ,with or without connection pooling.Users can use oracle services feature to connect to database.

What is the use of a service in Oracle RAC environment?Applications should use the services feature to connect to the Oracle database.Services enable us to define rules and characteristics to control how users and applications connect to database instances.

What are the characteristics controlled by Oracle services feature?The charateristics include a unique name, workload balancing and failover options,and high availability characteristics.

What enables the load balancing of applications in RAC?Oracle Net Services enable the load balancing of application connections across all of the instances in an Oracle RAC database.

What is a virtual IP address or VIP?A virtl IP address or VIP is an alternate IP address that the client connectins use instead of the standard public IP address. To configureVIP address, we need to reserve a spare IP address for each node, and the IP addresses must use the same subnet as the public network.

What is the use of VIP?If a node fails, then the node's VIP address fails over to another node on which the VIP address can accept TCP connections but it cannot accept Oracle connections.

Give situations under which VIP address failover happens:VIP addresses failover happens when the node on which the VIP address runs fails, all interfaces for the VIP address fails, all interfaces for the VIP address are disconnected from the network.

What is the significance of VIP address failover?When a VIP address failover happens, Clients that attempt to connect to the VIP address receive a rapid connection refused error .They don't have to wait for TCP connection timeout messages.

What are the administrative tools used for Oracle RAC environments?Oracle RAC cluster can be administered as a single image using OEM(Enterprise Manager),SQL*PLUS,Servercontrol(SRVCTL),clusterver ificationutility(cvu),DBCA,NETCA

How do we verify that RAC instances are running?Issue the following query from any one node connecting through SQL*PLUS.$connect sys/sys as sysdbaSQL>select * from V$ACTIVE_INSTANCES;The query gives the instance number under INST_NUMBER column,host_:instancename under INST_NAME column.

What is FAN?Fast application Notification as it abbreviates to FAN relates to the events related to instances,services and nodes.This is a notification mechanism that Oracle RAc uses to notify other processes about the configuration and service level information that includes service status changes such as,UP or DOWN events.Applications can respond to FAN events and take immediate action.

Where can we apply FAN UP and DOWN events?FAN UP and FAN DOWN events can be applied to instances,services and nodes.State the use of FAN events in case of a cluster configuration change?During times of cluster configuration changes,Oracle RAC high availability framework publishes a FAN event immediately when a state change occurs in the cluster.So applications can receive FAN events and react immediately.This prevents applications from polling database and detecting a problem after such a state change.

Why should we have seperate homes for ASM instance?It is a good practice to have ASM home seperate from the database hom(ORACLE_HOME).This helps in upgrading and patching ASM and the Oracle database software independent of each other.Also,we can deinstall the Oracle database software independent of the ASM instance.

What is the advantage of using ASM?Having ASM is the Oracle recommended storage option for RAC databases as the ASM maximizes performance by managing the storage configuration across the disks.ASM does this by distributing the database file across all of the available storage within our cluster database environment.

What is rolling upgrade?It is a new ASM feature from Database 11g.ASM instances in Oracle database 11g release(from 11.1) can be upgraded or patched using rolling upgrade feature. This enables us to patch or upgrade ASM nodes in a clustered environment without affecting database availability.During a rolling upgrade we can maintain a functional cluster while one or more of the nodes in the cluster are running in different software versions.

Can rolling upgrade be used to upgrade from 10g to 11g database?No,it can be used only for Oracle database 11g releases(from 11.1).

State the initialization parameters that must have same value for every instance in an Oracle RAC database:Some initialization parameters are critical at the database creation time and must have same values.Their value must be specified in SPFILE or PFILE for every instance.The list of parameters that must be identical on every instance are given below:ACTIVE_INSTANCE_COUNTARCHIVE_LAG_TARGETCOMPATIBLECLUSTER_DATABASECLUSTER_DATABASE_INSTANCECONTROL_FILESDB_BLOCK_SIZEDB_DOMAINDB_FILESDB_NAMEDB_RECOVERY_FILE_DESTDB_RECOVERY_FILE_DEST_SIZEDB_UNIQUE_NAMEINSTANCE_TYPE (RDBMS or ASM)PARALLEL_MAX_SERVERSREMOTE_LOGIN_passWORD_FILEUNDO_MANAGEMENT

What is ORA-00603: ORACLE server session terminated by fatal error or ORA-29702: error occurred in Cluster Group Service operation?RAC node name was listed in the loopback address...

Can the DML_LOCKS and RESULT_CACHE_MAX_SIZE be identical on all instances?These parameters can be identical on all instances only if these parameter values are set to zero.What two parameters must be set at the time of starting up an ASM instance in a RAC environment?The parameters CLUSTER_DATABASE and INSTANCE_TYPE must be set.

Mention the components of Oracle clusterware:Oracle clusterware is made up of components like voting disk and Oracle Cluster Registry(OCR).

What is a CRS resource?Oracle clusterware is used to manage high-availability operations in a cluster.Anything that Oracle Clusterware manages is known as a CRS resource.Some examples of CRS resources are database,an instance,a service,a listener,a VIP address,an application process etc.

What is the use of OCR?Oracle clusterware manages CRS resources based on the configuration information of CRS resources stored in OCR(Oracle Cluster Registry).

How does a Oracle Clusterware manage CRS resources?Oracle clusterware manages CRS resources based on the configuration information of CRS resources stored in OCR(Oracle Cluster Registry).

Name some Oracle clusterware tools and their uses?OIFCFG - allocating and deallocating network interfacesOCRCONFIG - Command-line tool for managing Oracle Cluster RegistryOCRDUMP - Identify the interconnect being usedCVU - Cluster verification utility to get status of CRS resources

What are the modes of deleting instances from ORacle Real Application cluster Databases?We can delete instances using silent mode or interactive mode using DBCA(Database Configuration Assistant).

How do we remove ASM from a Oracle RAC environment?We need to stop and delete the instance in the node first in interactive or silent mode.After that asm can be removed using srvctl tool as follows:srvctl stop asm -n node_namesrvctl remove asm -n node_nameWe can verify if ASM has been removed by issuing the following command:srvctl config asm -n node_name

How do we verify that an instance has been removed from OCR after deleting an instance?Issue the following srvctl command:srvctl config database -d database_namecd