Apun's Weblog: RAC

Showing posts with label RAC. Show all posts

Thursday, July 21, 2011

Oracle: Rename single/RAC Database

Method 1: Recreate control file with the new DB name, the old tried and tested method.
Method 2: Use new utility called NID
Steps to rename Databases using the new NID utility:
1.) Stop database
[oracle@testing1]~% srvctl status database -d test Instance test1 is running on node testing1 Instance test2 is running on node testing2 srvctl stop database -d test

2.) startup mount the database:
[oracle@testing1]~% sqlplus / as sysdba SQL*Plus: Release 11.2.0.2.0 Production on Thu Jul 21 00:50:44 2011 Copyright (c) 1982, 2010, Oracle. All rights reserved. Connected to an idle instance. SQL> startup mount ORACLE instance started. Total System Global Area 1.2527E+10 bytes Fixed Size 2238104 bytes Variable Size 8120174952 bytes Database Buffers 4294967296 bytes Redo Buffers 109346816 bytes Database mounted.
3.) Exit and run the "nid" utility

nid sys/oracle@test DBNAME=new

[oracle@testing1]~% nid sys/m4gent4 DBNAME=new DBNEWID: Release 11.2.0.2.0 - Production on Thu Jul 21 00:55:12 2011 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. NID-00002: Parse error: LRM-00108: invalid positional parameter value 'sys/m4gent4' Change of database ID failed during validation - database is intact. DBNEWID - Completed with validation errors.

Correct the syntax:
nid target=sys/oracle@test DBNAME=new

[oracle@testing1]~% nid target=sys/m4gent4 DBNAME=new DBNEWID: Release 11.2.0.2.0 - Production on Thu Jul 21 00:56:04 2011 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. Connected to database test (DBID=3952648861) NID-00120: Database should be mounted exclusively Change of database name failed during validation - database is intact. DBNEWID - Completed with validation errors.

Incase of RAC, we need to set cluster_database=FALSE to change DB name. Then stop and mount the DB again.

SQL> alter system set cluster_database=FALSE scope=spfile; System altered. SQL> shutdown immediate; ORA-01109: database not open Database dismounted. ORACLE instance shut down. SQL> startup mount ORACLE instance started. Total System Global Area 1.2527E+10 bytes Fixed Size 2238104 bytes Variable Size 8120174952 bytes Database Buffers 4294967296 bytes Redo Buffers 109346816 bytes Database mounted.

nid target=sys/oracle@test DBNAME=new
[oracle@testing1]~% nid target=sys/m4gent4 DBNAME=new DBNEWID: Release 11.2.0.2.0 - Production on Thu Jul 21 00:59:54 2011 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. Connected to database test (DBID=3952648861) Connected to server version 11.2.0 Control Files in database: +DG_SYS01/test/control01.ctl +DG_DATA01/test/control02.ctl Change database ID and database name test to NEW? (Y/[N]) => Y Proceeding with operation Changing database ID from 3952648861 to 388598347 Changing database name from test to NEW Control File +DG_SYS01/test/control01.ctl - modified Control File +DG_DATA01/test/control02.ctl - modified Datafile +DG_SYS01/test/datafile/system.261.75517911 - dbid changed, wrote new name Datafile +DG_SYS01/test/datafile/sysaux.262.75517911 - dbid changed, wrote new name Datafile +DG_DATA01/test/datafile/undotbs1.258.75518469 - dbid changed, wrote new name Datafile +DG_DATA01/test/datafile/undotbs2.260.75518577 - dbid changed, wrote new name Datafile +DG_DATA01/test/datafile/users.257.75518565 - dbid changed, wrote new name Control File +DG_SYS01/test/control01.ctl - dbid changed, wrote new name Control File +DG_DATA01/test/control02.ctl - dbid changed, wrote new name Instance shut down Database name changed to NEW. Modify parameter file and generate a new password file before restarting. Database ID for database NEW changed to 388598347. All previous backups and archived redo logs for this database are unusable. Database has been shutdown, open database with RESETLOGS option. Succesfully changed database name and ID. DBNEWID - Completed succesfully.

4.) Copy the old inittest1.ora as initnew1.ora, update the DB_NAME, instance_name parameter to reflect the new name.
Then startup mount
SQL> startup mount pfile='/home/oracle/software/dbcreate/test/inittest1.ora' ORACLE instance started. Total System Global Area 1.2527E+10 bytes Fixed Size 2238104 bytes Variable Size 8120174952 bytes Database Buffers 4294967296 bytes Redo Buffers 109346816 bytes Database mounted.

5.) Resetlog open the database.
SQL> ALTER DATABASE OPEN RESETLOGS; Database altered. SQL> select open_mode from v$database; OPEN_MODE -------------------- READ WRITE

6.) Create spfile
SQL> create spfile='+DG_SYS01/NEW/spfileNEW.ora' from pfile='/home/oracle/software/dbcreate/test/inittest1.ora'; File created.
NOTE: For single node DB, steps are over. Step 7 for RAC only.

7.) Add DB to the cluster
[oracle@testing1]~/software/dbcreate/test% srvctl add database -d NEW -o /home/oracle/product/11.2 -r primary -s OPEN -p +DG_SYS01/NEW/spfileNEW.ora [oracle@testing1]~/software/dbcreate/test% srvctl add instance -d NEW -i NEW1 -n testing1 [oracle@testing1]~/software/dbcreate/test% srvctl add instance -d NEW -i NEW2 -n testing2 [oracle@testing1]~/software/dbcreate/test% srvctl config database -d NEW -a Database unique name: NEW Database name: Oracle home: /home/oracle/product/11.2 Oracle user: oracle Spfile: +DG_SYS01/NEW/spfileNEW.ora Domain: Start options: open Stop options: immediate Database role: PRIMARY Management policy: AUTOMATIC Server pools: NEW Database instances: NEW1,NEW2 Disk Groups: Mount point paths: Services: Type: RAC Database is enabled Database is administrator managed [oracle@testing1]~/software/dbcreate/test% vi inittest1.ora [oracle@testing1]~/software/dbcreate/test% srvctl start database -d NEW [oracle@testing1]~/software/dbcreate/test% srvctl status database -d NEW Instance NEW1 is running on node testing1 Instance NEW2 is running on node testing2 INST_ID INSTANCE_NAM HOST_NAME STATUS STARTED VERSION -------- ------------------------- ----------------------------------- ---------- 1 new1 testing1 OPEN 20 Jul 18:25:20 11.2.0.2.0 2 new2 testing2 OPEN 20 Jul 18:25:22 11.2.0.2.0

Saturday, August 28, 2010

Oracle 11gR2 Grid Infrastructure Installation Steps

Step By Step Installation 11gR2 Non ASM
========================================
Connect as root user:
1.) groupadd dba
2.) useradd -d /home/oracle -g dba -G dba oracle
3.) Create directories for oracle grid installation, the directory path should not be same as $ORACLE_BASE. Assuming $ORACLE_BASE=/oracle.
mkdir /oragrid
Make sure the directory as atleast 5GB free space.
4.) Change the owner of the directory to oracle:dba
chown -R oracle:dba /oragrid
5.) RPM requirements (From Metalink note# 880989.1:
1.) binutils-2.17.50.0.6-6.el5 (x86_64) 2.) compat-libstdc++-33-3.2.3-61 (x86_64) <<< both ARCH's are required. See next line. 3.) compat-libstdc++-33-3.2.3-61 (i386) <<< both ARCH's are required. See previous line. 4.) elfutils-libelf-0.125-3.el5 (x86_64) 5.) glibc-2.5-24 (x86_64) <<< both ARCH's are required. See next line. 6.) glibc-2.5-24 (i686) <<< both ARCH's are required. See previous line. 7.) glibc-common-2.5-24 (x86_64) 8.) ksh-20060214-1.7 (x86_64) 9.) libaio-0.3.106-3.2 (x86_64) <<< both ARCH's are required. See next line. 10.) libaio-0.3.106-3.2 (i386) <<< both ARCH's are required. See previous line. 11.) libgcc-4.1.2-42.el5 (i386) <<< both ARCH's are required. See next line. 12.) libgcc-4.1.2-42.el5 (x86_64) <<< both ARCH's are required. See previous line. 13.) libstdc++-4.1.2-42.el5 (x86_64) <<< both ARCH's are required. See next line. 14.) libstdc++-4.1.2-42.el5 (i386) <<< both ARCH's are required. See previous line. 15.) make-3.81-3.el5 (x86_64) 16.) elfutils-libelf-devel-0.125-3.el5.x86_64.rpm a.) requires elfutils-libelf-devel-static-0.125-3.el5.x86_64.rpm as a prerequisite, as listed below. b.) elfutils-libelf-devel and elfutils-libelf-devel-static each depend upon the other. Therefore, they must be installed together, in one (1) "rpm -ivh" command as follows: rpm -ivh elfutils-libelf-devel-0.125-3.el5.x86_64.rpm elfutils-libelf-devel-static-0.125-3.el5.x86_64.rpm 17.) glibc-headers-2.5-24.x86_64.rpm a.) requires kernel-headers-2.6.18-92.el5.x86_64.rpm as a prerequisite, as listed below 18.) glibc-devel-2.5-24.x86_64.rpm <<< both ARCH's are required. See next item. 19.) glibc-devel-2.5-24.i386.rpm <<< both ARCH's are required. See previous item. 20.) gcc-4.1.2-42.el5.x86_64.rpm a.) requires libgomp-4.1.2-42.el5.x86_64.rpm as a prerequisite, as listed below 21.) libstdc++-devel-4.1.2-42.el5.x86_64.rpm 22.) gcc-c++-4.1.2-42.el5.x86_64.rpm 23.) libaio-devel-0.3.106-3.2.x86_64.rpm <<< both ARCH's are required. See next item 24.) libaio-devel-0.3.106-3.2.i386.rpm <<< both ARCH's are required. See previous item. 25.) sysstat-7.0.2-1.el5.x86_64.rpm 26.) unixODBC-2.2.11-7.1.x86_64.rpm <<< both ARCH's are required. See next item 27.) unixODBC-2.2.11-7.1.i386.rpm <<< both ARCH's are required. See previous item. 28.) unixODBC-devel-2.2.11-7.1.x86_64.rpm <<< both ARCH's are required. See next item 29.) unixODBC-devel-2.2.11-7.1.i386.rpm <<< both ARCH's are required. See previous item.
Command to check:
rpm -qa |grep -E "binutils|compat-libstdc++|elfutils-libelf-|glibc-|glibc-common-|ksh-|libaio-|libgcc-|libstdc++|make|gcc|sysstat|unixODBC"
6.) Values for /etc/sysctl.conf (these are suggested, these values should be calculated with respect to the memory/cpu/processes of the server.)
kernel.shmall = physical RAM size / pagesize For most systems kernel.shmmax = 1/2 of physical RAM eg. for 32g ram it should be 17179869184. kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 409200 fs.aio-max-nr = 1048576 net.ipv4.ip_local_port_range = 9000 65500 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576
7.) Make sure u have configured a scan IP, for more details on what is scan IP please refer to: http://www.oracle.com/technetwork/database/clustering/overview/scan-129069.pdf
Make sure your /etc/hosts file has entries or hostname, host IP address, Interconnect name, Interconnect IP address, virtual hostname, virtual host IP address for all RAC nodes and on all RAC nodes.

8a.) Since we are using Network Time Protocol (NTP) for synchronization of time across all the servers in the cluster, a mandatory requirement with 11gR2 is to enable the slewing option by adding ‘-x’ argument in the ntp configuration file as seen below: -
# vi /etc/sysconfig/ntpd
# Drop root to id 'ntp:ntp' by default. OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid" #add -x in the options string. # Set to 'yes' to sync hw clock after successful ntpdate SYNC_HWCLOCK=no # Additional options for ntpdate NTPDATE_OPTIONS=""
Restart ntpd daemon.
[root@node1 oracle]# /etc/init.d/ntpd stop
Shutting down ntpd: [ OK ]
[root@node1 oracle]# /etc/init.d/ntpd start
ntpd: Synchronizing with time server: [ OK ]
Starting ntpd: [ OK ]

8b.) Update /etc/security/limits.conf with:
oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536
9.) Add following line in the /etc/pam.d/login file:
session required pam_limits.so
10.) Add the following lines to /etc/profile:
if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -u 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi
11.) Setup user equivalence between the oracle user of the rac nodes.
12.) Run Cluverify (change directory where you have copied and unzipped the oracle 11gr2 grid software)
#./runcluvfy.sh stage -pre crsinst -n node1,node2 -verbose

--Fix any error you see.

13.) Start ./runInstaller
Screenshots:

Welcome Screen

14.)

Select Advanced Installation

15.)

Select all required languages, I have selected only English

16.)

Specify scan name, cluster name, and scan listener port.

17.)

Provide RAC node information, along with virtual name.

18.)

Review is the the interfaces, subnet mask are correct and select the public/private accordingly.(private is for interconnect).

19.)

Oracle will do some validation checks

20.)

Select ASM/shared file system, depending on what u are using. I am using shared file system.

21.)

Select location for OCR.

22.)

Select location of voting disks.

23.)

Failure Support, I dont plan to use this.

24.)

Select the OS groups.

25.)

Warning message, ignore it

26.)

Specify oracle_base and grid installation folder.

27.)

Specify Oracle Inventory Location.

28.)

Oracle perform pre-checks. I am ignoring swap space issue.

29.)

Final summary page!!

30.)

Ignore this warning

31.)

Shows installation Progress

32.)

Run the scripts

33.) Run /oracle/oraInventory/orainstRoot.sh
[root@node1 oraInventory]# ./orainstRoot.sh Changing permissions of /oracle/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /oracle/oraInventory to dba. The execution of the script is complete.

34.) Run $GRID_HOME/root.sh
[root@node1 11.2]# ./root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /oragrid/product/11.2 Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. 2010-08-27 23:25:19: Parsing the host name 2010-08-27 23:25:19: Checking for super user privileges 2010-08-27 23:25:19: User has super user privileges Using configuration parameter file: /oragrid/product/11.2/crs/install/crsconfig_params Creating trace directory LOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. root wallet root wallet cert root cert export peer wallet profile reader wallet pa wallet peer wallet keys pa wallet keys peer cert request pa cert request peer cert pa cert peer root cert TP profile reader root cert TP pa root cert TP peer pa cert TP pa peer cert TP profile reader pa cert TP profile reader peer cert TP peer user cert pa user cert Adding daemon to inittab CRS-4123: Oracle High Availability Services has been started. ohasd is starting CRS-2672: Attempting to start 'ora.gipcd' on 'node1' CRS-2672: Attempting to start 'ora.mdnsd' on 'node1' CRS-2676: Start of 'ora.gipcd' on 'node1' succeeded CRS-2676: Start of 'ora.mdnsd' on 'node1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'node1' CRS-2676: Start of 'ora.gpnpd' on 'node1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node1' CRS-2676: Start of 'ora.cssdmonitor' on 'node1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'node1' CRS-2672: Attempting to start 'ora.diskmon' on 'node1' CRS-2676: Start of 'ora.diskmon' on 'node1' succeeded CRS-2676: Start of 'ora.cssd' on 'node1' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'node1' CRS-2676: Start of 'ora.ctssd' on 'node1' succeeded clscfg: -install mode specified Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. CRS-2672: Attempting to start 'ora.crsd' on 'node1' CRS-2676: Start of 'ora.crsd' on 'node1' succeeded Now formatting voting disk: /u05/cludata/votedisk1. Now formatting voting disk: /u01/oradata/orcl/cludata/votedisk2. Now formatting voting disk: /u02/oradata/orcl/cludata/votedisk3. CRS-4603: Successful addition of voting disk /u05/cludata/votedisk1. CRS-4603: Successful addition of voting disk /u01/oradata/orcl/cludata/votedisk2. CRS-4603: Successful addition of voting disk /u02/oradata/orcl/cludata/votedisk3. ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE a893117617e74ffebfd7a379396f281d (/u05/cludata/votedisk1) [] 2. ONLINE 018eafc00dfd4fa9bf568fa245581dcf (/u01/oradata/orcl/cludata/votedisk2) [] 3. ONLINE 0460f0a8560d4f0abf624c47870654fd (/u02/oradata/orcl/cludata/votedisk3) [] Located 3 voting disk(s). CRS-2673: Attempting to stop 'ora.crsd' on 'node1' CRS-2677: Stop of 'ora.crsd' on 'node1' succeeded CRS-2673: Attempting to stop 'ora.ctssd' on 'node1' CRS-2677: Stop of 'ora.ctssd' on 'node1' succeeded CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'node1' CRS-2677: Stop of 'ora.cssdmonitor' on 'node1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'node1' CRS-2677: Stop of 'ora.cssd' on 'node1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1' CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'node1' CRS-2677: Stop of 'ora.gipcd' on 'node1' succeeded CRS-2673: Attempting to stop 'ora.mdnsd' on 'node1' CRS-2677: Stop of 'ora.mdnsd' on 'node1' succeeded CRS-2672: Attempting to start 'ora.mdnsd' on 'node1' CRS-2676: Start of 'ora.mdnsd' on 'node1' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'node1' CRS-2676: Start of 'ora.gipcd' on 'node1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'node1' CRS-2676: Start of 'ora.gpnpd' on 'node1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node1' CRS-2676: Start of 'ora.cssdmonitor' on 'node1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'node1' CRS-2672: Attempting to start 'ora.diskmon' on 'node1' CRS-2676: Start of 'ora.diskmon' on 'node1' succeeded CRS-2676: Start of 'ora.cssd' on 'node1' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'node1' CRS-2676: Start of 'ora.ctssd' on 'node1' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'node1' CRS-2676: Start of 'ora.crsd' on 'node1' succeeded CRS-2672: Attempting to start 'ora.evmd' on 'node1' CRS-2676: Start of 'ora.evmd' on 'node1' succeeded node1 2010/08/27 23:36:53 /oragrid/product/11.2/cdata/node1/backup_20100827_233653.olr Preparing packages for installation... cvuqdisk-1.0.7-1 Configure Oracle Grid Infrastructure for a Cluster ... succeeded Updating inventory properties for clusterware Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 12287 MB Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /oracle/oraInventory 'UpdateNodeList' was successful.

Run the same on node2

[root@node2 11.2]# ./root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /oragrid/product/11.2 Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. 2010-08-27 23:44:15: Parsing the host name 2010-08-27 23:44:15: Checking for super user privileges 2010-08-27 23:44:15: User has super user privileges Using configuration parameter file: /oragrid/product/11.2/crs/install/crsconfig_params Creating trace directory LOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Adding daemon to inittab CRS-4123: Oracle High Availability Services has been started. ohasd is starting CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node node1, number 1, and is terminating An active cluster was found during exclusive startup, restarting to join the cluster CRS-2672: Attempting to start 'ora.mdnsd' on 'node2' CRS-2676: Start of 'ora.mdnsd' on 'node2' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'node2' CRS-2676: Start of 'ora.gipcd' on 'node2' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'node2' CRS-2676: Start of 'ora.gpnpd' on 'node2' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node2' CRS-2676: Start of 'ora.cssdmonitor' on 'node2' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'node2' CRS-2672: Attempting to start 'ora.diskmon' on 'node2' CRS-2676: Start of 'ora.diskmon' on 'node2' succeeded CRS-2676: Start of 'ora.cssd' on 'node2' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'node2' CRS-2676: Start of 'ora.ctssd' on 'node2' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'node2' CRS-2676: Start of 'ora.crsd' on 'node2' succeeded CRS-2672: Attempting to start 'ora.evmd' on 'node2' CRS-2676: Start of 'ora.evmd' on 'node2' succeeded node2 2010/08/27 23:47:32 /oragrid/product/11.2/cdata/node2/backup_20100827_234732.olr Preparing packages for installation... cvuqdisk-1.0.7-1 Configure Oracle Grid Infrastructure for a Cluster ... succeeded Updating inventory properties for clusterware Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 12287 MB Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /oracle/oraInventory 'UpdateNodeList' was successful.

Press "OK" after running all the scripts on all the nodes.

35.)

I received the above error, this can be fixed later. Refer to the post
Oracle: Failed to initialize GPnP

Pres "OK" and then press "Skip" on the original screen.

36.)

Oracle 11gr2 Grid Infrastructure for a 2 Node RAC has completed successfully.

Wednesday, April 1, 2009

What is Voting Disk & Split Brain Syndrome in RAC

Voting Disk

Oracle Clusterware uses the voting disk to determine which instances are members of a cluster. The voting disk must reside on a shared disk. Basically all nodes in the RAC cluster register their heart-beat information on thes voting disks. The number decides the number of active nodes in the RAC cluster. These are also used for checking the availability of instances in RAC and remove the unavailable nodes out of the cluster. It helps in preventing split-brain condition and keeps database information intact. The split brain syndrome and its affects and how it has been managed in oracle is mentioned below.
For high availability, Oracle recommends that you have a minimum of three voting disks. If you configure a single voting disk, then you should use external mirroring to provide redundancy. You can have up to 32 voting disks in your cluster. What I could understand about the odd value of the number of voting disks is that a noe should see maximun number of voting disk to continue to function, so with 2, if it can see only 1, its not the maximum value but a half value of voting disk. I am still trying to search more on this concept.

Split Brain Syndrome:

In a Oracle RAC environment all the instances/servers communicate with each other using high-speed interconnects on the private network. This private network interface or interconnect are redundant and are only used for inter-instance oracle data block transfers. Now talking about split-brain concept with respect to oracle rac systems, it occurs when the instance members in a RAC fail to ping/connect to each other via this private interconnect, but the servers are all pysically up and running and the database instance on each of these servers is also running. These individual nodes are running fine and can conceptually accept user connections and work independently. So basically due to lack of commincation the instance thinks that the other instance that it is not able to connect is down and it needs to do something about the situation. The problem is if we leave these instance running, the sane block might get read, updated in these individual instances and there would be data integrity issue, as the blocks changed in one instance, will not be locked and could be over-written by another instance. Oracle has efficiently implemented check for the split brain syndrome.

What does RAC do incase node becomes inactive:

In RAC if any node becomes inactive, or if other nodes are unable to ping/connect to a node in the RAC, then the node which first detects that one of the node is not accessible, it will evict that node from the RAC group. e.g. there are 4 nodes in a rac instance, and node 3 becomes unavailable, and node 1 tries to connect to node 3 and finds it not responding, then node 1 will evict node 3 out of the RAC groups and will leave only Node1, Node2 & Node4 in the RAC group to continue functioning.
The split brain concepts can become more complicated in large RAC setups. For example there are 10 RAC nodes in a cluster. And say 4 nodes are not able to communicate with the other 6. So there are 2 groups formed in this 10 node RAC cluster ( one group of 4 nodes and other of 6 nodes). Now the nodes will quickly try to affirm their membership by locking controlfile, then the node that lock the controlfile will try to check the votes of the other nodes. The group with the most number of active nodes gets the preference and the others are evicted. Moreover, I have seen this node eviction issue with only 1 node getting evicted and the rest function fine, so I cannot really testify that if thats how it work by experience, but this is the theory behind it.
When we see that the node is evicted, usually oracle rac will reboot that node and try to do a cluster reconfiguration to include back the evicted node.
You will see oracle error: ORA-29740, when there is a node eviction in RAC. There are many reasons for a node eviction like heart beat not received by the controlfile, unable to communicate with the clusterware etc.
A good metalink note on understanding node eviction and how to address is Note ID: 219361.1

The CSS (Cluster Synchronization Service) daemon in the clusterware maintains the heart beat to the voting disk.

Monday, April 28, 2008

Step by Step 11gR1 RAC CRS Installation

Oracle Global Support strongly recommends that you use Red Hat Enterprise Linux ES/AS 4 (update 3 or higher). This is kernel 2.6.9-34 or greater.

Requried RPMS: (check details on Metalink Note:437123.1)

compat-libstdc++-33-3.2.3.x86_64.rpm
elfutils-libelf-devel-0.97-5.x86_64.rpm
glibc-devel-2.3.4-2.19.x86_64.rpm
glibc-devel-2.3.4-2.19.i386.rpm (32-Bit)
gcc-3.4.5-2.x86_64.rpm
libstdc++-devel-3.4.5-2.x86_64.rpm
gcc-c++-3.4.5-2.x86_64.rpm
libaio-0.3.105-2.x86_64.rpm
libaio-0.3.105-2.i386.rpm (32-Bit)
libaio-devel-0.3.105-2.x86_64.rpm
sysstat-5.0.5-7.rhel4.x86_64.rpm
glibc-headers-2.3.4-2.19.x86_64.rpm
glibc-kernheaders-2.4-9.1.98.EL.x86_64.rpm

To query rpm installed in your system:
rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" | grep compat-libstdc++

Sample sysctl.conf
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default=262144
net.core.rmem_max=262144
net.core.wmem_default=262144
net.core.wmem_max=262144
net.ipv4.tcp_wmem = 262144 262144 262144
net.ipv4.tcp_rmem = 4194304 4194304 4194304

Add the following settings to /etc/security/limits.conf

oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

Add or edit the following line in the /etc/pam.d/login file, if it does not already exist:
session required pam_limits.so

Add the following lines to /etc/profile:

if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
This is how the /etc/hosts file look like:
[oracle@oradbdev02 ~]$ more /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
#127.0.0.1 localhost.localdomain localhost
127.0.0.1 oradbdev02.xxxx.com oradbdev02
# Public
10.66.81.61 oradbdev01.xxxx.com oradbdev01
10.66.81.62 oradbdev02.xxxx.com oradbdev02
#Private
192.168.0.61 oradbdev01-priv.xxxx.com oradbdev01-priv
192.168.0.62 oradbdev02-priv.xxxx.com oradbdev02-priv
#Virtual
10.66.81.198 oradbdev01-vip.xxxx.com oradbdev01-vip
10.66.81.199 oradbdev02-vip.xxxx.com oradbdev02-vip

Imp metalink notes to look at before installation: 452780.1.

Setup user equivalence between the rac nodes.
Check that oracle is able to ssh between the rac servers without having to type the password.
Setup up location of OCR & Voting disk files. For test purposes I used the same mount point with different folders for each e.g.
/DB01/ocr
/DB01/voting_disk

Now run the cluster verify utility to do a pre-check before the cluster installation.
[oracle@oradbdev01 clusterware]$ ./runcluvfy.sh stage -pre crsinst -n oradbdev01,oradbdev02 -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "oradbdev01"
Destination Node Reachable?
------------------------------------ ------------------------
oradbdev01 yes
oradbdev02 yes
Result: Node reachability check passed from node "oradbdev01".

Checking user equivalence...

Check: User equivalence for user "oracle"
Node Name Comment
------------------------------------ ------------------------
oradbdev02 passed
oradbdev01 passed
Result: User equivalence check passed for user "oracle".

Checking administrative privileges...

Check: Existence of user "oracle"
Node Name User Exists Comment
------------ ------------------------ ------------------------
oradbdev02 yes passed
oradbdev01 yes passed
Result: User existence check passed for "oracle".

Check: Existence of group "oinstall"
Node Name Status Group ID
------------ ------------------------ ------------------------
oradbdev02 exists 503
oradbdev01 exists 503
Result: Group existence check passed for "oinstall".

Check: Membership of user "oracle" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Comment
---------------- ------------ ------------ ------------ ------------ ------------
oradbdev02 yes yes yes no failed
oradbdev01 yes yes yes no failed
Result: Membership check for user "oracle" in group "oinstall" [as Primary] failed.

Administrative privileges check failed.

Checking node connectivity...

Interface information for node "oradbdev02"
Interface Name IP Address Subnet Subnet Gateway Default Gateway Hardware Address
---------------- ------------ ------------ ------------ ------------ ------------
eth0 10.66.81.62 10.66.80.0 0.0.0.0 10.66.80.1 00:1B:78:DF:CC:9C
eth1 192.168.0.62 192.168.0.0 0.0.0.0 10.66.80.1 00:1B:78:DF:CC:84

Interface information for node "oradbdev01"
Interface Name IP Address Subnet Subnet Gateway Default Gateway Hardware Address
---------------- ------------ ------------ ------------ ------------ ------------
eth0 10.66.81.61 10.66.80.0 0.0.0.0 10.66.80.1 00:1B:78:DF:AC:0E
eth1 192.168.0.61 192.168.0.0 0.0.0.0 10.66.80.1 00:1B:78:DF:AC:08

Check: Node connectivity of subnet "10.66.80.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
oradbdev02:eth0 oradbdev01:eth0 yes
Result: Node connectivity check passed for subnet "10.66.80.0" with node(s) oradbdev02,oradbdev01.

Check: Node connectivity of subnet "192.168.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
oradbdev02:eth1 oradbdev01:eth1 yes
Result: Node connectivity check passed for subnet "192.168.0.0" with node(s) oradbdev02,oradbdev01.

Interfaces found on subnet "10.66.80.0" that are likely candidates for VIP:
oradbdev02 eth0:10.66.81.62
oradbdev01 eth0:10.66.81.61

Interfaces found on subnet "192.168.0.0" that are likely candidates for a private interconnect:
oradbdev02 eth1:192.168.0.62
oradbdev01 eth1:192.168.0.61

Result: Node connectivity check passed.

Checking system requirements for 'crs'...

Check: Total memory
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
oradbdev02 7.79GB (8164980KB) 1GB (1048576KB) passed
oradbdev01 7.79GB (8164980KB) 1GB (1048576KB) passed
Result: Total memory check passed.

Check: Free disk space in "/tmp" dir
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
oradbdev02 199.68GB (209378492KB) 400MB (409600KB) passed
oradbdev01 216.94GB (227478076KB) 400MB (409600KB) passed
Result: Free disk space check passed.

Check: Swap space
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
oradbdev02 15.81GB (16579072KB) 1.5GB (1572864KB) passed
oradbdev01 15.8GB (16571036KB) 1.5GB (1572864KB) passed
Result: Swap space check passed.

Check: System architecture
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
oradbdev02 x86_64 x86_64 passed
oradbdev01 x86_64 x86_64 passed
Result: System architecture check passed.

Check: Kernel version
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
oradbdev02 2.6.9-55.ELsmp 2.6.9-11.EL passed
oradbdev01 2.6.9-55.ELsmp 2.6.9-11.EL passed
Result: Kernel version check passed.

Check: Package existence for "make-3.80"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 make-3.80-6.EL4 passed
oradbdev01 make-3.80-6.EL4 passed
Result: Package existence check passed for "make-3.80".

Check: Package existence for "binutils-2.15.92.0.2"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 binutils-2.15.92.0.2-22 passed
oradbdev01 binutils-2.15.92.0.2-22 passed
Result: Package existence check passed for "binutils-2.15.92.0.2".

Check: Package existence for "gcc-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 gcc-3.4.6-9 passed
oradbdev01 gcc-3.4.6-8 passed
Result: Package existence check passed for "gcc-3.4.5".

Check: Package existence for "libaio-0.3.105"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libaio-0.3.105-2 passed
oradbdev01 libaio-0.3.105-2 passed
Result: Package existence check passed for "libaio-0.3.105".

Check: Package existence for "libaio-0.3.105"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libaio-0.3.105-2 passed
oradbdev01 libaio-0.3.105-2 passed
Result: Package existence check passed for "libaio-0.3.105".

Check: Package existence for "libaio-devel-0.3.105"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libaio-devel-0.3.105-2 passed
oradbdev01 libaio-devel-0.3.105-2 passed
Result: Package existence check passed for "libaio-devel-0.3.105".

Check: Package existence for "libstdc++-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libstdc++-3.4.6-9 passed
oradbdev01 libstdc++-3.4.6-8 passed
Result: Package existence check passed for "libstdc++-3.4.5".

Check: Package existence for "libstdc++-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libstdc++-3.4.6-9 passed
oradbdev01 libstdc++-3.4.6-8 passed
Result: Package existence check passed for "libstdc++-3.4.5".

Check: Package existence for "elfutils-libelf-devel-0.97"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 elfutils-libelf-devel-0.97.1-5 passed
oradbdev01 elfutils-libelf-devel-0.97.1-4 passed
Result: Package existence check passed for "elfutils-libelf-devel-0.97".

Check: Package existence for "sysstat-5.0.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 sysstat-5.0.5-16.rhel4 passed
oradbdev01 sysstat-5.0.5-14.rhel4 passed
Result: Package existence check passed for "sysstat-5.0.5".

Check: Package existence for "libgcc-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libgcc-3.4.6-8 passed
oradbdev01 libgcc-3.4.6-8 passed
Result: Package existence check passed for "libgcc-3.4.5".

Check: Package existence for "libgcc-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libgcc-3.4.6-9 passed
oradbdev01 libgcc-3.4.6-8 passed
Result: Package existence check passed for "libgcc-3.4.5".

Check: Package existence for "libstdc++-devel-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libstdc++-devel-3.4.6-9 passed
oradbdev01 libstdc++-devel-3.4.6-8 passed
Result: Package existence check passed for "libstdc++-devel-3.4.5".

Check: Package existence for "elfutils-libelf-0.97"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 elfutils-libelf-0.97.1-4elfutils-libelf-0.97.1-5 passed
oradbdev01 elfutils-libelf-0.97.1-4 passed
Result: Package existence check passed for "elfutils-libelf-0.97".

Check: Package existence for "glibc-2.3.4-2.19"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 failed
oradbdev01 failed
Result: Package existence check failed for "glibc-2.3.4-2.19".

Check: Package existence for "glibc-2.3.4-2.19"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 glibc-2.3.4-2.36 passed
oradbdev01 glibc-2.3.4-2.36 passed
Result: Package existence check passed for "glibc-2.3.4-2.19".

Check: Package existence for "glibc-common-2.3.4"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 glibc-common-2.3.4-2.36 passed
oradbdev01 glibc-common-2.3.4-2.36 passed
Result: Package existence check passed for "glibc-common-2.3.4".

Check: Package existence for "glibc-devel-2.3.4"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 glibc-devel-2.3.4-2.36 passed
oradbdev01 glibc-devel-2.3.4-2.36 passed
Result: Package existence check passed for "glibc-devel-2.3.4".

Check: Package existence for "glibc-devel-2.3.4"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 glibc-devel-2.3.4-2.36 passed
oradbdev01 glibc-devel-2.3.4-2.36 passed
Result: Package existence check passed for "glibc-devel-2.3.4".

Check: Package existence for "gcc-c++-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 gcc-c++-3.4.6-9 passed
oradbdev01 gcc-c++-3.4.6-8 passed
Result: Package existence check passed for "gcc-c++-3.4.5".

Check: Package existence for "compat-libstdc++-33-3.2.3"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 compat-libstdc++-33-3.2.3-47.3 passed
oradbdev01 compat-libstdc++-33-3.2.3-47.3 passed
Result: Package existence check passed for "compat-libstdc++-33-3.2.3".

Check: Package existence for "compat-libstdc++-33-3.2.3"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 compat-libstdc++-33-3.2.3-47.3 passed
oradbdev01 compat-libstdc++-33-3.2.3-47.3 passed
Result: Package existence check passed for "compat-libstdc++-33-3.2.3".

Check: Group existence for "dba"
Node Name Status Comment
------------ ------------------------ ------------------------
oradbdev02 exists passed
oradbdev01 exists passed
Result: Group existence check passed for "dba".

Check: Group existence for "oinstall"
Node Name Status Comment
------------ ------------------------ ------------------------
oradbdev02 exists passed
oradbdev01 exists passed
Result: Group existence check passed for "oinstall".

Check: User existence for "nobody"
Node Name Status Comment
------------ ------------------------ ------------------------
oradbdev02 exists passed
oradbdev01 exists passed
Result: User existence check passed for "nobody".

System requirement failed for 'crs'

Pre-check for cluster services setup was unsuccessful on all the nodes.

The check has failed for me for checking if oracle users primary group is "oinstall". I am using dba as the groups. Hence I am omitting this error.

Now we are all set to start the OUI
[oracle@oradbdev01 clusterware]$ /home/oracle/software/11g/Cluster/clusterware/runInstaller
Now follow the screen shots:
The Welcome Screen - Press Next:

Apun's Weblog

AWS

Non Oracle

RAC & DG Related

DBA Related

Popular Posts

Blog Archive

Thursday, July 21, 2011

Oracle: Rename single/RAC Database

Saturday, August 28, 2010

Oracle 11gR2 Grid Infrastructure Installation Steps

Wednesday, April 1, 2009

What is Voting Disk & Split Brain Syndrome in RAC

Monday, April 28, 2008

Step by Step 11gR1 RAC CRS Installation

Monday, April 21, 2008

10.2.0.1 CRS Installation and upgrade to 10.2.0.3 for RAC

About Me

Followers