Step by Step Data Guard Setup for Oracle 10g

Steps:
1.) Make sure archive log mode is enabled on your database:
SQL> archive log list
Database log mode Archive Mode
Automatic archival Enabled
Archive destination /opt/app/oracle/oradata/orcl/archive
Oldest online log sequence 108
Next log sequence to archive 109
Current log sequence 109

SQL> select name, log_mode from v$database;

NAME LOG_MODE
--------- ------------
ORCL ARCHIVELOG

If archive log mode is not enabled. Please enable it using the following link.
How to enable archivelog mode in Oracle 11g database

2.) Enable force logging on the database, so that there is no problems with no logging operations in the future.

SQL> alter database force logging;

Database altered.

3.) Create password file, if you do not have one already.

[oracle@APP3 dbs]$ cd $ORACLE_HOME/dbs
[oracle@APP3 dbs]$ orapwd file=orapworcl password=oracle force=y
[oracle@APP3 dbs]$ ls -lrt orapworcl
-rw-r----- 1 oracle oinstall 1536 Sep 14 08:21 orapworcl

SQL> select * from v$pwfile_users;

USERNAME SYSDB SYSOP
------------------------------ ----- -----
SYS TRUE TRUE


4.) Create Standby Redo Logfiles on primary DB.

Current logfile:
SQL> col member a40
SQL> select a.group#,a.status,a.member,b.bytes/1024/1024 from v$logfile a,v$log b
2 where a.group#=b.group#;

GROUP# STATUS MEMBER B.BYTES/1024/1024
---------- ------- ---------------------------------------- -----------------
1 /opt/app/oracle/oradata/orcl/redo01.log 50
2 /opt/app/oracle/oradata/orcl/redo02.log 50

Add standby redo log groups:

SQL> alter database add standby logfile group 3 size 50M;

Database altered.

SQL> alter database add standby logfile group 4 size 50M;

Database altered.

SQL> select * from v$logfile;

GROUP# STATUS TYPE MEMBER IS_
---------- ------- ------- --------------------------------------------------------------------------- ---
1 ONLINE /opt/app/oracle/oradata/ORCL/redo01.log NO
2 ONLINE /opt/app/oracle/oradata/ORCL/redo02.log NO
3 STANDBY /opt/app/oracle/flash_recovery_area/ORCL/onlinelog/o1_mf_3_5bvzkzgs_.log YES
4 STANDBY /opt/app/oracle/flash_recovery_area/ORCL/onlinelog/o1_mf_4_5bvzl8hf_.log YES

SQL> select * from v$standby_log;

GROUP# DBID THREAD# SEQUENCE# BYTES USED ARC STATUS FIRST_CHANGE# FIRST_TIM LAST_CHANGE# LAST_TIME
----------------------------------------------------------------------------------------------------------------------
3 UNASSIGNED 0 0 52428800 512 YES UNASSIGNED 0 0
4 UNASSIGNED 0 0 52428800 512 YES UNASSIGNED 0 0

5.) Check parameter db_unique_name
SQL> show parameters unique

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_unique_name string orcl

6.) Add standby related entried to Primary database:
SQL> create pfile='/home/oracle/initprim.ora' from spfile;

Sample init.ora from Primary:
orcl.__db_cache_size=2097152000
orcl.__java_pool_size=16777216
orcl.__large_pool_size=16777216
orcl.__shared_pool_size=536870912
orcl.__streams_pool_size=0
*.audit_file_dest='/opt/app/oracle/admin/orcl/adump'
*.background_dump_dest='/opt/app/oracle/admin/orcl/bdump'
*.compatible='10.2.0.3.0'
*.control_files='/opt/app/oracle/oradata/orcl/control01.ctl','/opt/app/oracle/oradata/orcl/control02.ctl','/opt/app/oracle/oradata/orcl/control03.ctl'
*.core_dump_dest='/opt/app/oracle/admin/orcl/cdump'
*.db_block_size=8192
*.db_domain=''
*.db_file_multiblock_read_count=16
*.db_name='orcl'
*.db_recovery_file_dest='/opt/app/oracle/flash_recovery_area'
*.db_recovery_file_dest_size=2147483648
*.db_unique_name='orcl'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=orclXDB)'
*.job_queue_processes=10
*.log_archive_dest_1='location=/opt/app/oracle/oradata/orcl/archive VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=orcl'
*.open_cursors=300
*.pga_aggregate_target=823132160
*.processes=500
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_target=2684354560
*.undo_management='AUTO'
*.undo_tablespace='UNDOTBS1'
*.user_dump_dest='/opt/app/oracle/admin/orcl/udump'
db_unique_name=orcl
LOG_ARCHIVE_CONFIG='DG_CONFIG=(orcl,orcl1)'
*.LOG_ARCHIVE_DEST_2='SERVICE=ORCL1 LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=orcl1'
LOG_ARCHIVE_DEST_STATE_1=ENABLE
LOG_ARCHIVE_DEST_STATE_2=ENABLE
LOG_ARCHIVE_FORMAT=%t_%s_%r.arc
LOG_ARCHIVE_MAX_PROCESSES=3
DB_FILE_NAME_CONVERT='/u01/oradata/orcl','/opt/app/oracle/oradata/orcl'
LOG_FILE_NAME_CONVERT='/u01/oradata/orcl/archive',/opt/app/oracle/oradata/orcl/archive','/u01/oradata/flash_recovery_area/orcl','/opt/app/oracle/flash_recovery_area/orcl/onlinelog'
FAL_SERVER=orcl1
FAL_CLIENT=orcl

Copy the init.ora and make necessary changes to the file to be used at standby side. Changes like location of various files, FAL_SERVER, FAL_CLIENT etc.

Sample init.ora in Standby DB:
orcl.__db_cache_size=2097152000
orcl.__java_pool_size=16777216
orcl.__large_pool_size=16777216
orcl.__shared_pool_size=536870912
orcl.__streams_pool_size=0
*.audit_file_dest='/u01/oradata/orcl/adump'
*.background_dump_dest='/u01/oradata/orcl/bdump'
*.compatible='10.2.0.3.0'
*.control_files='/u01/oradata/orcl/control01.ctl','/u01/oradata/orcl/control02.ctl','/u01/oradata/orcl/control03.ctl'
*.core_dump_dest='/u01/oradata/orcl/cdump'
*.db_block_size=8192
*.db_domain=''
*.db_file_multiblock_read_count=16
*.db_name='orcl'
*.db_recovery_file_dest='/u01/oradata/flash_recovery_area/orcl'
*.db_recovery_file_dest_size=2147483648
*.db_unique_name='orcl1'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=orclXDB)'
*.job_queue_processes=10
*.log_archive_dest_1='location=/u01/oradata/orcl/archive VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=orcl'
*.open_cursors=300
*.pga_aggregate_target=823132160
*.processes=500
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_target=2684354560
*.undo_management='AUTO'
*.undo_tablespace='UNDOTBS1'
*.user_dump_dest='/u01/oradata/orcl/udump'
LOG_ARCHIVE_CONFIG='DG_CONFIG=(orcl,orcl1)'
*.LOG_ARCHIVE_DEST_2='SERVICE=ORCL LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=orcl'
LOG_ARCHIVE_DEST_STATE_1=ENABLE
LOG_ARCHIVE_DEST_STATE_2=ENABLE
LOG_ARCHIVE_FORMAT=%t_%s_%r.arc
LOG_ARCHIVE_MAX_PROCESSES=3
DB_FILE_NAME_CONVERT='/opt/app/oracle/oradata/orcl','/u01/oradata/orcl'
LOG_FILE_NAME_CONVERT='/opt/app/oracle/oradata/orcl/archive','/u01/oradata/orcl/archive','/opt/app/oracle/flash_recovery_area/orcl/onlinelog','/u01/oradata/flash_recovery_area/orcl'
FAL_SERVER=orcl
FAL_CLIENT=orcl1

7.) Shutdown the primary database. Use the newly created pfile to startup nomount the database. Then create a spfile for the database. Mount the database and create a standby controlfile.
Shutdown the database and take a cold back of the database, all files including the redo log files. You can also create a standby DB from hot backup.

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup nomount pfile='/home/oracle/pfileorcl.ora'
ORACLE instance started.

Total System Global Area 2684354560 bytes
Fixed Size 2086352 bytes
Variable Size 570427952 bytes
Database Buffers 2097152000 bytes
Redo Buffers 14688256 bytes
SQL> create spfile from pfile='/home/oracle/pfileorcl.ora';

File created.
Meanwhile I also received the error:
create spfile from pfile='/home/oracle/pfileregdb.ora'
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [kspsetpao1], [1753], [1700], [*],
[user_dump_dest], [33], [], []

Note: This error usually comes when the syntax of the pfile is wrong somewhere, please fix the pfile and try again. It worked for me.

SQL> shutdown immediate;
ORA-01507: database not mounted


ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 2684354560 bytes
Fixed Size 2086352 bytes
Variable Size 570427952 bytes
Database Buffers 2097152000 bytes
Redo Buffers 14688256 bytes
Database mounted.
Database opened.

8.) Shutdown the database again and take a cold backup of all files.

9.) Create standby control file.
SQL> startup mount
ORACLE instance started.
Total System Global Area 2684354560 bytes
Fixed Size 2086352 bytes
Variable Size 570427952 bytes
Database Buffers 2097152000 bytes
Redo Buffers 14688256 bytes
Database mounted.

--Then mount and create a standby controlfile.
SQL> alter database create standby controlfile as 'standby.ctl';
Database altered.

-- Open the primary read write.
SQL> alter database open;
Database altered.

10.) Transfer all the file from the cold backup from Primary to Standby server. Also copy the password file from primary to standby.
Also copy the standby controlfile created in step 9 and copy if with the right name and location on standby server.
I use SFTP for transferring the files.

11.) Add entries for the primary db and standby DB in both primary and standby servers. i.e. primary server should have its own (orcl) and standby server (orcl1) tns entry.

12.) Copy the pfile from step 6 for standby DB.
Now try to nomount the standby database with the new pfile.

[oracle@dbtest dbs]$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.1.0 - Production on Tue Sep 15 04:57:32 2009

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile='/home/oracle/oracle/product/10.2.0/db_1/dbs/pfilestbregdb.ora';
ORACLE instance started.

Total System Global Area 1694498816 bytes
Fixed Size 1219784 bytes
Variable Size 402654008 bytes
Database Buffers 1275068416 bytes
Redo Buffers 15556608 bytes

13.) Create spfile from pfile.
SQL> create spfile from pfile='/home/oracle/oracle/product/10.2.0/db_1/dbs/pfilestbregdb.ora';

File created.

14.) Shutdown the DB and do a startup mount.
SQL>startup mount;

15.) Start REDO apply process:
SQL> alter database recover managed standby database disconnect from session;
OR
SQL> alter database recover managed standby database nodelay disconnect parallel 8;

16.) Verification.
SQL> SELECT PROCESS, STATUS, THREAD#, SEQUENCE#, BLOCK#, BLOCKS FROM V$MANAGED_STANDBY;

To check archive gap:
SELECT thread#, low_sequence#, high_sequence# from V$archive_gap;
for RAC
SELECT thread#, low_sequence#, high_sequence# from gv$archive_gap;

To stop redo apply:
alter database recover managed standby database cancel;

17.) Check alert log files and verify that you did not receive any error.

18.) Switch some logfiles on the Primary and check if the same are getting applied to the standby.
on Primary:
SQL> alter system switch logfile;

on standby:
SQL> SELECT PROCESS, STATUS, THREAD#, SEQUENCE#, BLOCK#, BLOCKS FROM V$MANAGED_STANDBY;

Thanks should be it, your Physical Standby DB should be working fine.

Step by Step 11gR1 RAC CRS Installation

Oracle Global Support strongly recommends that you use Red Hat Enterprise Linux ES/AS 4 (update 3 or higher). This is kernel 2.6.9-34 or greater.

Requried RPMS: (check details on Metalink Note:437123.1)

compat-libstdc++-33-3.2.3.x86_64.rpm
elfutils-libelf-devel-0.97-5.x86_64.rpm
glibc-devel-2.3.4-2.19.x86_64.rpm
glibc-devel-2.3.4-2.19.i386.rpm (32-Bit)
gcc-3.4.5-2.x86_64.rpm
libstdc++-devel-3.4.5-2.x86_64.rpm
gcc-c++-3.4.5-2.x86_64.rpm
libaio-0.3.105-2.x86_64.rpm
libaio-0.3.105-2.i386.rpm (32-Bit)
libaio-devel-0.3.105-2.x86_64.rpm
sysstat-5.0.5-7.rhel4.x86_64.rpm
glibc-headers-2.3.4-2.19.x86_64.rpm
glibc-kernheaders-2.4-9.1.98.EL.x86_64.rpm

To query rpm installed in your system:
rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" | grep compat-libstdc++

Sample sysctl.conf
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default=262144
net.core.rmem_max=262144
net.core.wmem_default=262144
net.core.wmem_max=262144
net.ipv4.tcp_wmem = 262144 262144 262144
net.ipv4.tcp_rmem = 4194304 4194304 4194304

Add the following settings to /etc/security/limits.conf

oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

Add or edit the following line in the /etc/pam.d/login file, if it does not already exist:
session required pam_limits.so

Add the following lines to /etc/profile:

if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
This is how the /etc/hosts file look like:
[oracle@oradbdev02 ~]$ more /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
#127.0.0.1 localhost.localdomain localhost
127.0.0.1 oradbdev02.xxxx.com oradbdev02
# Public
10.66.81.61 oradbdev01.xxxx.com oradbdev01
10.66.81.62 oradbdev02.xxxx.com oradbdev02
#Private
192.168.0.61 oradbdev01-priv.xxxx.com oradbdev01-priv
192.168.0.62 oradbdev02-priv.xxxx.com oradbdev02-priv
#Virtual
10.66.81.198 oradbdev01-vip.xxxx.com oradbdev01-vip
10.66.81.199 oradbdev02-vip.xxxx.com oradbdev02-vip

Imp metalink notes to look at before installation: 452780.1.

Setup user equivalence between the rac nodes.
Check that oracle is able to ssh between the rac servers without having to type the password.
Setup up location of OCR & Voting disk files. For test purposes I used the same mount point with different folders for each e.g.
/DB01/ocr
/DB01/voting_disk

Now run the cluster verify utility to do a pre-check before the cluster installation.
[oracle@oradbdev01 clusterware]$ ./runcluvfy.sh stage -pre crsinst -n oradbdev01,oradbdev02 -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "oradbdev01"
Destination Node Reachable?
------------------------------------ ------------------------
oradbdev01 yes
oradbdev02 yes
Result: Node reachability check passed from node "oradbdev01".


Checking user equivalence...

Check: User equivalence for user "oracle"
Node Name Comment
------------------------------------ ------------------------
oradbdev02 passed
oradbdev01 passed
Result: User equivalence check passed for user "oracle".

Checking administrative privileges...

Check: Existence of user "oracle"
Node Name User Exists Comment
------------ ------------------------ ------------------------
oradbdev02 yes passed
oradbdev01 yes passed
Result: User existence check passed for "oracle".

Check: Existence of group "oinstall"
Node Name Status Group ID
------------ ------------------------ ------------------------
oradbdev02 exists 503
oradbdev01 exists 503
Result: Group existence check passed for "oinstall".

Check: Membership of user "oracle" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Comment
---------------- ------------ ------------ ------------ ------------ ------------
oradbdev02 yes yes yes no failed
oradbdev01 yes yes yes no failed
Result: Membership check for user "oracle" in group "oinstall" [as Primary] failed.

Administrative privileges check failed.

Checking node connectivity...


Interface information for node "oradbdev02"
Interface Name IP Address Subnet Subnet Gateway Default Gateway Hardware Address
---------------- ------------ ------------ ------------ ------------ ------------
eth0 10.66.81.62 10.66.80.0 0.0.0.0 10.66.80.1 00:1B:78:DF:CC:9C
eth1 192.168.0.62 192.168.0.0 0.0.0.0 10.66.80.1 00:1B:78:DF:CC:84


Interface information for node "oradbdev01"
Interface Name IP Address Subnet Subnet Gateway Default Gateway Hardware Address
---------------- ------------ ------------ ------------ ------------ ------------
eth0 10.66.81.61 10.66.80.0 0.0.0.0 10.66.80.1 00:1B:78:DF:AC:0E
eth1 192.168.0.61 192.168.0.0 0.0.0.0 10.66.80.1 00:1B:78:DF:AC:08


Check: Node connectivity of subnet "10.66.80.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
oradbdev02:eth0 oradbdev01:eth0 yes
Result: Node connectivity check passed for subnet "10.66.80.0" with node(s) oradbdev02,oradbdev01.

Check: Node connectivity of subnet "192.168.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
oradbdev02:eth1 oradbdev01:eth1 yes
Result: Node connectivity check passed for subnet "192.168.0.0" with node(s) oradbdev02,oradbdev01.

Interfaces found on subnet "10.66.80.0" that are likely candidates for VIP:
oradbdev02 eth0:10.66.81.62
oradbdev01 eth0:10.66.81.61

Interfaces found on subnet "192.168.0.0" that are likely candidates for a private interconnect:
oradbdev02 eth1:192.168.0.62
oradbdev01 eth1:192.168.0.61

Result: Node connectivity check passed.


Checking system requirements for 'crs'...

Check: Total memory
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
oradbdev02 7.79GB (8164980KB) 1GB (1048576KB) passed
oradbdev01 7.79GB (8164980KB) 1GB (1048576KB) passed
Result: Total memory check passed.

Check: Free disk space in "/tmp" dir
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
oradbdev02 199.68GB (209378492KB) 400MB (409600KB) passed
oradbdev01 216.94GB (227478076KB) 400MB (409600KB) passed
Result: Free disk space check passed.

Check: Swap space
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
oradbdev02 15.81GB (16579072KB) 1.5GB (1572864KB) passed
oradbdev01 15.8GB (16571036KB) 1.5GB (1572864KB) passed
Result: Swap space check passed.

Check: System architecture
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
oradbdev02 x86_64 x86_64 passed
oradbdev01 x86_64 x86_64 passed
Result: System architecture check passed.

Check: Kernel version
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
oradbdev02 2.6.9-55.ELsmp 2.6.9-11.EL passed
oradbdev01 2.6.9-55.ELsmp 2.6.9-11.EL passed
Result: Kernel version check passed.

Check: Package existence for "make-3.80"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 make-3.80-6.EL4 passed
oradbdev01 make-3.80-6.EL4 passed
Result: Package existence check passed for "make-3.80".

Check: Package existence for "binutils-2.15.92.0.2"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 binutils-2.15.92.0.2-22 passed
oradbdev01 binutils-2.15.92.0.2-22 passed
Result: Package existence check passed for "binutils-2.15.92.0.2".

Check: Package existence for "gcc-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 gcc-3.4.6-9 passed
oradbdev01 gcc-3.4.6-8 passed
Result: Package existence check passed for "gcc-3.4.5".

Check: Package existence for "libaio-0.3.105"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libaio-0.3.105-2 passed
oradbdev01 libaio-0.3.105-2 passed
Result: Package existence check passed for "libaio-0.3.105".

Check: Package existence for "libaio-0.3.105"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libaio-0.3.105-2 passed
oradbdev01 libaio-0.3.105-2 passed
Result: Package existence check passed for "libaio-0.3.105".

Check: Package existence for "libaio-devel-0.3.105"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libaio-devel-0.3.105-2 passed
oradbdev01 libaio-devel-0.3.105-2 passed
Result: Package existence check passed for "libaio-devel-0.3.105".

Check: Package existence for "libstdc++-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libstdc++-3.4.6-9 passed
oradbdev01 libstdc++-3.4.6-8 passed
Result: Package existence check passed for "libstdc++-3.4.5".

Check: Package existence for "libstdc++-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libstdc++-3.4.6-9 passed
oradbdev01 libstdc++-3.4.6-8 passed
Result: Package existence check passed for "libstdc++-3.4.5".

Check: Package existence for "elfutils-libelf-devel-0.97"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 elfutils-libelf-devel-0.97.1-5 passed
oradbdev01 elfutils-libelf-devel-0.97.1-4 passed
Result: Package existence check passed for "elfutils-libelf-devel-0.97".

Check: Package existence for "sysstat-5.0.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 sysstat-5.0.5-16.rhel4 passed
oradbdev01 sysstat-5.0.5-14.rhel4 passed
Result: Package existence check passed for "sysstat-5.0.5".

Check: Package existence for "libgcc-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libgcc-3.4.6-8 passed
oradbdev01 libgcc-3.4.6-8 passed
Result: Package existence check passed for "libgcc-3.4.5".

Check: Package existence for "libgcc-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libgcc-3.4.6-9 passed
oradbdev01 libgcc-3.4.6-8 passed
Result: Package existence check passed for "libgcc-3.4.5".

Check: Package existence for "libstdc++-devel-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 libstdc++-devel-3.4.6-9 passed
oradbdev01 libstdc++-devel-3.4.6-8 passed
Result: Package existence check passed for "libstdc++-devel-3.4.5".

Check: Package existence for "elfutils-libelf-0.97"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 elfutils-libelf-0.97.1-4elfutils-libelf-0.97.1-5 passed
oradbdev01 elfutils-libelf-0.97.1-4 passed
Result: Package existence check passed for "elfutils-libelf-0.97".

Check: Package existence for "glibc-2.3.4-2.19"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 failed
oradbdev01 failed
Result: Package existence check failed for "glibc-2.3.4-2.19".

Check: Package existence for "glibc-2.3.4-2.19"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 glibc-2.3.4-2.36 passed
oradbdev01 glibc-2.3.4-2.36 passed
Result: Package existence check passed for "glibc-2.3.4-2.19".

Check: Package existence for "glibc-common-2.3.4"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 glibc-common-2.3.4-2.36 passed
oradbdev01 glibc-common-2.3.4-2.36 passed
Result: Package existence check passed for "glibc-common-2.3.4".

Check: Package existence for "glibc-devel-2.3.4"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 glibc-devel-2.3.4-2.36 passed
oradbdev01 glibc-devel-2.3.4-2.36 passed
Result: Package existence check passed for "glibc-devel-2.3.4".

Check: Package existence for "glibc-devel-2.3.4"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 glibc-devel-2.3.4-2.36 passed
oradbdev01 glibc-devel-2.3.4-2.36 passed
Result: Package existence check passed for "glibc-devel-2.3.4".

Check: Package existence for "gcc-c++-3.4.5"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 gcc-c++-3.4.6-9 passed
oradbdev01 gcc-c++-3.4.6-8 passed
Result: Package existence check passed for "gcc-c++-3.4.5".

Check: Package existence for "compat-libstdc++-33-3.2.3"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 compat-libstdc++-33-3.2.3-47.3 passed
oradbdev01 compat-libstdc++-33-3.2.3-47.3 passed
Result: Package existence check passed for "compat-libstdc++-33-3.2.3".

Check: Package existence for "compat-libstdc++-33-3.2.3"
Node Name Status Comment
------------------------------ ------------------------------ ----------------
oradbdev02 compat-libstdc++-33-3.2.3-47.3 passed
oradbdev01 compat-libstdc++-33-3.2.3-47.3 passed
Result: Package existence check passed for "compat-libstdc++-33-3.2.3".

Check: Group existence for "dba"
Node Name Status Comment
------------ ------------------------ ------------------------
oradbdev02 exists passed
oradbdev01 exists passed
Result: Group existence check passed for "dba".

Check: Group existence for "oinstall"
Node Name Status Comment
------------ ------------------------ ------------------------
oradbdev02 exists passed
oradbdev01 exists passed
Result: Group existence check passed for "oinstall".

Check: User existence for "nobody"
Node Name Status Comment
------------ ------------------------ ------------------------
oradbdev02 exists passed
oradbdev01 exists passed
Result: User existence check passed for "nobody".

System requirement failed for 'crs'

Pre-check for cluster services setup was unsuccessful on all the nodes.

The check has failed for me for checking if oracle users primary group is "oinstall". I am using dba as the groups. Hence I am omitting this error.

Now we are all set to start the OUI
[oracle@oradbdev01 clusterware]$ /home/oracle/software/11g/Cluster/clusterware/runInstaller
Now follow the screen shots:
The Welcome Screen - Press Next:

From For Blog

Select the location for your CRS_HOME, where the crs will get installed and Press Next:

From For Blog

Oracle does some pre-check, resolve any issues and Press Next:

From For Blog

Add information about the cluster nodes and the Press Next:

From For Blog

Example of how to add information:

From For Blog

Select the Public and Private IP address subnets and Press Next:

From For Blog

Specify OCR Location and Press Next:

From For Blog

Specify Voting Disk Location and Press Next:

From For Blog

Installation summary window, check for the remote node and Press Install:

From For Blog

Installation in Progress:

From For Blog

I received this error as I had not setup my environment properly before starting the installation.
Error from make.log
/usr/bin/make -f ins_rdbms.mk ipc_none ORACLE_HOME=/home/oracle/product/11.1.0/db//home/oracle/product/11.1.0/db/rdbms/lib/ins_rdbms.mk:2: /home/oracle/product/11.1.0/db/rdbms/lib/env_rdbms.mk: No such file or directory
make: *** No rule to make target `/home/oracle/product/11.1.0/db/rdbms/lib/env_rdbms.mk'. Stop.
I stopped the installation and setup ORACLE_HOME again and started the install again.
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0x0000002a9591bb92, pid=8270, tid=1094084960
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_11-b03 mixed mode)
# Problematic frame:
# V [libjvm.so+0x3a4b92]
#
# An error report file with more information is saved as hs_err_pid8270.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp

Again at this point my $PATH showed wrong path. Hence remove unnecessary entries from $PATH and restarted installation and it worked this time.

From For Blog

Run root.sh on both the nodes:

From For Blog

[oracle@oradbdev01 crs]$ sudo ./root.sh
Password:
WARNING: directory '/home/oracle/product/11.1.0' is not owned by root
WARNING: directory '/home/oracle/product' is not owned by root
WARNING: directory '/home/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up Network socket directories
PROT-1: Failed to initialize ocrconfig
Failed to upgrade Oracle Cluster Registry configuration

There were permission problems on the folders in node 2.
Resolved the issue and restarted the installation.
Node 1:
Running root.sh:
[oracle@oradbdev01 crs]$ sudo ./root.sh
WARNING: directory '/home/oracle/product/11.1.0' is not owned by root
WARNING: directory '/home/oracle/product' is not owned by root
WARNING: directory '/home/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.

Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
The directory '/home/oracle/product/11.1.0' is not owned by root. Changing owner to root
The directory '/home/oracle/product' is not owned by root. Changing owner to root
The directory '/home/oracle' is not owned by root. Changing owner to root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: oradbdev01 oradbdev01-priv oradbdev01
node 2: oradbdev02 oradbdev02-priv oradbdev02
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /DB01/11g_rac/voting_disk/vote1
Now formatting voting device: /DB01/11g_rac/voting_disk/vote2
Now formatting voting device: /DB01/11g_rac/voting_disk/vote3
Format of 3 voting devices complete.
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
oradbdev01
Cluster Synchronization Services is inactive on these nodes.
oradbdev02
Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.

Node 2:
[oracle@oradbdev02 crs]$ sudo ./root.sh
Password:
WARNING: directory '/home/oracle/product/11.1.0' is not owned by root
WARNING: directory '/home/oracle/product' is not owned by root
WARNING: directory '/home/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.

Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
The directory '/home/oracle/product/11.1.0' is not owned by root. Changing owner to root
The directory '/home/oracle/product' is not owned by root. Changing owner to root
The directory '/home/oracle' is not owned by root. Changing owner to root
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: oradbdev01 oradbdev01-priv oradbdev01
node 2: oradbdev02 oradbdev02-priv oradbdev02
clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
oradbdev01
oradbdev02
Cluster Synchronization Services is active on all the nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps

Creating VIP application resource on (2) nodes...
Creating GSD application resource on (2) nodes...
Creating ONS application resource on (2) nodes...
Starting VIP application resource on (2) nodes...
Starting GSD application resource on (2) nodes...
Starting ONS application resource on (2) nodes...


Done.

Oracle Run vipca and other configurations: Press Next after it completes:

From For Blog

Final Window - Installation Completed!!! - Press Exit:

From For Blog

[oracle@oradbdev01 bin]$ ./crsctl check crs
Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthy

[oracle@oradbdev02 bin]$ ./crsctl check crs
Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthy

This shows that crs installation was successful on both the nodes!!