Oracle 11gR2 Grid Infrastructure Installation Steps

Step By Step Installation 11gR2 Non ASM
========================================
Connect as root user:
1.) groupadd dba
2.) useradd -d /home/oracle -g dba -G dba oracle
3.) Create directories for oracle grid installation, the directory path should not be same as $ORACLE_BASE. Assuming $ORACLE_BASE=/oracle.
mkdir /oragrid
Make sure the directory as atleast 5GB free space.
4.) Change the owner of the directory to oracle:dba
chown -R oracle:dba /oragrid
5.) RPM requirements (From Metalink note# 880989.1:

1.) binutils-2.17.50.0.6-6.el5 (x86_64)
2.) compat-libstdc++-33-3.2.3-61 (x86_64) <<< both ARCH's are required. See next line.
3.) compat-libstdc++-33-3.2.3-61 (i386) <<< both ARCH's are required. See previous line.
4.) elfutils-libelf-0.125-3.el5 (x86_64)
5.) glibc-2.5-24 (x86_64) <<< both ARCH's are required. See next line.
6.) glibc-2.5-24 (i686) <<< both ARCH's are required. See previous line.
7.) glibc-common-2.5-24 (x86_64)
8.) ksh-20060214-1.7 (x86_64)
9.) libaio-0.3.106-3.2 (x86_64) <<< both ARCH's are required. See next line.
10.) libaio-0.3.106-3.2 (i386) <<< both ARCH's are required. See previous line.
11.) libgcc-4.1.2-42.el5 (i386) <<< both ARCH's are required. See next line.
12.) libgcc-4.1.2-42.el5 (x86_64) <<< both ARCH's are required. See previous line.
13.) libstdc++-4.1.2-42.el5 (x86_64) <<< both ARCH's are required. See next line.
14.) libstdc++-4.1.2-42.el5 (i386) <<< both ARCH's are required. See previous line.
15.) make-3.81-3.el5 (x86_64)
16.) elfutils-libelf-devel-0.125-3.el5.x86_64.rpm
a.) requires elfutils-libelf-devel-static-0.125-3.el5.x86_64.rpm as a prerequisite, as listed below.
b.) elfutils-libelf-devel and elfutils-libelf-devel-static each depend upon the other. Therefore, they must be installed together, in one (1) "rpm -ivh" command as follows:
rpm -ivh elfutils-libelf-devel-0.125-3.el5.x86_64.rpm elfutils-libelf-devel-static-0.125-3.el5.x86_64.rpm
17.) glibc-headers-2.5-24.x86_64.rpm
a.) requires kernel-headers-2.6.18-92.el5.x86_64.rpm as a prerequisite, as listed below
18.) glibc-devel-2.5-24.x86_64.rpm <<< both ARCH's are required. See next item.
19.) glibc-devel-2.5-24.i386.rpm <<< both ARCH's are required. See previous item.
20.) gcc-4.1.2-42.el5.x86_64.rpm
a.) requires libgomp-4.1.2-42.el5.x86_64.rpm as a prerequisite, as listed below
21.) libstdc++-devel-4.1.2-42.el5.x86_64.rpm
22.) gcc-c++-4.1.2-42.el5.x86_64.rpm
23.) libaio-devel-0.3.106-3.2.x86_64.rpm <<< both ARCH's are required. See next item
24.) libaio-devel-0.3.106-3.2.i386.rpm <<< both ARCH's are required. See previous item.
25.) sysstat-7.0.2-1.el5.x86_64.rpm
26.) unixODBC-2.2.11-7.1.x86_64.rpm <<< both ARCH's are required. See next item
27.) unixODBC-2.2.11-7.1.i386.rpm <<< both ARCH's are required. See previous item.
28.) unixODBC-devel-2.2.11-7.1.x86_64.rpm <<< both ARCH's are required. See next item
29.) unixODBC-devel-2.2.11-7.1.i386.rpm <<< both ARCH's are required. See previous item.

Command to check:

rpm -qa |grep -E "binutils|compat-libstdc++|elfutils-libelf-|glibc-|glibc-common-|ksh-|libaio-|libgcc-|libstdc++|make|gcc|sysstat|unixODBC"

6.) Values for /etc/sysctl.conf (these are suggested, these values should be calculated with respect to the memory/cpu/processes of the server.)

kernel.shmall = physical RAM size / pagesize For most systems
kernel.shmmax = 1/2 of physical RAM eg. for 32g ram it should be 17179869184.
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 409200
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

7.) Make sure u have configured a scan IP, for more details on what is scan IP please refer to: http://www.oracle.com/technetwork/database/clustering/overview/scan-129069.pdf
Make sure your /etc/hosts file has entries or hostname, host IP address, Interconnect name, Interconnect IP address, virtual hostname, virtual host IP address for all RAC nodes and on all RAC nodes.

8a.) Since we are using Network Time Protocol (NTP) for synchronization of time across all the servers in the cluster, a mandatory requirement with 11gR2 is to enable the slewing option by adding ‘-x’ argument in the ntp configuration file as seen below: -
# vi /etc/sysconfig/ntpd

# Drop root to id 'ntp:ntp' by default.
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid" #add -x in the options string.
# Set to 'yes' to sync hw clock after successful ntpdate
SYNC_HWCLOCK=no
# Additional options for ntpdate
NTPDATE_OPTIONS=""

Restart ntpd daemon.
[root@node1 oracle]# /etc/init.d/ntpd stop
Shutting down ntpd: [ OK ]
[root@node1 oracle]# /etc/init.d/ntpd start
ntpd: Synchronizing with time server: [ OK ]
Starting ntpd: [ OK ]

8b.) Update /etc/security/limits.conf with:

oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

9.) Add following line in the /etc/pam.d/login file:

session required pam_limits.so

10.) Add the following lines to /etc/profile:

if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -u 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi

11.) Setup user equivalence between the oracle user of the rac nodes.
12.) Run Cluverify (change directory where you have copied and unzipped the oracle 11gr2 grid software)
#./runcluvfy.sh stage -pre crsinst -n node1,node2 -verbose

--Fix any error you see.

13.) Start ./runInstaller
Screenshots:



Welcome Screen

14.)


Select Advanced Installation

15.)


Select all required languages, I have selected only English

16.)


Specify scan name, cluster name, and scan listener port.

17.)


Provide RAC node information, along with virtual name.

18.)


Review is the the interfaces, subnet mask are correct and select the public/private accordingly.(private is for interconnect).


19.)


Oracle will do some validation checks

20.)


Select ASM/shared file system, depending on what u are using. I am using shared file system.

21.)


Select location for OCR.

22.)


Select location of voting disks.

23.)


Failure Support, I dont plan to use this.

24.)


Select the OS groups.

25.)


Warning message, ignore it

26.)


Specify oracle_base and grid installation folder.

27.)


Specify Oracle Inventory Location.

28.)


Oracle perform pre-checks. I am ignoring swap space issue.

29.)


Final summary page!!

30.)


Ignore this warning

31.)


Shows installation Progress

32.)


Run the scripts

33.) Run /oracle/oraInventory/orainstRoot.sh

[root@node1 oraInventory]# ./orainstRoot.sh
Changing permissions of /oracle/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /oracle/oraInventory to dba.
The execution of the script is complete.


34.) Run $GRID_HOME/root.sh

[root@node1 11.2]# ./root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oragrid/product/11.2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-08-27 23:25:19: Parsing the host name
2010-08-27 23:25:19: Checking for super user privileges
2010-08-27 23:25:19: User has super user privileges
Using configuration parameter file: /oragrid/product/11.2/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'node1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'node1'
CRS-2676: Start of 'ora.gipcd' on 'node1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'node1'
CRS-2676: Start of 'ora.gpnpd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node1'
CRS-2676: Start of 'ora.cssdmonitor' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node1'
CRS-2672: Attempting to start 'ora.diskmon' on 'node1'
CRS-2676: Start of 'ora.diskmon' on 'node1' succeeded
CRS-2676: Start of 'ora.cssd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'node1'
CRS-2676: Start of 'ora.ctssd' on 'node1' succeeded
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'node1'
CRS-2676: Start of 'ora.crsd' on 'node1' succeeded
Now formatting voting disk: /u05/cludata/votedisk1.
Now formatting voting disk: /u01/oradata/orcl/cludata/votedisk2.
Now formatting voting disk: /u02/oradata/orcl/cludata/votedisk3.
CRS-4603: Successful addition of voting disk /u05/cludata/votedisk1.
CRS-4603: Successful addition of voting disk /u01/oradata/orcl/cludata/votedisk2.
CRS-4603: Successful addition of voting disk /u02/oradata/orcl/cludata/votedisk3.
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a893117617e74ffebfd7a379396f281d (/u05/cludata/votedisk1) []
2. ONLINE 018eafc00dfd4fa9bf568fa245581dcf (/u01/oradata/orcl/cludata/votedisk2) []
3. ONLINE 0460f0a8560d4f0abf624c47870654fd (/u02/oradata/orcl/cludata/votedisk3) []
Located 3 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'node1'
CRS-2677: Stop of 'ora.crsd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'node1'
CRS-2677: Stop of 'ora.ctssd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'node1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node1'
CRS-2677: Stop of 'ora.cssd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1'
CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node1'
CRS-2677: Stop of 'ora.gipcd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node1'
CRS-2677: Stop of 'ora.mdnsd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.mdnsd' on 'node1'
CRS-2676: Start of 'ora.mdnsd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'node1'
CRS-2676: Start of 'ora.gipcd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'node1'
CRS-2676: Start of 'ora.gpnpd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node1'
CRS-2676: Start of 'ora.cssdmonitor' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node1'
CRS-2672: Attempting to start 'ora.diskmon' on 'node1'
CRS-2676: Start of 'ora.diskmon' on 'node1' succeeded
CRS-2676: Start of 'ora.cssd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'node1'
CRS-2676: Start of 'ora.ctssd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'node1'
CRS-2676: Start of 'ora.crsd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'node1'
CRS-2676: Start of 'ora.evmd' on 'node1' succeeded

node1 2010/08/27 23:36:53 /oragrid/product/11.2/cdata/node1/backup_20100827_233653.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 12287 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /oracle/oraInventory
'UpdateNodeList' was successful.


Run the same on node2


[root@node2 11.2]# ./root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oragrid/product/11.2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-08-27 23:44:15: Parsing the host name
2010-08-27 23:44:15: Checking for super user privileges
2010-08-27 23:44:15: User has super user privileges
Using configuration parameter file: /oragrid/product/11.2/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node node1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'node2'
CRS-2676: Start of 'ora.mdnsd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'node2'
CRS-2676: Start of 'ora.gipcd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'node2'
CRS-2676: Start of 'ora.gpnpd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node2'
CRS-2676: Start of 'ora.cssdmonitor' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node2'
CRS-2672: Attempting to start 'ora.diskmon' on 'node2'
CRS-2676: Start of 'ora.diskmon' on 'node2' succeeded
CRS-2676: Start of 'ora.cssd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'node2'
CRS-2676: Start of 'ora.ctssd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'node2'
CRS-2676: Start of 'ora.crsd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'node2'
CRS-2676: Start of 'ora.evmd' on 'node2' succeeded

node2 2010/08/27 23:47:32 /oragrid/product/11.2/cdata/node2/backup_20100827_234732.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 12287 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /oracle/oraInventory
'UpdateNodeList' was successful.


Press "OK" after running all the scripts on all the nodes.

35.)


I received the above error, this can be fixed later. Refer to the post
Oracle: Failed to initialize GPnP

Pres "OK" and then press "Skip" on the original screen.



36.)


Oracle 11gr2 Grid Infrastructure for a 2 Node RAC has completed successfully.