Installing and Configuring Oracle ASMlib
Oracle ASMlib is software provided to ease the configuration of Disk Devices.
Please download the library files from http://www.oracle.com/technology/tech/linux/asmlib/index.html
The library are available with respect to your OS and OS version. The current version of ASMlib is 2.0. With respect to my operating system I downloaded:
oracleasm-support-2.0.3-1.x86_64.rpm
oracleasmlib-2.0.2-1.x86_64.rpm
oracleasm-2.6.9-55.ELsmp-2.0.3-1.x86_64.rpm
oracleasm-2.6.9-55.EL-2.0.3-1.x86_64.rpm
Installation of these binaries:
rpm –Uvh oracleasm-support-2.0.3-1.x86_64.rpm
rpm –Uvh oracleasm-2.6.9-55.EL-2.0.3-1.x86_64.rpm
rpm –Uvh oracleasmlib-2.0.2-1.x86_64.rpm
rpm –Uvh oracleasm-2.6.9-55.ELsmp-2.0.3-1.x86_64.rpm
Install these binaries on all RAC nodes.
Initialize oracleasm (All Nodes):
# /etc/init.d/oracleasm configure
Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]
...
Note: Please make sure to enter default user and group to own the driver interface.
Creation of Disk Partitions and Oracle ASM Disk:
We are using 1 lun of 2GB to create partitions. We will create partitions for OCR, Voting Disk, and other database files.
You can also create separate luns for OCR, Voting Disk and DB files.
As we are using iSCSI devices we do not need to create disk partitions we can directly use /dev/sd* devices or as we are using multipath devices, we can directly use /dev/mpath/mpath*
For example sake I am creating disk partitions for this installation:
[root@oradbdev01 mpath]# fdisk /dev/dm-8 à LUN (multipath device)
Command (m for help): p
Disk /dev/dm-8: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes
Device Boot Start End Blocks Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1009, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1009, default 1009): +128M àOCR FILE
Command (m for help): p
Disk /dev/dm-8: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes
Device Boot Start End Blocks Id System
/dev/dm-8p1 1 61 126666 83 Linux
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (62-1009, default 62):
Using default value 62
Last cylinder or +size or +sizeM or +sizeK (62-1009, default 1009): +128M àOCR FILE
Command (m for help): p
Disk /dev/dm-8: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes
Device Boot Start End Blocks Id System
/dev/dm-8p1 1 61 126666 83 Linux
/dev/dm-8p2 62 122 126697 83 Linux
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (123-1009, default 123):
Using default value 123
Last cylinder or +size or +sizeM or +sizeK (123-1009, default 1009): +1000M àData File
Command (m for help): p
Disk /dev/dm-8: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes
Device Boot Start End Blocks Id System
/dev/dm-8p1 1 61 126666 83 Linux
/dev/dm-8p2 62 122 126697 83 Linux
/dev/dm-8p3 123 593 978267 83 Linux
Command (m for help): n
Command action
e extended
p primary partition (1-4)
e
Selected partition 4
First cylinder (594-1009, default 594):
Using default value 594
Last cylinder or +size or +sizeM or +sizeK (594-1009, default 1009):
Using default value 1009 à Creating extended partition of the remaing space.
Command (m for help): p
Disk /dev/dm-8: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes
Device Boot Start End Blocks Id System
/dev/dm-8p1 1 61 126666 83 Linux
/dev/dm-8p2 62 122 126697 83 Linux
/dev/dm-8p3 123 593 978267 83 Linux
/dev/dm-8p4 594 1009 864032 5 Extended
Command (m for help): n
First cylinder (594-1009, default 594):
Using default value 594
Last cylinder or +size or +sizeM or +sizeK (594-1009, default 1009): +20M àVoting Disk
Command (m for help): n
First cylinder (604-1009, default 604):
Using default value 604
Last cylinder or +size or +sizeM or +sizeK (604-1009, default 1009): +20M àVoting Disk
Command (m for help): n
First cylinder (614-1009, default 614): +20M
Value out of range.
First cylinder (614-1009, default 614):
Using default value 614
Last cylinder or +size or +sizeM or +sizeK (614-1009, default 1009): +20M àVoting Disk
Command (m for help): n
First cylinder (624-1009, default 624):
Using default value 624
Last cylinder or +size or +sizeM or +sizeK (624-1009, default 1009):
Using default value 1009 à Remaining space for Database files
Command (m for help): p
Disk /dev/dm-8: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes
Device Boot Start End Blocks Id System
/dev/dm-8p1 1 61 126666 83 Linux àOCR FILE
/dev/dm-8p2 62 122 126697 83 Linux àOCR FILE
/dev/dm-8p3 123 593 978267 83 Linux àDatabase
/dev/dm-8p4 594 1009 864032 5 Extended
/dev/dm-8p5 594 603 20739 83 Linux àVoting Disk
/dev/dm-8p6 604 613 20739 83 Linux àVoting Disk
/dev/dm-8p7 614 623 20739 83 Linux àVoting Disk
/dev/dm-8p8 624 1009 801691 83 Linux àDatabase
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
Reboot all nodes to read the partition table.
The devices will look like this:
[oracle@oradbdev01 ~]$ ls -lrt /dev/mpath/mpath8*
lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8 -> ../dm-0
lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8p2 -> ../dm-4
lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8p6 -> ../dm-8
lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8p1 -> ../dm-2
lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8p5 -> ../dm-7
lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8p7 -> ../dm-9
lrwxrwxrwx 1 root root 8 Apr 10 15:02 /dev/mpath/mpath8p8 -> ../dm-10
lrwxrwxrwx 1 root root 7 Apr 10 15:02 /dev/mpath/mpath8p3 -> ../dm-6
Now add database disk devices to oracle asm:
[root@oradbdev01 mpath]# /etc/init.d/oracleasm createdisk DISK3 /dev/mpath/mpath8p3;
Marking disk "/dev/mpath/mpath8p3" as an ASM disk: [ OK ]
[root@oradbdev01 mpath]# /etc/init.d/oracleasm createdisk DISK4 /dev/mpath/mpath8p8;
Marking disk "/dev/mpath/mpath8p8" as an ASM disk: [ OK ]
Run the command "/etc/init.d/oracleasm scandisks" on all other nodes so that ASM disks will show up on all other nodes.
[root@oradbdev02 ~]# /etc/init.d/oracleasm scandisks
Scanning system for ASM disks: [ OK ]
Run the command /etc/init.d/oracleasm listdisks and /etc/init.d/oracleasm querydisk on all nodes to verify the disks has been marked as ASM disks. For example:
[oracle@oradbdev01 ~]$ ssh oradbdev02 /etc/init.d/oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
[oracle@oradbdev01 ~]$ /etc/init.d/oracleasm querydisk DISK3
Disk "DISK3" is a valid ASM disk on device [8, 19]
[oracle@oradbdev01 ~]$ ssh oradbdev02 /etc/init.d/oracleasm querydisk DISK3
Disk "DISK3" is a valid ASM disk on device [8, 19]
[oracle@oradbdev01 ~]$ /etc/init.d/oracleasm querydisk DISK4
Disk "DISK4" is a valid ASM disk on device [8, 24]
[oracle@oradbdev01 ~]$ ssh oradbdev02 /etc/init.d/oracleasm querydisk DISK4
Disk "DISK4" is a valid ASM disk on device [8, 24]
[root@oradbdev01 ~]# /etc/init.d/oracleasm querydisk /dev/mpath/mpath8p3
Disk "/dev/mpath/mpath8p3" is marked an ASM disk with the label "DISK3"
[root@oradbdev01 ~]# /etc/init.d/oracleasm querydisk /dev/mpath/mpath8p8
Disk "/dev/mpath/mpath8p8" is marked an ASM disk with the label "DISK4"
To find out the exact disk reflected in the output of oracleasm, use the entries in /proc/partitions file.
Configuring ASMlib on Multipath Disk:
[root@oradbdev01 ~]# vi /etc/sysconfig/oracleasm
#
# This is a configuration file for automatic loading of the Oracle
# Automatic Storage Management library kernel driver. It is generated
# By running /etc/init.d/oracleasm configure. Please use that method
# to modify this file
#
# ORACLEASM_ENABELED: 'true' means to load the driver on boot.
ORACLEASM_ENABLED=true
# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
ORACLEASM_UID=oracle
# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
ORACLEASM_GID=dba
# ORACLEASM_SCANBOOT: 'true' means fix disk perms on boot
ORACLEASM_SCANBOOT=true
# ORACLEASM_CLEARBOOT: 'true' means clean old disk perms on boot
ORACLEASM_CLEARBOOT=true
# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="dm" ##Updated it to “dm”
# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd" ##Updated it to “sd”
Make these changes in all the nodes.
Preparing & Configuring OCR and Voting Disk on Raw devices using Multipath or Udev:
On Linux platform, the LUNs presented from NAS or SCSI devices are identified with the device name (or path) /dev/sdx (/dev/sda, dev/sdb, etc). A common issue is that the device names (including multipath device name like /dev/mpath/mpath0 if multipath is enabled) can be changed when some disks are removed or added (as planned or due to I/O failures), SCSI or FC is reconfigured (like adding or removing ISCSI channels) or even host is rebooted. Even these devices are bound to raw with the consistent name (like /dev/raw/raw1) and assigned to OCR and voting disks, starting CRS can still fail because of the content of the device is not the OCR or the voting disk. To address this issue, you can use either multipath software or udev utility to setup devices with consistent name for OCR and voting disks.
multipath - multipath is a native Linux dynamic multipathing/failover software. An I/O path generally consists of an initiator port, target port, and LUN. Each permutation of this I/O is considered an independent path. Dynamic multipathing/failover tools (software or hardware) aggregate these independent paths into a single logical path. This path virtualization provides I/O load-balancing and non-disruptive failover across the host bus adapters (HBAs) or network interface card (NIC). Multipath software also can assign a consistent device name to LUNs as described in next section.
More info: http://en.wikipedia.org/wiki/Multipath_I/O
udev - udev is a new feature introduced in Linux 2.6. It provides a dynamic device directory containing only the files for actually present devices. It creates or removes device node files usually located in the /dev directory, or it renames network interfaces. As part of the hotplug subsystem, udev is executed if a kernel device is added or removed from the system. On device creation, udev reads the sysfs directory of the given device to collect device attributes like label, serial number or bus device number. These attributes may be used as keys to determine a unique name for the device.
More info: http://en.wikipedia.org/wiki/Udev
Configuring OCR & Voting Disk:
1. Choose the multipath device which will hold the ocr & voting disk files.
e.g /dev/mpath/mpath8
[root@oradbdev01 dev]# multipath -d -l mpath8
mpath8 (360a9800043346d77674a486332443238)
[size=2 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [active]
\_ 0:0:0:1 sdb 8:16 [active]
2. Rediscover all multipath devices on all nodes:
Note: Please shutdown all databases and ASM running on the affect nodes, as this will remove all multipath devices.
[root@oradbdev01 dev]# multipath –F
[root@oradbdev01 dev]# multipath –v2
As we have already created paritions, for probing these new partition devices on all the other nodes run:
[root@oradbdev02 dev]# partprobe
Instead you can also do:
/etc/init.d/iscsi reload
3. Make these changes to the /etc/sysconfig/rawdevices file, these are for OCR & Voting Disk.
[root@oradbdev01 dev]# more /etc/sysconfig/rawdevices
# This file and interface are deprecated.
# Applications needing raw device access should open regular
# block devices with O_DIRECT.
# raw device bindings
# format:
#
# example: /dev/raw/raw1 /dev/sda1
# /dev/raw/raw2 8 5
/dev/raw/raw1 /dev/mpath/mpath8p1
/dev/raw/raw2 /dev/mpath/mpath8p2
/dev/raw/raw3 /dev/mpath/mpath8p3
/dev/raw/raw8 /dev/mpath/mpath8p8
/dev/raw/raw4 /dev/mpath/mpath8p5
/dev/raw/raw5 /dev/mpath/mpath8p6
/dev/raw/raw6 /dev/mpath/mpath8p7
4. Bind the disk devices to the raw devices on all nodes as follow:
/sbin/service rawdevices restart
5. Run the following command as root on all nodes so that system automatically binds the devices listed in the rawdevices file when it reboots.
/sbin/chkconfig --level 345 rawdevices on
6. Setup the owner, group and permissions on the OCR and voting disks as root on all nodes as follow:
For OCR:
chown root:dba /dev/raw/raw1
chown root:dba /dev/raw/raw2
chmod 660 /dev/raw/raw1
chmod 660 /dev/raw/raw2
For voting disks:
chown oracle:dba /dev/raw/raw4
chown oracle:dba /dev/raw/raw5
chown oracle:dba /dev/raw/raw6
chmod 660 /dev/raw/raw4
chmod 660 /dev/raw/raw5
chmod 660 /dev/raw/raw6
7. For RHEL4, add the following lines in /etc/rc.d/rc.local as root on all nodes so that system will setup the owner, group and permissions properly after reboot.
# For OCR:
chown root:dba /dev/raw/raw1
chown root:dba /dev/raw/raw2
chmod 660 /dev/raw/raw1
chmod 660 /dev/raw/raw2
# For voting disks:
chown oracle:dba /dev/raw/raw4
chown oracle:dba /dev/raw/raw5
chown oracle:dba /dev/raw/raw6
chmod 660 /dev/raw/raw4
chmod 660 /dev/raw/raw5
chmod 660 /dev/raw/raw6
8. Initialize the OCR and voting disk as root on one node as follow.
dd if=/dev/zero of=/dev/mpath/mpath8p1 bs=125829120 count=1
dd if=/dev/zero of=/dev/mpath/mpath8p2 bs=125829120 count=1
dd if=/dev/zero of=/dev/mpath/mpath8p5 bs=20971520 count=1
dd if=/dev/zero of=/dev/mpath/mpath8p6 bs=20971520 count=1
dd if=/dev/zero of=/dev/mpath/mpath8p7 bs=20971520 count=1
10. Now most importantly setting up udev permission.
Incase you do not make this changes during CRS installation you will get error.
When you would specify the OCR device location in the CRS installation page it will give error that “The specified shared raw partition /dev/raw/raw1 may not have correct permission. Verify that the partition is owned by Oracle User.”
This is a misleading error, and will make us do changes to the actual permission of the devices. I had struggled with this error for a couple of day before I realized that I need to create a file /etc/udev/permissions.d/40-udev.permissions.
[oracle@oradbdev01 logs]$ more /etc/udev/permissions.d/40-udev.permissions
# dm devices
# ocr devices --inserted by apun
raw/raw1:root:dba:660
raw/raw2:root:dba:660
# voting disk --inserted by apun
raw/raw4:oracle:dba:660
raw/raw5:oracle:dba:660
raw/raw6:oracle:dba:660
Also comment out all entries for these devices in /etc/udev/permissions.d/50-udev.permissions file. The OS & oracle needs to refer to only /etc/udev/permissions.d/40-udev.permissions file.
Make these changes on all the RAC servers.
And reboot all the nodes.
This should complete all steps prior to starting any installation. Now you can download oracle crs and database software and start installation.
Important links to refer for more information;
http://www.oracle.com/technology/tech/linux/asmlib/index.html
0 Comments:
Post a Comment