How to configure Multipath disks for Oracle ASM use Step-By-Step

0) Scan your disks

for i in 0 1 2 3 4 5 6; do echo "- - -" > /sys/class/scsi_host/host$i/scan; done

0,5) Find your disks

[root@rac1 ~]# multipath -ll

1) On both nodes, change oracleasm configuration file 

[root@rac1 ~]# nano -w /etc/sysconfig/oracleasm
[root@rac2 ~]# nano -w /etc/sysconfig/oracleasm

Add mpath

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="mpath dm"

2) Format disk and check output on both nodes

[root@rac1 ~]# fdisk /dev/mapper/mpathjl

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel with disk identifier 0x35dd781b.

Changes will remain in memory only, until you decide to write them.

After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to

         switch off the mode (command 'c') and change display units to

         sectors (command 'u').

Command (m for help): p

Disk /dev/mapper/mpathjl: 1074 MB, 1074462720 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x35dd781b

              Device Boot      Start         End      Blocks   Id  System

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-130, default 1):

Using default value 1

Last cylinder, +cylinders or +size{K,M,G} (1-130, default 130):

Using default value 130

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.

The kernel still uses the old table. The new table will be used at

the next reboot or after you run partprobe(8) or kpartx(8)

Syncing disks.

[root@rac1 ~]# fdisk -l /dev/mapper/mpathjl

Disk /dev/mapper/mpathjl: 1074 MB, 1074462720 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x35dd781b

              Device Boot      Start         End      Blocks   Id  System

/dev/mapper/mpathjlp1               1         130     1044193+  83  Linux

[root@rac2 ~]# fdisk -l /dev/mapper/mpathjl

Disk /dev/mapper/mpathjl: 1074 MB, 1074462720 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x35dd781b

              Device Boot      Start         End      Blocks   Id  System

/dev/mapper/mpathjlp1               1         130     1044193+  83  Linux

2.1) Copy fdisk partition

sfdisk -d mpathjl > /tmp/50GB_LUN.txt

2.2) Apply fdisk partition to empty LUN

sfdisk mpath < /tmp/50GB_LUN.txt

for i in $(cat /tmp/100gb_mpath); do sfdisk $i < /tmp/mpath100gb.txt done

3) Change ownership of mpath disk

[root@rac1 ~]# fdisk -l "/dev/mapper/mpathjl"

Disk /dev/mapper/mpathjl: 1074 MB, 1074462720 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x35dd781b

              Device Boot      Start         End      Blocks   Id  System

/dev/mapper/mpathjlp1               1         130     1044193+  83  Linux

[root@rac1 ~]# chown oragrid:oinstall /dev/mapper/mpathjlp1

[root@rac2 ~]# chown oragrid:oinstall /dev/mapper/mpathjlp1

[root@rac1 ~]# ls -l /dev/dm-254

brw-rw---- 1 oragrid oinstall 252, 254 Nov 13 18:51 /dev/dm-254

[root@rac1 ~]# ls -l /dev/dm-254

brw-rw---- 1 oragrid oinstall 252, 254 Nov 13 18:51 /dev/dm-254

3.5) If second node does not see the disks you might need to use partprobe

E.g partprobe /dev/mapper/mpathjlp1

4) Create ASM disk

[root@rac1 ~]# /etc/init.d/oracleasm  createdisk  VOTEVPLEX /dev/mapper/mpathjlp1

Marking disk "VOTEVPLEX" as an ASM disk:                   [  OK  ]

[root@rac1 ~]# /etc/init.d/oracleasm  listdisks

[root@rac2 ~]# /etc/init.d/oracleasm  scandisks

Scanning the system for Oracle ASMLib disks:               [  OK  ]

[root@rac2 ~]# /etc/init.d/oracleasm  listdisks

5) Check ASM disk

[root@rac1 ~]# /etc/init.d/oracleasm querydisk -d VOTEVPLEX

Disk "VOTEVPLEX" is a valid ASM disk on device [252,254]

[root@rac2 ~]# /etc/init.d/oracleasm querydisk -d VOTEVPLEX

Disk "VOTEVPLEX" is a valid ASM disk on device [252,254]

[root@rac1 ~]# oracleasm querydisk /dev/dm-254

Device "/dev/dm-254" is marked an ASM disk with the label "VOTEVPLEX"

[root@rac2 ~]# oracleasm querydisk /dev/dm-254

Device "/dev/dm-254" is marked an ASM disk with the label "VOTEVPLEX"

[root@rac1 ~]# multipath -ll mpathjl

mpathjl (36000144000000010303f2aae3350e2e2) dm-253 XXX

size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  |- 1:0:6:106 sdob 128:368 active ready running

  |- 1:0:7:106 sdoc 128:384 active ready running

  |- 1:0:8:106 sdod 128:400 active ready running

  `- 1:0:9:106 sdoe 128:416 active ready running

[root@rac2 ~]# multipath -ll mpathjl

Nov 13 20:00:58 | multipath.conf line 4, invalid keyword: selector

Nov 13 20:00:58 | multipath.conf line 7, invalid keyword: prio_callout

mpathjl (36000144000000010303f2aae3350e2e2) dm-253 XXX

size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  |- 1:0:4:106 sdto 65:608  active ready running

  |- 2:0:8:106 sdts 65:672  active ready running

  |- 1:0:5:106 sdtp 65:624  active ready running

  |- 2:0:6:106 sdtt 65:688  active ready running

  |- 1:0:6:106 sdtr 65:656  active ready running

  |- 2:0:7:106 sdtu 65:704  active ready running

  `- 1:0:7:106 sdtq 65:640  active ready running