ASM rebalancing commands

For ASM rebalancing good commands always come handy. This one shows current ASM disks with group info and mount status with very well formed way.

 column path format a20  
 set lines 132  
 set pages 50  
 select path, group_number group_#, disk_number disk_#, mount_status,  
 header_status, state, total_mb, free_mb  
 from v$asm_disk   
 order by group_number;  

The next one does good formatting for showing disks in failgroup and diskgroup

 SET ECHO    OFF  
 SET FEEDBACK  6  
 SET HEADING   ON  
 SET LINESIZE  180  
 SET PAGESIZE  50000  
 SET TERMOUT   ON  
 SET TIMING   OFF  
 SET TRIMOUT   ON  
 SET TRIMSPOOL  ON  
 SET VERIFY   OFF  
 CLEAR COLUMNS  
 CLEAR BREAKS  
 CLEAR COMPUTES  
 COLUMN disk_group_name    FORMAT a25      HEAD 'Disk Group Name'  
 COLUMN disk_file_path     FORMAT a20      HEAD 'Path'  
 COLUMN disk_file_name     FORMAT a20      HEAD 'File Name'  
 COLUMN disk_file_fail_group  FORMAT a20      HEAD 'Fail Group'  
 SELECT  
   NVL(a.name, '[CANDIDATE]')            disk_group_name  
  , b.path                      disk_file_path  
  , b.name                      disk_file_name  
  , b.failgroup                   disk_file_fail_group  
 FROM  
   v$asm_diskgroup a RIGHT OUTER JOIN v$asm_disk b USING (group_number)  
 ORDER BY  
   a.name  
 /  

Adding and removing normal disks from diskgroup

 alter diskgroup DISKDG add disk 'NEWDISK' rebalance power 11;  
 alter diskgroup DISKDG drop disk 'OLDDISK' rebalance power 11;  

Adding and removing several disks from diskgroup

 alter diskgroup DISKDG add disk 'ORCL:NEWDISK', 'ORCL:NEWDISK2' rebalance power 11;  
 alter diskgroup DISKDG drop disk 'OLDDISK', 'OLDIDSK2' rebalance power 11;  

Adding disks into failgroup

 ALTER DISKGROUP FAILDISKDG ADD   
 FAILGROUP FAILONE DISK 'ORCL:NEWDISK3', 'ORCL:NEWDISK4'  
 FAILGROUP FAILTWO DISK 'ORCL:NEWDISK5', 'ORCL:NEWDISK6'  
 REBALANCE POWER 11;  

Checking status of ASM rebalance (remember to check ASM alert log, this will only give estimate of rebalancing)

 SELECT group_number, operation, state, power, est_minutes FROM v$asm_operation;  

Create new ASM diskgroup for voting disk

1) Create new diskgroup for ASM diskgroup

CREATE DISKGROUP VOTEVPLEX EXTERNAL REDUNDANCY DISK 'ORCL:VOTEVPLEX' ATTRIBUTE 'au_size'='4M';

alter diskgroup VOTEVPLEX set attribute 'compatible.asm'='11.2;

2) Migrate voting file to new ASM diskgroup

[root@rac1 ~]# su - oragrid

+ASM1_rac1 $ sqlplus / as sysasm

SQL> CREATE DISKGROUP VOTEVPLEX EXTERNAL REDUNDANCY DISK 'ORCL:VOTEVPLEX' ATTRIBUTE 'au_size'='4M';

SQL> alter diskgroup VOTEVPLEX set attribute 'compatible.asm'='11.2';

Diskgroup altered.


Diskgroup created.

+ASM1_rac1 $ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       4384
         Available space (kbytes) :     257736
         ID                       :   70121363
         Device/File Name         :     +OCRDG
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user


+ASM1_rac1 $ crsctl   query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   12e224acaa594fcfbf245fd6486e1ed0 (ORCL:VOTEDC1) [VOTEDG]
 2. ONLINE   355415a275c64fabbf0159a43d7bac62 (ORCL:VOTEDC2) [VOTEDG]
Located 2 voting disk(s).

3) Check results

+ASM1_rac1 $ crsctl  replace votedisk +VOTEVPLEX
Successful addition of voting disk 0d588bf13f684f32bfb3d45688c28826.
Successful deletion of voting disk 12e224acaa594fcfbf245fd6486e1ed0.
Successful deletion of voting disk 355415a275c64fabbf0159a43d7bac62.
Successfully replaced voting disk group with +VOTEVPLEX.
CRS-4266: Voting file(s) successfully replaced

+ASM1_rac1 $ crsctl   query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   0d588bf13f684f32bfb3d45688c28826 (ORCL:VOTEVPLEX) [VOTEVPLEX]

How to configure Multipath disks for Oracle ASM use Step-By-Step

0) Scan your disks

for i in 0 1 2 3 4 5 6; do echo "- - -" > /sys/class/scsi_host/host$i/scan; done

0,5) Find your disks

[root@rac1 ~]# multipath -ll

1) On both nodes, change oracleasm configuration file 

[root@rac1 ~]# nano -w /etc/sysconfig/oracleasm
[root@rac2 ~]# nano -w /etc/sysconfig/oracleasm

Add mpath

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="mpath dm"

2) Format disk and check output on both nodes

[root@rac1 ~]# fdisk /dev/mapper/mpathjl

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel with disk identifier 0x35dd781b.

Changes will remain in memory only, until you decide to write them.

After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to

         switch off the mode (command 'c') and change display units to

         sectors (command 'u').

Command (m for help): p

Disk /dev/mapper/mpathjl: 1074 MB, 1074462720 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x35dd781b

              Device Boot      Start         End      Blocks   Id  System

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-130, default 1):

Using default value 1

Last cylinder, +cylinders or +size{K,M,G} (1-130, default 130):

Using default value 130

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.

The kernel still uses the old table. The new table will be used at

the next reboot or after you run partprobe(8) or kpartx(8)

Syncing disks.

[root@rac1 ~]# fdisk -l /dev/mapper/mpathjl

Disk /dev/mapper/mpathjl: 1074 MB, 1074462720 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x35dd781b

              Device Boot      Start         End      Blocks   Id  System

/dev/mapper/mpathjlp1               1         130     1044193+  83  Linux

[root@rac2 ~]# fdisk -l /dev/mapper/mpathjl

Disk /dev/mapper/mpathjl: 1074 MB, 1074462720 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x35dd781b

              Device Boot      Start         End      Blocks   Id  System

/dev/mapper/mpathjlp1               1         130     1044193+  83  Linux

2.1) Copy fdisk partition

sfdisk -d mpathjl > /tmp/50GB_LUN.txt

2.2) Apply fdisk partition to empty LUN

sfdisk mpath < /tmp/50GB_LUN.txt

for i in $(cat /tmp/100gb_mpath); do sfdisk $i < /tmp/mpath100gb.txt done

3) Change ownership of mpath disk

[root@rac1 ~]# fdisk -l "/dev/mapper/mpathjl"

Disk /dev/mapper/mpathjl: 1074 MB, 1074462720 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x35dd781b

              Device Boot      Start         End      Blocks   Id  System

/dev/mapper/mpathjlp1               1         130     1044193+  83  Linux

[root@rac1 ~]# chown oragrid:oinstall /dev/mapper/mpathjlp1

[root@rac2 ~]# chown oragrid:oinstall /dev/mapper/mpathjlp1

[root@rac1 ~]# ls -l /dev/dm-254

brw-rw---- 1 oragrid oinstall 252, 254 Nov 13 18:51 /dev/dm-254

[root@rac1 ~]# ls -l /dev/dm-254

brw-rw---- 1 oragrid oinstall 252, 254 Nov 13 18:51 /dev/dm-254

3.5) If second node does not see the disks you might need to use partprobe

E.g partprobe /dev/mapper/mpathjlp1

4) Create ASM disk

[root@rac1 ~]# /etc/init.d/oracleasm  createdisk  VOTEVPLEX /dev/mapper/mpathjlp1

Marking disk "VOTEVPLEX" as an ASM disk:                   [  OK  ]

[root@rac1 ~]# /etc/init.d/oracleasm  listdisks

[root@rac2 ~]# /etc/init.d/oracleasm  scandisks

Scanning the system for Oracle ASMLib disks:               [  OK  ]

[root@rac2 ~]# /etc/init.d/oracleasm  listdisks

5) Check ASM disk

[root@rac1 ~]# /etc/init.d/oracleasm querydisk -d VOTEVPLEX

Disk "VOTEVPLEX" is a valid ASM disk on device [252,254]

[root@rac2 ~]# /etc/init.d/oracleasm querydisk -d VOTEVPLEX

Disk "VOTEVPLEX" is a valid ASM disk on device [252,254]

[root@rac1 ~]# oracleasm querydisk /dev/dm-254

Device "/dev/dm-254" is marked an ASM disk with the label "VOTEVPLEX"

[root@rac2 ~]# oracleasm querydisk /dev/dm-254

Device "/dev/dm-254" is marked an ASM disk with the label "VOTEVPLEX"

[root@rac1 ~]# multipath -ll mpathjl

mpathjl (36000144000000010303f2aae3350e2e2) dm-253 XXX

size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  |- 1:0:6:106 sdob 128:368 active ready running

  |- 1:0:7:106 sdoc 128:384 active ready running

  |- 1:0:8:106 sdod 128:400 active ready running

  `- 1:0:9:106 sdoe 128:416 active ready running

[root@rac2 ~]# multipath -ll mpathjl

Nov 13 20:00:58 | multipath.conf line 4, invalid keyword: selector

Nov 13 20:00:58 | multipath.conf line 7, invalid keyword: prio_callout

mpathjl (36000144000000010303f2aae3350e2e2) dm-253 XXX

size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  |- 1:0:4:106 sdto 65:608  active ready running

  |- 2:0:8:106 sdts 65:672  active ready running

  |- 1:0:5:106 sdtp 65:624  active ready running

  |- 2:0:6:106 sdtt 65:688  active ready running

  |- 1:0:6:106 sdtr 65:656  active ready running

  |- 2:0:7:106 sdtu 65:704  active ready running

  `- 1:0:7:106 sdtq 65:640  active ready running

How to move disks inside ASM diskgroups

Simple commands (without any details go as follow). Modify the datafile names to suit your own needs. These files do not use OMF and hence full filename needs to said during RMAN commands.

Locate file which needs to be moved;
SQL> SELECT file_name FROM dba_data_files;


Take the file offline;
SQL> ALTER DATABASE DATAFILE '+DATA/database/datafile/db_name_data01.dbf' OFFLINE;


Log on to RMAN
OS> rman target / 


Copy the datafile to new location
RMAN> COPY DATAFILE '+DATA/database/datafile/db_name_data01.dbf' TO '+DATA_DISK_GROUP_1/database/datafile/db_name_data01.dbf';


Log on to SQL to rename the file again
SQL> ALTER DATABASE RENAME FILE '+DATA/database/datafile/db_name_data01.dbf' TO '+DATA_DISK_GROUP_1/database/datafile/db_name_data01.dbf';


Log on to RMAN
OS> rman target /

Switch datafile
RMAN> SWITCH DATAFILE '+DATA_DISK_GROUP_1/database/datafile/db_name_data01.dbf' TO COPY;

Log on to SQL  to recover and put the file online again.

SQL> RECOVER DATAFILE '+DATA_DISK_GROUP_1/database/datafile/db_name_data01.dbf';
SQL> ALTER DATABASE DATAFILE '+DATA_DISK_GROUP_1/database/datafile/db_name_data01.dbf' ONLINE;

Done.

Adding ASM disks through SQL*Plus

List ASM diskgroup and make note of diskgroup name

SQL> !cat list_asm_diskgroups.sql
set lines 200
column GROUP_NUMBER format a30
column NAME format a20
column SECTOR_SIZE format a20
column STATE format a10
column TYPE format a10
SELECT GROUP_NUMBER as NO, NAME,SECTOR_SIZE as SS ,BLOCK_SIZE,ALLOCATION_UNIT_SIZE,STATE,TYPE,TOTAL_MB,FREE_MB FROM V$ASM_DISKGROUP;

SQL> @list_asm_diskgroups.sql

        NO NAME                         SS BLOCK_SIZE ALLOCATION_UNIT_SIZE STATE      TYPE         TOTAL_MB    FREE_MB

---------- -------------------- ---------- ---------- -------------------- ---------- ---------- ---------- ----------

         1 GRID                        512       4096              1048576 MOUNTED    NORMAL          30720      29794

         2 XXPROD_DATA                 512       4096              4194304 MOUNTED    EXTERN        2048000    1202260

         3 XXPROD_FRA                  512       4096              4194304 MOUNTED    EXTERN         102400      88680

List possible candidate disks

SQL> !cat show_asm_candidates.sql

column name format a20

column header_status format a10

column path format a30

SELECT name, header_status, path FROM V$ASM_DISK

WHERE HEADER_STATUS='CANDIDATE';

SQL> @show_asm_candidates.sql

NAME                 HEADER_STA PATH

-------------------- ---------- ------------------------------

                     CANDIDATE  /dev/rhdisk56

                     CANDIDATE  /dev/rhdisk55

Add candidate ASM disks by issuing following commands

SQL> ALTER DISKGROUP XXPROD_DATA ADD DISK '/dev/rhdisk55' NAME XXPROD_FRA_0001 SIZE 102400 M REBALANCE POWER 11;

Diskgroup altered.

SQL> ALTER DISKGROUP XXPROD_DATA ADD DISK '/dev/rhdisk56' NAME XXPROD_FRA_0002 SIZE 102400 M REBALANCE POWER 11;

Done.