ASM rebalancing commands

For ASM rebalancing good commands always come handy. This one shows current ASM disks with group info and mount status with very well formed way.

 column path format a20  
 set lines 132  
 set pages 50  
 select path, group_number group_#, disk_number disk_#, mount_status,  
 header_status, state, total_mb, free_mb  
 from v$asm_disk   
 order by group_number;  

The next one does good formatting for showing disks in failgroup and diskgroup

 SET ECHO    OFF  
 SET FEEDBACK  6  
 SET HEADING   ON  
 SET LINESIZE  180  
 SET PAGESIZE  50000  
 SET TERMOUT   ON  
 SET TIMING   OFF  
 SET TRIMOUT   ON  
 SET TRIMSPOOL  ON  
 SET VERIFY   OFF  
 CLEAR COLUMNS  
 CLEAR BREAKS  
 CLEAR COMPUTES  
 COLUMN disk_group_name    FORMAT a25      HEAD 'Disk Group Name'  
 COLUMN disk_file_path     FORMAT a20      HEAD 'Path'  
 COLUMN disk_file_name     FORMAT a20      HEAD 'File Name'  
 COLUMN disk_file_fail_group  FORMAT a20      HEAD 'Fail Group'  
 SELECT  
   NVL(a.name, '[CANDIDATE]')            disk_group_name  
  , b.path                      disk_file_path  
  , b.name                      disk_file_name  
  , b.failgroup                   disk_file_fail_group  
 FROM  
   v$asm_diskgroup a RIGHT OUTER JOIN v$asm_disk b USING (group_number)  
 ORDER BY  
   a.name  
 /  

Adding and removing normal disks from diskgroup

 alter diskgroup DISKDG add disk 'NEWDISK' rebalance power 11;  
 alter diskgroup DISKDG drop disk 'OLDDISK' rebalance power 11;  

Adding and removing several disks from diskgroup

 alter diskgroup DISKDG add disk 'ORCL:NEWDISK', 'ORCL:NEWDISK2' rebalance power 11;  
 alter diskgroup DISKDG drop disk 'OLDDISK', 'OLDIDSK2' rebalance power 11;  

Adding disks into failgroup

 ALTER DISKGROUP FAILDISKDG ADD   
 FAILGROUP FAILONE DISK 'ORCL:NEWDISK3', 'ORCL:NEWDISK4'  
 FAILGROUP FAILTWO DISK 'ORCL:NEWDISK5', 'ORCL:NEWDISK6'  
 REBALANCE POWER 11;  

Checking status of ASM rebalance (remember to check ASM alert log, this will only give estimate of rebalancing)

 SELECT group_number, operation, state, power, est_minutes FROM v$asm_operation;  

Create new ASM diskgroup for voting disk

1) Create new diskgroup for ASM diskgroup

CREATE DISKGROUP VOTEVPLEX EXTERNAL REDUNDANCY DISK 'ORCL:VOTEVPLEX' ATTRIBUTE 'au_size'='4M';

alter diskgroup VOTEVPLEX set attribute 'compatible.asm'='11.2;

2) Migrate voting file to new ASM diskgroup

[root@rac1 ~]# su - oragrid

+ASM1_rac1 $ sqlplus / as sysasm

SQL> CREATE DISKGROUP VOTEVPLEX EXTERNAL REDUNDANCY DISK 'ORCL:VOTEVPLEX' ATTRIBUTE 'au_size'='4M';

SQL> alter diskgroup VOTEVPLEX set attribute 'compatible.asm'='11.2';

Diskgroup altered.


Diskgroup created.

+ASM1_rac1 $ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       4384
         Available space (kbytes) :     257736
         ID                       :   70121363
         Device/File Name         :     +OCRDG
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user


+ASM1_rac1 $ crsctl   query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   12e224acaa594fcfbf245fd6486e1ed0 (ORCL:VOTEDC1) [VOTEDG]
 2. ONLINE   355415a275c64fabbf0159a43d7bac62 (ORCL:VOTEDC2) [VOTEDG]
Located 2 voting disk(s).

3) Check results

+ASM1_rac1 $ crsctl  replace votedisk +VOTEVPLEX
Successful addition of voting disk 0d588bf13f684f32bfb3d45688c28826.
Successful deletion of voting disk 12e224acaa594fcfbf245fd6486e1ed0.
Successful deletion of voting disk 355415a275c64fabbf0159a43d7bac62.
Successfully replaced voting disk group with +VOTEVPLEX.
CRS-4266: Voting file(s) successfully replaced

+ASM1_rac1 $ crsctl   query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   0d588bf13f684f32bfb3d45688c28826 (ORCL:VOTEVPLEX) [VOTEVPLEX]

How to configure Multipath disks for Oracle ASM use Step-By-Step

0) Scan your disks

for i in 0 1 2 3 4 5 6; do echo "- - -" > /sys/class/scsi_host/host$i/scan; done

0,5) Find your disks

[root@rac1 ~]# multipath -ll

1) On both nodes, change oracleasm configuration file 

[root@rac1 ~]# nano -w /etc/sysconfig/oracleasm
[root@rac2 ~]# nano -w /etc/sysconfig/oracleasm

Add mpath

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="mpath dm"

2) Format disk and check output on both nodes

[root@rac1 ~]# fdisk /dev/mapper/mpathjl

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel with disk identifier 0x35dd781b.

Changes will remain in memory only, until you decide to write them.

After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to

         switch off the mode (command 'c') and change display units to

         sectors (command 'u').

Command (m for help): p

Disk /dev/mapper/mpathjl: 1074 MB, 1074462720 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x35dd781b

              Device Boot      Start         End      Blocks   Id  System

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-130, default 1):

Using default value 1

Last cylinder, +cylinders or +size{K,M,G} (1-130, default 130):

Using default value 130

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.

The kernel still uses the old table. The new table will be used at

the next reboot or after you run partprobe(8) or kpartx(8)

Syncing disks.

[root@rac1 ~]# fdisk -l /dev/mapper/mpathjl

Disk /dev/mapper/mpathjl: 1074 MB, 1074462720 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x35dd781b

              Device Boot      Start         End      Blocks   Id  System

/dev/mapper/mpathjlp1               1         130     1044193+  83  Linux

[root@rac2 ~]# fdisk -l /dev/mapper/mpathjl

Disk /dev/mapper/mpathjl: 1074 MB, 1074462720 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x35dd781b

              Device Boot      Start         End      Blocks   Id  System

/dev/mapper/mpathjlp1               1         130     1044193+  83  Linux

2.1) Copy fdisk partition

sfdisk -d mpathjl > /tmp/50GB_LUN.txt

2.2) Apply fdisk partition to empty LUN

sfdisk mpath < /tmp/50GB_LUN.txt

for i in $(cat /tmp/100gb_mpath); do sfdisk $i < /tmp/mpath100gb.txt done

3) Change ownership of mpath disk

[root@rac1 ~]# fdisk -l "/dev/mapper/mpathjl"

Disk /dev/mapper/mpathjl: 1074 MB, 1074462720 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x35dd781b

              Device Boot      Start         End      Blocks   Id  System

/dev/mapper/mpathjlp1               1         130     1044193+  83  Linux

[root@rac1 ~]# chown oragrid:oinstall /dev/mapper/mpathjlp1

[root@rac2 ~]# chown oragrid:oinstall /dev/mapper/mpathjlp1

[root@rac1 ~]# ls -l /dev/dm-254

brw-rw---- 1 oragrid oinstall 252, 254 Nov 13 18:51 /dev/dm-254

[root@rac1 ~]# ls -l /dev/dm-254

brw-rw---- 1 oragrid oinstall 252, 254 Nov 13 18:51 /dev/dm-254

3.5) If second node does not see the disks you might need to use partprobe

E.g partprobe /dev/mapper/mpathjlp1

4) Create ASM disk

[root@rac1 ~]# /etc/init.d/oracleasm  createdisk  VOTEVPLEX /dev/mapper/mpathjlp1

Marking disk "VOTEVPLEX" as an ASM disk:                   [  OK  ]

[root@rac1 ~]# /etc/init.d/oracleasm  listdisks

[root@rac2 ~]# /etc/init.d/oracleasm  scandisks

Scanning the system for Oracle ASMLib disks:               [  OK  ]

[root@rac2 ~]# /etc/init.d/oracleasm  listdisks

5) Check ASM disk

[root@rac1 ~]# /etc/init.d/oracleasm querydisk -d VOTEVPLEX

Disk "VOTEVPLEX" is a valid ASM disk on device [252,254]

[root@rac2 ~]# /etc/init.d/oracleasm querydisk -d VOTEVPLEX

Disk "VOTEVPLEX" is a valid ASM disk on device [252,254]

[root@rac1 ~]# oracleasm querydisk /dev/dm-254

Device "/dev/dm-254" is marked an ASM disk with the label "VOTEVPLEX"

[root@rac2 ~]# oracleasm querydisk /dev/dm-254

Device "/dev/dm-254" is marked an ASM disk with the label "VOTEVPLEX"

[root@rac1 ~]# multipath -ll mpathjl

mpathjl (36000144000000010303f2aae3350e2e2) dm-253 XXX

size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  |- 1:0:6:106 sdob 128:368 active ready running

  |- 1:0:7:106 sdoc 128:384 active ready running

  |- 1:0:8:106 sdod 128:400 active ready running

  `- 1:0:9:106 sdoe 128:416 active ready running

[root@rac2 ~]# multipath -ll mpathjl

Nov 13 20:00:58 | multipath.conf line 4, invalid keyword: selector

Nov 13 20:00:58 | multipath.conf line 7, invalid keyword: prio_callout

mpathjl (36000144000000010303f2aae3350e2e2) dm-253 XXX

size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  |- 1:0:4:106 sdto 65:608  active ready running

  |- 2:0:8:106 sdts 65:672  active ready running

  |- 1:0:5:106 sdtp 65:624  active ready running

  |- 2:0:6:106 sdtt 65:688  active ready running

  |- 1:0:6:106 sdtr 65:656  active ready running

  |- 2:0:7:106 sdtu 65:704  active ready running

  `- 1:0:7:106 sdtq 65:640  active ready running

Get multiple Oracle SIDs into variable without using oratab

Here's the script for the lazy ones. After script I have explained how the script actually works. 

This script requires connection as sqlplus / as sysdba without giving password

#
# 1.0   17.09.2015      Mika Heino
#
# List Oracle SIDs in server in human readable output
# and connect SQL plus as sysdba
# 

db=`ps -ef | grep [p]mon  | awk {'print $8'} | sed -e 's,ora_pmon_,,' | sed -e s,asm_pmon_,,`
for i in $db ; do
  if [ "$i" == "sed" ] || [ "$i" == *ASM* ]
  then
     continue
  else
         export ORAENV_ASK=NO
         export ORACLE_SID=$i
        . oraenv
        export ORAENV_ASK=YES
   sqlplus -S / as sysdba @your_sql_script
  fi
done

What it actually does? 

ps -ef | grep [p]mon -- List all active pmon processes. Use [] to avoid grep coming into the result list

awk {'print $8'} -- print from output only the 8th column, which is the process name

sed -e 's,ora_pmon_,,' -- remove 'ora_pmon_' text from output

sed -e s,asm_pmon_,, -- remove 'asm_pmon_' text from output

Final output looks like this. It will still output 'sed' text two times, but this will be removed in the clumsy if -clause

[oracle@server ~]$ ps -ef | grep [p]mon  | awk {'print $8'} | sed -e 's,ora_pmon_,,' | sed -e s,asm_pmon_,,

+ASM1

INSTANCE2

INSTANCE3

INSTANCE4

sed

sed

This output is then inserted into variable $db. 

db=`ps -ef | grep [p]mon  | awk {'print $8'} | sed -e 's,ora_pmon_,,' | sed -e s,asm_pmon_,,`

Do for loop for all values within the variable 

for i in $db ; do

Skip and continue if the value is 'sed' or 'ASM' 

  if [ "$i" == "sed" ] || [ "$i" == *ASM* ]
  then
     continue
  else

Otherwise export the Oracle environment, use . oraenv and execute the desired SQL script. 

         export ORAENV_ASK=NO
         export ORACLE_SID=$i
        . oraenv
        export ORAENV_ASK=YES

If your environment has passwords, use them right here using variables.

   sqlplus -S / as sysdba @your_sql_script
  fi
done

Checklist for Windows 2012 Cluster installations -- SQL Server cluster installation taking forever

Failover Cluster Manager will hang unless following KB is installed (on Windows Server 2012)
https://support.microsoft.com/en-us/kb/2803748

If you can't use VMware/Hyper-V mounts, transfer SQL Server installation medias to C:\temp\ as ISO file and mount them as Z: or anything else as chosen cluster disks letter to avoid errors

SQL Server installation needs be started from cmd by using switches listed in KB: https://support.microsoft.com/en-us/kb/953748 if Windows Cluster has errors in it

• Setup /SkipRules=Cluster_VerifyForErrors /Action=InstallFailoverCluster on first node
• Setup /SkipRules=Cluster_VerifyForErrors /Action=AddNode on second node

When the installer is started, it will take VERY LONG time (HOURS) to process any operation unless admin shares are enabled

To enable admin shares do following

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System
Change the value of LocalAccountTokenFilterPolicy to 1

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Lanmanserver\Parameters
Change the value of AutoShareWks to 1 
Change the value of AutoShareServer to 1

Patching SQL Servers with SCCM (Microsoft System Center Configuration Manager)

I've been looking from Google opinions about whether to do or not do SQL Server patching with SCCM. I've only found one topic from SQL Server Central (http://www.sqlservercentral.com/Forums/Topic1589012-1550-1.aspx -- might require userid to see) with one answer 'I do not recommend'.

This was not the answer which I was looking for so I went ahead together with our company's SCCM guys and made some test patching of SQL Server's (2008/2008 R2 and 2012 -versions). Those patch runs executed without any hassle and issues.

I'll agree that this process is missing the part where the application connection is tested afterwards and no backup is taken before patching. These are though things that can be tackled by regular backup schedule. Application connection testing would require end-to-end monitoring, but usually rebooting the application server(s) will re-establish the db connection and would end to errors if the connection is not working.

So if you're like me and trying to find courage from Google to start patching SQL Server by using SCCM I can assure you that it's not a big of deal and the trade off is quite huge. No more time on moving the Service Pack's into servers and executing them manually. We got several (tens -- not hundreds) of internal SQL Server's back into shape in a day.

How to delete spfile entries in Oracle 11gr2

Do not use '' or 'NULL' values. Use instead following reset -value

alter system reset parameter scope sid='*';

e.g.

alter system reset db_file_name_convert scope=spfile sid='*';