ASMdisk Status – Candidate disk after reboot [ recover ASM header files ]

If you cannot recover the data by yourself, ask Parnassusdata, the professional ORACLE database recovery team for help.

Parnassusdata Software Database Recovery Team

Service Hotline:  +86 13764045638 E-mail: [email protected]

 

Customer has  migrated  oracle databases running on old SAN to new SAN using ASM  rebalance operation. Customer is using External   redundancy. After completing rebalance operation, customer rebooted all server and removed old SAN device entry on weekend.  Customer is unable to bring databases online on  One of the 4 servers. Customer is getting following error :
ORA-15032: not all alterations performed
ORA-15017: diskgroup “DATA” cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “DATA”

 

 

I have attached asm-kfed result for your reference. Is it possible to recover ASM header files without so customer doesn’t need to backup/restore  5 TB database ?

 

Total System Global Area  284008448 bytes
Fixed Size                  2158616 bytes
Variable Size             256684008 bytes
ASM Cache                  25165824 bytes
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"
 

SQL> select group_number,disk_number,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,NAME,PATH from V$asm_disk;
 
GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE    NAME                           PATH
------------ ----------- ------- ------------ ------- -------- ------------------------------ ----------------------------------------
           0           0 CLOSED  CANDIDATE    ONLINE  NORMAL                                  /dev/rdsk/c0d26s6
           0          23 IGNORED MEMBER       ONLINE  NORMAL                                  /dev/rdsk/c0d22s6
           0           2 IGNORED MEMBER       ONLINE  NORMAL                                  /dev/rdsk/san03dp_dbs05dp_ASM01
           0           3 IGNORED MEMBER       ONLINE  NORMAL                                  /dev/rdsk/san03dp_dbs05dp_ASM12
           0           4 CLOSED  CANDIDATE    ONLINE  NORMAL                                  /dev/rdsk/c0d27s6
           0           5 CLOSED  CANDIDATE    ONLINE  NORMAL                                  /dev/rdsk/c0d25s6
           0           6 CLOSED  CANDIDATE    ONLINE  NORMAL                                  /dev/rdsk/san03dp_dbs05dp_ASM05
           0           7 IGNORED MEMBER       ONLINE  NORMAL                                  /dev/rdsk/san03dp_dbs05dp_ASM02
           0           8 CLOSED  CANDIDATE    ONLINE  NORMAL                                  /dev/rdsk/san03dp_dbs05dp_ASM08
           0           9 CLOSED  CANDIDATE    ONLINE  NORMAL                                  /dev/rdsk/c0d24s6
           0          10 IGNORED MEMBER       ONLINE  NORMAL                                  /dev/rdsk/san03dp_dbs05dp_ASM11
           0          11 IGNORED MEMBER       ONLINE  NORMAL                                  /dev/rdsk/c0d30s6
           0          12 CLOSED  CANDIDATE    ONLINE  NORMAL                                  /dev/rdsk/san03dp_dbs05dp_ASM07
           0          13 IGNORED MEMBER       ONLINE  NORMAL                                  /dev/rdsk/c0d31s6
           0          14 IGNORED MEMBER       ONLINE  NORMAL                                  /dev/rdsk/c0d21s6
           0          15 IGNORED MEMBER       ONLINE  NORMAL                                  /dev/rdsk/san03dp_dbs05dp_ASM10
           0          16 CLOSED  CANDIDATE    ONLINE  NORMAL                                  /dev/rdsk/c0d23s6
           0          17 IGNORED MEMBER       ONLINE  NORMAL                                  /dev/rdsk/san03dp_dbs05dp_ASM09
           0          18 CLOSED  CANDIDATE    ONLINE  NORMAL                                  /dev/rdsk/san03dp_dbs05dp_ASM03
           0          19 IGNORED MEMBER       ONLINE  NORMAL                                  /dev/rdsk/c0d29s6
           0          20 CLOSED  CANDIDATE    ONLINE  NORMAL                                  /dev/rdsk/c0d28s6
           0          21 CLOSED  CANDIDATE    ONLINE  NORMAL                                  /dev/rdsk/san03dp_dbs05dp_ASM04
           0          22 IGNORED MEMBER       ONLINE  NORMAL                                  /dev/rdsk/c0d32s6
           0           1 CLOSED  CANDIDATE    ONLINE  NORMAL                                  /dev/rdsk/san03dp_dbs05dp_ASM06





dev/rdsk/san03dp_dbs05dp_ASM03
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
10037FE00 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

/dev/rdsk/san03dp_dbs05dp_ASM04
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
10037FE00 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

/dev/rdsk/san03dp_dbs05dp_ASM05
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
10037FE00 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

/dev/rdsk/san03dp_dbs05dp_ASM06
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
10037FE00 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

/dev/rdsk/san03dp_dbs05dp_ASM07
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
10037FE00 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

/dev/rdsk/san03dp_dbs05dp_ASM08
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
10037FE00 00000000 00000000 00000000 00000000 [................]
Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

 

 

ASM disks – ASM03/04/05/06/07/08 –  showing the status as “CANDIDATE” is a bit worry.  But if the devices were not part of DATA diskgroup before, these devices are not the major cause of ORA-15063.
Please check ASM alert.log whether these 5 devices were belong to DATA.

I’m more concerned about the following 4 devices as they show the status as “IGNORED” which indicates there are other devices showing the same disk information given asm_diskstring parameter.
– ASM01/02/0910/11

Chances are that the following devices below show the same disk information as ASM01/02/0910/11 and there is good chance that these different path point to the same physical devices.
~~
/dev/rdsk/c0d22s6
/dev/rdsk/c0d30s6
/dev/rdsk/c0d31s6
/dev/rdsk/c0d21s6
/dev/rdsk/c0d29s6
/dev/rdsk/c0d32s6
~~

Please check all disk header which disks show the duplicate ASM disk information using the following perl script in the note below
And if duplicate paths point to the same physical device, the additional device path should be disabled using “chmod 000 <device_path>”
KFED.PL for diagnosing – ORA-15063 ORA-15042 ORA-15020 (Doc ID 1346190.1)


Posted

in

by

Tags:

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *