——– CRS 10.2.0.3 ASM 10.2.0.3 RDBMS 10.2.0.2 I have a customer  has 2 nodes cluster and ASM doesn’t mount the diskgroup   Alert.log ASM1 ————– Thu Feb 7 21:52:31 2008 NOTE: starting recovery of thread=1 ckpt=56.6552 group=1 ORA-15196: invalid ASM block header [kfr.c:6086] [endian_kfbh] [3] [6558] [0 != 1] NOTE: cache initiating offline of disk 0 group 1 WARNING: offlining disk 0.3916356260 (DATA_0000) with mask 0x3 NOTE: PST update: grp = 1, dsk = 0, mode = 0x6 … Thu Feb 7 21:52:31 2008 @Errors in file /opt/oracle/admin/+ASM/bdump/+asm1_b000_17753.trc: ORA-15001: diskgroup “DATA” does not exist or is not mounted Alert.log ASM2 ————– Thu Feb 7 21:52:31 2008 @Errors in file /opt/oracle/admin/+ASM/udump/+asm2_ora_4385.trc: ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +DATA/dwdev/parameterfile/spfiledwdev.ora ORA-17503: ksfdopn:2 Failed to open file +DATA/dwdev/parameterfile/spfiledwdev.ora ORA-15001: diskgroup “DATA” does not exist or is not mounted Patch 5554692 was applied to prevent problem in the future . Ct doesn’t have a valid backup . DISK Header ———– dd if=/dev/sdc1 bs=8192 count=1 | od -c 0000000 001 202 001 001 \0 \0 \0 \0 \0 \0 \0 200 367 354 327 \ 0000020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000040 O R C L D I S K D A T A \0 \0 \0 \0 0000060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000100 \0 \0 020 \n \0 \0 001 003 D A T A _ 0 0 0 0000120 0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000140 \0 \0 \0 \0 \0 \0 \0 \0 D A T A \0 \0 \0 \0 0000160 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000200 \0 \0 \0 \0 \0 \0 \0 \0 D A T A _ 0 0 0 0000220 0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000240 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 . . dd if=/dev/sdi1 bs=8192 count=1 | od -c 1+0 records in 1+0 records out 0000000 001 202 001 001 \0 \0 \0 \0 \0 \0 \0 200 367 354 327 \ 0000020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000040 O R C L D I S K D A T A \0 \0 \0 \0 0000060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000100 \0 \0 020 \n \0 \0 001 003 D A T A _ 0 0 0 0000120 0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000140 \0 \0 \0 \0 \0 \0 \0 \0 D A T A \0 \0 \0 \0 0000160 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000200 \0 \0 \0 \0 \0 \0 \0 \0 D A T A _ 0 0 0 0000220 0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \     “alert_ASM1.log-new2.txt”: . last time DATA was mounted successfully: . Thu Feb 7 19:15:34 2008 NOTE: cache mounting group 2/0x554FBB0C (DATA) succeeded SUCCESS: diskgroup DATA was mounted . … . Reconfiguration complete Thu Feb 7 19:44:23 2008 Starting background process ASMB ASMB started with pid=24, OS id=2278 Thu Feb 7 19:44:41 2008 WARNING: cache failed to read fn=453 indblk=0 from disk(s): 0 ORA-15196: invalid ASM block header [kfc.c:7997] [endian_kfbh] [453] [2147483648] [0 != 1] NOTE: a corrupted block was dumped to the trace file System State dumped to trace file /opt/oracle/admin/+ASM/udump/+asm1_ora_4636.trc NOTE: cache initiating offline of disk 0 group 2 WARNING: offlining disk 0.3916384773 (DATA_0000) with mask 0x3 NOTE: PST update: grp = 2, dsk = 0, mode = 0x6 Thu Feb 7 19:44:42 2008 ERROR: too many offline disks in PST (grp 2) . . alert_ASM2.log: . Thu Feb 7 19:50:04 2008 NOTE: cache mounting group 2/0x457D4099 (DATA) succeeded SUCCESS: diskgroup DATA was mounted . . econfiguration complete Thu Feb 7 20:17:04 2008 Starting background process ASMB ASMB started with pid=21, OS id=13439 Thu Feb 7 20:17:15 2008 WARNING: cache failed to read fn=453 indblk=0 from disk(s): 0 ORA-15196: invalid ASM block header [kfc.c:7997] [endian_kfbh] [453] [2147483648] [0 != 1] NOTE: a corrupted block was dumped to the trace file System State dumped to trace file /opt/oracle/admin/+ASM/udump/+asm2_ora_14194.trc NOTE: cache initiating offline of disk 0 group 2 WARNING: offlining disk 0.3937251428 (DATA_0000) with mask 0x3 NOTE: PST update: grp = 2, dsk = 0, mode = 0x6 Thu Feb 7 20:17:18 2008 ERROR: too many offline disks in PST (grp 2) Thu Feb 7 20:17:18 2008 NOTE: halting all I/Os to diskgroup DATA NOTE: active pin found: 0x0x659fe548 Thu Feb 7 20:17:18 2008 ERROR: PST-initiated MANDATORY DISMOUNT of group DATA . . 1). Customer is using external redundancy. Since it’s external redundancy, ct should check with their storage vendor to see if they can restore it. From oracle perspetive, because we don’t store copy of data with exteranl redundancy, the only way is to do diskgroup restore. 2). Please upload bdump/udump of all ASM instances 3). please provide the kfed output of disk 0 (DATA_0000) 4). please provide the first 80 M of DATA_0000 dd if=<target disk> of=<file> bs=4096 count=20480 . [oracle@drac1 bin]$ kfed read /dev/databases |more kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0 kfbh.block.obj: 2147483648 ; 0x008: TYPE=0x8 NUMB=0x0 kfbh.check: 1557654775 ; 0x00c: 0x5cd7ecf7 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdhdb.driver.provstr: ORCLDISKDATA ; 0x000: length=12 kfdhdb.driver.reserved[0]: 1096040772 ; 0x008: 0x41544144 kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000 kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000 kfdhdb.compat: 168820736 ; 0x020: 0x0a100000 kfdhdb.dsknum: 0 ; 0x024: 0x0000 kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER kfdhdb.dskname: DATA_0000 ; 0x028: length=9 kfdhdb.grpname: DATA ; 0x048: length=4 kfdhdb.fgname: DATA_0000 ; 0x068: length=9 kfdhdb.capname: ; 0x088: length=0 kfdhdb.crestmp.hi: 32900398 ; 0x0a8: HOUR=0xe DAYS=0x9 MNTH=0x1 YEAR=0x7d8 kfdhdb.crestmp.lo: 224448512 ; 0x0ac: USEC=0x0 MSEC=0x34 SECS=0x16 MINS=0x3 kfdhdb.mntstmp.hi: 32901363 ; 0x0b0: HOUR=0x13 DAYS=0x7 MNTH=0x2 YEAR=0x7d8 kfdhdb.mntstmp.lo: 3359888384 ; 0x0b4: USEC=0x0 MSEC=0xf5 SECS=0x4 MINS=0x32 kfdhdb.secsize: 512 ; 0x0b8: 0x0200 kfdhdb.blksize: 4096 ; 0x0ba: 0x1000 kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000 kfdhdb.mfact: 113792 ; 0x0c0: 0x0001bc80 kfdhdb.dsksize: 5722043 ; 0x0c4: 0x00574fbb kfdhdb.pmcnt: 52 ; 0x0c8: 0x00000034 kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001 kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002 kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002 kfdhdb.redomirrors[0]: 0 ; 0x0d8: 0x0000 kfdhdb.redomirrors[1]: 0 ; 0x0da: 0x0000 kfdhdb.redomirrors[2]: 0 ; 0x0dc: 0x0000 kfdhdb.redomirrors[3]: 0 ; 0x0de: 0x0000 kfdhdb.dbcompat: 168820736 ; 0x0e0: 0x0a100000 kfdhdb.grpstmp.hi: 32900398 ; 0x0e4: HOUR=0xe DAYS=0x9 MNTH=0x1 YEAR=0x7d8 kfdhdb.grpstmp.lo: 223643648 ; 0x0e8: USEC=0x0 MSEC=0x122 SECS=0x15 MINS=0x3 kfdhdb.ub4spare[0]: 0 ; 0x0ec: 0x00000000 kfdhdb.ub4spare[1]: 0 ; 0x0f0: 0x00000000 kfdhdb.ub4spare[2]: 0 ; 0x0f4: 0x00000000 kfdhdb.ub4spare[3]: 0 ; 0x0f8: 0x00000000 kfdhdb.ub4spare[4]: 0 ; 0x0fc: 0x00000000 kfdhdb.ub4spare[5]: 0 ; 0x100: 0x00000000 kfdhdb.ub4spare[6]: 0 ; 0x104: 0x00000000 kfdhdb.ub4spare[7]: 0 ; 0x108: 0x00000000 kfdhdb.ub4spare[8]: 0 ; 0x10c: 0x00000000 kfdhdb.ub4spare[9]: 0 ; 0x110: 0x00000000 . . [oracle@drac1 bin]$ kfed read /dev/raw/raw1 kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 4290772992 ; 0x004: T=1 NUMB=0x7fc00000 kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 44045 ; 0x010: 0x0000ac0d kfbh.fcn.wrap: 4096 ; 0x014: 0x00001000 kfbh.spare1: 51147 ; 0x018: 0x0000c7cb kfbh.spare2: 2054913149 ; 0x01c: 0x7a7b7c7d [oracle@drac1 bin]$ kfed read /dev/raw/raw2 kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 34 ; 0x001: 0x22 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 4290772992 ; 0x004: T=1 NUMB=0x7fc00000 kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 59301 ; 0x010: 0x0000e7a5 kfbh.fcn.wrap: 512 ; 0x014: 0x00000200 kfbh.spare1: 409189 ; 0x018: 0x00063e65 kfbh.spare2: 2054913149 ; 0x01c: 0x7a7b7c7d alert_+ASM1.log: . Thu Feb 7 19:44:23 2008 Starting background process ASMB ASMB started with pid=24, OS id=2278 Thu Feb 7 19:44:41 2008 WARNING: cache failed to read fn=453 indblk=0 from disk(s): 0 ORA-15196: invalid ASM block header [kfc.c:7997] [endian_kfbh] [453] [2147483648] [0 != 1] NOTE: a corrupted block was dumped to the trace file System State dumped to trace file /opt/oracle/admin/+ASM/udump/+asm1_ora_4636.trc NOTE: cache initiating offline of disk 0 group 2 WARNING: offlining disk 0.3916384773 (DATA_0000) with mask 0x3 NOTE: PST update: grp = 2, dsk = 0, mode = 0x6 Thu Feb 7 19:44:42 2008 ERROR: too many offline disks in PST (grp 2) . asm1_ora_4636.trc: . *** SERVICE NAME:() 2008-02-07 19:44:41.691 *** SESSION ID:(100.102) 2008-02-07 19:44:41.690 OSM metadata block dump: kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 162 ; 0x001: 0xa2 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 114084224 ; 0x004: T=0 NUMB=0x6ccc980 kfbh.block.obj: 0 ; 0x008: TYPE=0x0 NUMB=0x0 kfbh.check: 83951616 ; 0x00c: 0x05010000 kfbh.fcn.base: 26700 ; 0x010: 0x0000684c kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 —– Abridged Call Stack Trace —– kfcReadBlk()+1181 kfcReadBuffer()+917 kfcGet0()+14974 kfcGet1Priv()+161 kfcGetPriv()+178 kffSendMap()+2050 kffIdentify()+1552 kfnsFileIdentify()+595 kfnDispatch()+86 opiodr()+984 ttcpip()+1235 opitsk()+1298 Cannot find symbol Cannot find symbol opiino()+1028 opiodr()+984 opidrv()+547 sou2o()+114 opimai_real()+163 main()+116 Cannot find symbol <0x3b49e1c3fb> It seems that we have two block corruptions since the last time diskgroup was mounted. . [32BIT oracle@orcl bdump]$ grep -n ‘15196’ alert_+ASM1.log | sort | uniq ORA-15196: invalid ASM block header [kfc.c:7997] [endian_kfbh] [453] [2147483648] [0 != 1] ORA-15196: invalid ASM block header [kfr.c:6086] [endian_kfbh] [3] [6558] [0 != 1] . Usually this type of corruption would require restore of diskgroup if the redundancy is external You may want to check if DUL can help in this case. Check http://www.parnassusdata.com/en/emergency-services