asm实例报ORA-15196: invalid ASM block header [kfc.c:8281] [check_kfbh] [2147483649] [2] [2162363223 != 2162394711]

huangtingzhong 发布于 2013-09-29 19:12ASM, 故障处理抢沙发

环境介绍：os:rhel 5.8 db:10.2.0.5

数据库以外被停了

Errors in file /u01/app/oracle/admin/asm10g/bdump/asm10g_ckpt_9203.trc:
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DATA/asm10g/controlfile/current.260.826785429'
ORA-15078: ASM diskgroup was forcibly dismounted
Mon Sep 30 01:45:25 CST 2013
Errors in file /u01/app/oracle/admin/asm10g/bdump/asm10g_ckpt_9203.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DATA/asm10g/controlfile/current.260.826785429'
ORA-15078: ASM diskgroup was forcibly dismounted
Mon Sep 30 01:45:25 CST 2013
CKPT: terminating instance due to error 221
Instance terminated by CKPT, pid = 9203

查看asm实例的实例发现如下的信息

NOTE: assigning ARB0 to group 2/0xb3f85b88 (DATA)
Mon Sep 30 01:45:21 CST 2013
WARNING: cache read a corrupted block gn=2 dsk=1 blk=2 from disk 1
NOTE: a corrupted block was dumped to the trace file
ERROR: cache failed to read dsk=1  blk=2 from disk(s): 1
ORA-15196: invalid ASM block header [kfc.c:8281] [check_kfbh] [2147483649] [2] [2162363223 != 2162394711]
System State dumped to trace file /u01/app/oracle/admin/+ASM/bdump/+asm_arb0_9255.trc
NOTE: cache initiating offline of disk 1  group 2
WARNING: process 9255 initiating offline of disk 1.3915950986 (DATA_0001) with mask 0x3 in group 2
WARNING: Disk 1 in group 2 in mode: 0x7,state: 0x2 will be taken offline
NOTE: PST update: grp = 2, dsk = 1, mode = 0x6

通过上面的值可以发现在读取disk 1，block 为2时发现check kfbh时发现值不一样。

通过trace出来的文件可以发现下面的信息

/u01/app/oracle/admin/+ASM/bdump/+asm_arb0_9255.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db
System name:    Linux
Node name:      asm10g
Release:        2.6.18-308.el5
Version:        #1 SMP Fri Jan 27 17:17:51 EST 2012
Machine:        x86_64
Instance name: +ASM
Redo thread mounted by this instance: 0 <none>
Oracle process number: 13
Unix process pid: 9255, image: oracle@asm10g (ARB0)

*** SERVICE NAME:() 2013-09-30 01:45:21.569
*** SESSION ID:(38.5) 2013-09-30 01:45:21.569
ARB0 relocating file +DATA.1.1 (1 entries)
*** 2013-09-30 01:45:21.576
WARNING: cache read a corrupted block gn=2 dsk=1 blk=2 from disk 1
OSM metadata block dump:
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       2 ; 0x004: T=0 NUMB=0x2
kfbh.block.obj:              2147483649 ; 0x008: TYPE=0x8 NUMB=0x1
kfbh.check:                  2162363223 ; 0x00c: 0x80e30b57
kfbh.fcn.base:                     1412 ; 0x010: 0x00000584
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdatb.aunum:                         0 ; 0x000: 0x00000000

可以发现当前check不正确，修改check值，这些使用的bbed来修改

BBED> show all
        FILE#           0
        BLOCK#          17
        OFFSET          0
        DBA             0x00000000 (0 0,17)
        FILENAME        /dev/mapper/asm2
        BIFILE          bifile.bbd
        LISTFILE       
        BLOCKSIZE       512
        MODE            Edit
        EDIT            Unrecoverable
        IBASE           Dec
        OBASE           Dec
        WIDTH           80
        COUNT           512
        LOGFILE         log.bbd
        SPOOL           No

BBED> map
 File: /dev/mapper/asm2 (0)
 Block: 17                                    Dba:0x00000000
------------------------------------------------------------
 Undo Segment Header

 struct kcbh, 20 bytes                      @0       

 struct ktect, 44 bytes                     @20      

 struct ktetb[8388610], 67108880 bytes      @64      

 struct ktuxc, 104 bytes                    @468     

 struct ktuxe[0], 0 bytes                   @572     

 ub4 tailchk                                @508     


BBED> print kcbh
struct kcbh, 20 bytes                       @0       
   ub1 type_kcbh                            @0        0x01
   ub1 frmt_kcbh                            @1        0x82
   ub1 spare1_kcbh                          @2        0x03
   ub1 spare2_kcbh                          @3        0x01
   ub4 rdba_kcbh                            @4        0x00000002
   ub4 bas_kcbh                             @8        0x80000001
   ub2 wrp_kcbh                             @12       0x0b57
   ub1 seq_kcbh                             @14       0xe3
   ub1 flg_kcbh                             @15       0x80 (NONE)
   ub2 chkval_kcbh                          @16       0x0584
   ub2 spare3_kcbh                          @18       0x0000


BBED> dump
 File: /dev/mapper/asm2 (0)
 Block: 17               Offsets:    0 to  511           Dba:0x00000000
------------------------------------------------------------------------
 01820301 02000000 01000080 570be380 84050000 00000000 00000000 00000000 
 00000000 c0010000 08000800 0000c001 10001000 00000000 18001800 00000000 
 20002000 00000000 00000000 00008000 00000000 00008000 00000000 02008000 
 00000000 03008000 02000000 03008000 03000000 03008000 05000000 03008000 
 06000000 03008000 08000000 03008000 09000000 03008000 0b000000 03008000 
 0c000000 03008000 0e000000 03008000 0f000000 03008000 11000000 03008000 
 12000000 03008000 14000000 03008000 15000000 03008000 17000000 03008000 
 18000000 03008000 1a000000 03008000 1b000000 03008000 1d000000 03008000 
 1e000000 03008000 20000000 03008000 21000000 03008000 23000000 03008000 
 24000000 03008000 26000000 03008000 27000000 03008000 29000000 03008000 
 00000000 04008000 00000000 05008000 00000000 06008000 01000000 01008000 
 00000000 00018000 02000000 00018000 03000000 00018000 05000000 00018000 
 06000000 00018000 08000000 00018000 09000000 00018000 0b000000 00018000 
 0c000000 00018000 0e000000 00018000 0f000000 00018000 11000000 00018000 
 12000000 00018000 14000000 00018000 15000000 00018000 17000000 00018000 
 18000000 00018000 1a000000 00018000 1b000000 00018000 1d000000 00018000 
 
80E38657


BBED> modify /x 5786 offset 12
 File: /dev/mapper/asm2 (0)
 Block: 17               Offsets:   12 to  511           Dba:0x00000000
------------------------------------------------------------------------
 5786e380 84050000 00000000 00000000 00000000 00000000 c0010000 08000800 
 0000c001 10001000 00000000 18001800 00000000 20002000 00000000 00000000 
 00008000 00000000 00008000 00000000 02008000 00000000 03008000 02000000 
 03008000 03000000 03008000 05000000 03008000 06000000 03008000 08000000 
 03008000 09000000 03008000 0b000000 03008000 0c000000 03008000 0e000000 
 03008000 0f000000 03008000 11000000 03008000 12000000 03008000 14000000 
 
 
BBED> sum apply
Check value for File 0, Block 17:
current = 0x0584, required = 0x0584

手动mount磁盘组

SQL> alter diskgroup data mount;

Diskgroup altered.


SUCCESS: diskgroup DATA was mounted
Mon Sep 30 02:01:12 CST 2013
NOTE: recovering COD for group 2/0xf8685b9d (DATA)
SUCCESS: completed COD recovery for group 2/0xf8685b9d (DATA)
Mon Sep 30 02:01:12 CST 2013
NOTE: starting rebalance of group 2/0xf8685b9d (DATA) at power 1
Starting background process ARB0
ARB0 started with pid=12, OS id=9313
Mon Sep 30 02:01:12 CST 2013
NOTE: assigning ARB0 to group 2/0xf8685b9d (DATA)
NOTE: stopping process ARB0
Mon Sep 30 02:01:18 CST 2013
SUCCESS: rebalance completed for group 2/0xf8685b9d (DATA) 
Mon Sep 30 02:01:18 CST 2013
SUCCESS: rebalance completed for group 2/0xf8685b9d (DATA) 
NOTE: PST update: grp = 2
NOTE: PST update: grp = 2

已经看到pst的信息已经更新完成。

赏