如何在无审计的环境中追踪Truncate/Drop等危险的DDL操作

在有充分审计Audit   SQL的情况下,定位某条DROp/Truncate DDl还是比较容易的。

 

问题是 没有任何SQL审计时呢?

 

首先需要 明确的是 对于 关键的产品数据库系统而言 有效的审计非常重要,不能将以下的方法当做审计选项来用。

其次对于重要的环境和重要的对象, 一般推荐创建一个DDL Trigger 让直接运行的DDL报错,而需要先将对应的DDL Disable之后才能成功 执行DDL, 这样可以降低表/索引因为人为失误导致的误操作。

 

最简单的仍是通过 dba_objects 定位其最后的DDL时间,虽然这个LAST_DDL_TIME 未必是真实的案发时间了, 但如何与应用程序报错结合起来 还是能大致了解问题发生的时间段的, 而这个时间段 对于问题追查至关重要。 此外对于DROP命令而言显然弄不到这个LAST_DDL_TIME。

 

接着可以通过ASH 视图 DBA_HIST_ACTIVE_SESS_HISTORY和 V$ACTIVE_SESSION_HISTORY来定位一些DDL语句, 由于ASH默认是1秒采样一次,所以如果遇到了一些例如RAC 中truncate/drop 常见的 DFS Lock Handle、Enqueue Lock等等待,那么一般ASH都能捕捉到这个DDL,当然这也看运气,毕竟ASH不是审计功能。

 

SQL_OPCODE    12 为DROP TABLE  10为 DROP INDEX、85为TRUNCATE TABLE、86为TRUNCATE CLUSTER

select USER_ID, SQL_OPCODE, XID, SAMPLE_TIME,MODULE,MACHINE
from v$active_session_history
where SQL_OPCODE in (12, 10, 85, 86)
and SAMPLE_TIME between xx and xx;

select USER_ID, SQL_OPCODE, XID, SAMPLE_TIME,MODULE,MACHINE
from dba_hist_active_sess_history
where SQL_OPCODE in (12, 10, 85, 86)
and SAMPLE_TIME between xx and xx;

 

如果上述查询给出少量精准结果(例如MACHINE和MODULE很特殊),那么一般就很容易定位了。  如果查出大量结果,一般优先排除应用程序模块例如 JDBC Thin,如果应用程序本身有BUG导致莫名的DDL,那么理论上应当经常发生。 如果看到一些例如SQLPLUS、PL/SQL Developer认为登陆执行上述命令的记录,则需要特别关注。

 

由于DDL语句不作为 共享SQL保存在V$SQL、V$SQLAREA中所以 就算你获得了SQL_ID还是看不到这些SQL语句的,所以无法通过SQL_TEXT来定位这些SQL到底是什么样子。

 

这个时候往往需要做Logminer了, 但好在我们有大致的Sample  Time和XID 这样定位SQL就很简单。

 

EXECUTE DBMS_LOGMNR.ADD_LOGFILE( LOGFILENAME => ‘/oracle/logs/log1.f’, OPTIONS => DBMS_LOGMNR.NEW);

EXECUTE DBMS_LOGMNR.ADD_LOGFILE( LOGFILENAME => ‘/oracle/logs/log2.f’, OPTIONS => DBMS_LOGMNR.ADDFILE);
execute DBMS_LOGMNR.START_LOGMNR(DBMS_LOGMNR.DICT_FROM_ONLINE_CATALOG+DBMS_LOGMNR.COMMITTED_DATA_ONLY);
SELECT (XIDUSN || ‘.’ || XIDSLT || ‘.’ || XIDSQN) AS XID,
USERNAME, SQL_REDO FROM V$LOGMNR_CONTENTS WHERE XID=’XXX’;

EXECUTE DBMS_LOGMNR.END_LOGMNR;

 

 

此外在没有审计的情况下 值得参考的数据还有 Listener监听器的日志、OS登陆的Shell日志等。

【数据恢复】详解ORA-1410错误

ORA-1410 invalid rows错误是与ORA-8103相似的Oracle数据库逻辑层面的讹误。

了解ORA-1410逻辑坏块问题的成因,以及有效的解决手段十分重要。

解决方案之一:

可以通过如下PL/SQL过程将健康数据复制到新建表中,对于问题数据块中的数据将被跳过,对于能够容忍数据丢失的场景可以考虑这样恢复,之后truncate 原表/分区并将健康数据加载进去。 具体的脚本见下面的链接:

【数据恢复】利用构造ROWID实现无备份情况下绕过ORA-1578、ORA-8103、ORA-1410等逻辑/物理坏块问题

 

oerr ora 1410
01410, 00000, “invalid ROWID”
// *Cause:
// *Action:

如果对ORA-1410做errorstack 一般会看到下面的LOG:

OBJD MISMATCH typ=6, seg.obj=%d, diskobj=%d, dsflg=%d, dsobj=%d, tid=%d, cls=%d

 

触发ORA-1410错误的stack call一般都是:  kcbgtcr=>kcbzib=>kcbz_check_objd_typ,即在对数据块做逻辑读时运行到kcbz_check_objd_typ函数时,检测到OBJD 不一致的问题。由于seg.obj和diskobj不一致,而10g以后的kcbz_check_objd_typ函数负责验证块上的objd是否mistmatch,若不一致则触发ORA-1410错误。

造成objd mimatch的主要可能有几种:

1、 写丢失 Lost Write, 写丢失造成相关数据块没有为现有对象正常格式化,导致虽然该数据块的checksum是正确的,但对应数据字典却是不一致的。 写丢失也可能由磁盘或卷组镜像同步软件的不完整复制造成。

 

If the on-disk objd is < kcbdsobj, then there is possibility of Oracle messing up or IO layer (OS Cache, Volume mgr etc) missing writes.

 

对于Lost Write在10g版本中没有太好的预防方案,隐藏”_db_lost_write_checking”控制在DBWR写数据文件后立即去读被写的块以便检测出Lost Write,但是该参数对性能的损耗较大,不建议设置。

11g中引入了DB_LOST_WRITE_PROTECT参数配合Data Guard使用可以有效检测出Lost Write问题。

 

DB_LOST_WRITE_PROTECT enables or disables lost write detection. A data block lost write occurs when an I/O subsystem acknowledges the completion of the block write, while in fact the write did not occur in the persistent storage. 

When the parameter is set to TYPICAL on the primary database, the instance logs buffer cache reads for read-write tablespaces in the redo log, which is necessary for detection of lost writes.

 

When the parameter is set to FULL on the primary database, the instance logs reads for read-only tablespaces as well as read-write tablespaces.

 

When the parameter is set to TYPICAL or FULL on the standby database or on the primary database during media recovery, the instance performs lost write detection.

 

When the parameter is set to NONE on either the primary database or the standby database, no lost write detection functionality is enabled.

 

 

2、 一些DDL操作例如Exchange Partition 造成block级别的不一致,同一个数据块被2个数据对象所使用,而当这2个对象被使用时都可能覆盖问题数据块。 实际上这种情况也可能是Lost Write所引起的。

 

3、 文档Summary Of Bugs Containing ORA 1410 (Doc ID 422771.1)介绍了引起ORA-1410的主要BUG,其中BUG 4592596(Corruption (ORA-1410) from multi-table insert with direct load) 和 BUG 3868753 (Concurrent export / INSERT of ASSM segment can fail with ORA-1410 / ORA-8103)均为对表的Direct path/Parallel INSERT引起后续对表的SELECT操作报ORA-1410错误。

这说明了Direct Path/Parallel Insert操作有小概率引发ORA-1410错误发生的可能,而常规的conventional insert则不会引发ORA-1410。

 

4、 objd mimatch也可能仅仅是Oracle Buffer Cache内存中的block存在不一致,而Disk磁盘上的block仍是完好的。这一般是Oracle Buffer层的BUG引起的,对于该种现象一般flush buffer_cache即可解决问题。虽然在本例中flush buffer_cache未能解决问题,但是若问题仅仅发生在Memory层,则仍建议先考虑flush buffer_cache。

 

针对该由于OBJD MISMATCH所引起的ORA-1410问题可以采取如下措施:

1、 尝试刷新buffer cache:

alter system flush buffer_cache;   ==>如果是RAC建议2个实例都要flush

刷新后再次运行触发ORA-1410错误的语句,若不再报错则说明刷新BUFFER_CACHE有效

 

 

2、 若flush buffer_cache解决不了问题,那么做analyze validate structure 和收集errorstack操作,分析原因:

alter session set events ’1410 trace  name errorstack  level 3′;

运行会触发ORA-1410的语句,收集生成的trace文件

analyze table XXX   validate structure online;  ==>在线validate structure

@?/rdbms/admin/utlvaild   ==>对于分区对象需要运行utlvaild脚本

analyze table XXX partition (partition_name) validate structure online;

 

 

 

 

3、对于已经在磁盘上形成OBJD MISMATCH现象的数据对象:

a. 考虑通过move table、partition、subpartition来尝试解决该问题

alter table xxx move tablespace;

or

alter table move partition xxx tablespace;

or

alter table move sunpartition xxx tablespace;

 

ORA-1410问题相关的一些BUG罗列如下:

 
Bug 5637976
Abstract: ORA-8103/ORA-1410 from concurrent INSERT / export on ASSM tables
This occurs in 10gR2 when there are concurrent inserts and direct path exports. The newly created/updated blocks are not being flushed to disk, so the export is getting a stale version of the block from disk.
Fixed in 10.2.0.4 and 11.1.0.6

Unpublished Bug 4592596
Abstract: Corruption (ORA-1410) from multi-table insert with direct load
This error occurs if a SQL plan is compiled for a parallel run with a Degree of Parallelism (DOP) > 1, but at the time of running, due to lack of resources, it runs serial. Then the problem of invalid rowid will happen.
Fixed in 10.2.0.4 and 11.1.0.6.

Bug 5596325
Abstract: Text query gives wrong results or fails with ORA-1410 ORA-29903
If CONTAINS queries return ORA-1410: invalid rowid errors, and there are more than 200,000,000 documents in the index, then you may have encountered this bug.
Fixed in 10.2.0.4 and 11.1.0.6

Unpublished Bug 6444339
Abstract: TRUNCATE/PURGE DOES NOT CLEAN DEPENDENCIES PROPERLY.
DDL statements to an object were not invalidating all dependencies, so a stale rowid could remain in cache and produce a ORA-1410 if used.
Fixed in 11.2 and 10.2.0.5

Bug 8740993
Abstract: ORA-1410 OCCURRED ON ADG STANDBY DATABASE DURING TABLE SCAN.
This bug applies to standby databases and occurs when the standby is re-applying DDL for table drops/truncates/shrinks. The buffer cache is not being updated for the new object numbers.
Fixed in 12.1, 11.2.0.2

【数据恢复】详解ORA-8103错误

ORA-8103是我们Database Consultant 经常要遇到的一个问题,了解ORA-8103的成因非常重要。

【数据恢复】利用构造ROWID实现无备份情况下绕过ORA-1578、ORA-8103、ORA-1410等逻辑/物理坏块问题

简单来说ORA-8103 的主要成因有2类:

  • 数据块的 block type 类型 是 无效的 或者读出来的块类型与Oracle期望的不一致。  例如 Oracle 认为该数据块的类型为data(type=6),但实际却不是。
  • 数据块中的data_object_id 和 数据字典中的data_object_id不匹配

 

针对ORA-8103问题 我们优先推荐一些措施:

ORA-08103问题的诊断最好是能生成8103错误的ERROR STACK TRACE, 在TRACE中会记录具体引发8103的对象的OBJ和OBJD,这便于我们定位可能存在corruption的对象。

问题在于往往前台进程遇到ORA-08103错误不会在后台生成TRACE文件,这需要我们手动设置8103 触发ERRORSTACK的EVENTS:

ALTER SYSTEM SET EVENTS ’8103 TRACE NAME ERRORSTACK LEVEL 3′;

解决思路包括:
1. 通过OBJD和DBA定位到具体的表名和块号
2. 有条件的情况下对该表做一个analyze .. validate structure
3. 有条件的情况下对该表所在tablespace做一个 dbms_space_admin.ASSM_TABLESPACE_VERIFY
4. 有条件的情况下move这张表或者相关的分区,尝试绕过该问题
5. 有条件的情况下降该表或分区移动到MSSM表空间上,绕过该问题

execute dbms_space_admin.tablespace_verify(‘&tablespace_name’)
oradebug setmypid
oradebug tracefile_name

execute dbms_space_admin.assm_tablespace_verify(‘&tablespace_name’,dbms_space_admin.TS_VERIFY_BITMAPS)
oradebug setmypid
oradebug tracefile_name

 

 

针对不同的 analyze validate structure 后得到的结果 , 我们可以得到一些初步的结论:

 

如果执行 flush buffer cache之后再次analyze validate structure不再报ORA-8103错误则说明:

可能是完全正常的现象,之前的ORA-8103正是也因为对象正在被DROP/TRUNCATE而导致SELECT报ORA-8103。一般来说Call Stack会显示进程正尝试访问该段的segment header。 更多信息可以参考BUG 7441661

也可能该问题仅仅发生在buffer cache层,而没有发生在DISK上。通过flush buffer_cache若能解决,则一般是这种情况,往往是Buffer Cache管理的BUG 。

 

 

如果执行 flush buffer cache之后再次analyze validate structure再次报ORA-8103错误则说明:

如果dump对应的数据块发现 该块在逻辑上是完整一致的(也可以用bbed/dbv工具验证), 则有可能是Lost Write,则不是被其他对象重格式化使用了。

这里判断Lost Write的一个重要手段是 对块做recover/blockrecover,如果recover能修复该块,则说明是因为Lost Write引起了本ORA-8103问题,如果不是则说明99%的可能性是BUG引起的。

常见的一种现象是 使用第三方工具在数据库打开的情况下copy 数据库,这些工具的BUG可能导致copy 老的版本的block到目标新库中。

 

另一种可能是 extent盘区级别的不一致。 同一个数据块/extent 可能 同时属于 2个数据段segment,这导致其中的一个被后者覆盖。 通过recover的方式是无法修复这种场景的, 因为这种逻辑的讹误发生在表空间级别的extent信息上。 可以检查dba_extents/dba_segments/dba_free_space这些视图来确定问题数据块到底是否同时属于多个对象, 或者 一个数据块 同时出现在dba_extents/dba_segments/dba_free_space 三个视图中, 因为 used extent 不该出现在dba_free_space中,而free extent不该在dba_extents,当然要排除recyclebin中对象的影响。 绝大多数情况下这种extent逻辑不一致的现象, 被称作extent overlap , 通常是Oracle Space Management空间管理层面的BUG。

 

在对ORA-8103问题的诊断过程中 定位问题的OBJD异常重要。应当说准确地将ORA-8103错误与BUG定位起来是有难度的,因为这往往需要涉及到redo dump以发现到底是哪些opcode造成了后续的objd 或 block type 不一致。在一些BUG中我们发现,由于可能的变量陈旧,造成objd的结构未合理清除, 之后就发现block上的objd是错的了,可能遇到ORA-8103也可能是ORA-1410, 这引起了后续其他的逻辑讹误,以至于很难通过TRACE/REDO LOG DUMP来定位原始问题所在。 这也是为什么虽然在例如版本10.2.0.4上有几个ORA-8103的bug Note, 但这些BUG最终未被close为real software  bug即真的软件BUG , 大多都是不了了之,因为在用户现场的TRACE和REDO DUMP都未必能真实定位到问题所在,这也是为什么我们要说逻辑讹误的分析和处理原要比物理讹误来的复杂。

 

Maclean的经验是 在有大量Oracle DB的环境下 一年出个几次的逻辑/物理坏块是很正常的事情, 对于物理讹误 我们只要切实备份即可99%得解决。 而对于逻辑坏块可做的 事情不多, 打最新的补丁 开 db_block_checking、db_block_checksum几件事情而已。

 

值得一说的是 如果去读一下ORA-8103的一些Bug Note,可以发现使用 LOB、APPEND INSERT、PARALLEL INSERT、exchange partition 、Split partition、advanced compression、HCC 混合列压缩往往是引起ORA-8103的高危操作 , 但实际我们又不可能放弃上述操作。

 

目前已知ORA-8103相关的BUG 列表:

 

NB Bug Fixed Description
13910420 11.2.0.3.BP09, 12.1.0.0 ORA-8103 during insert / update of basicfile LOB in assm segment using space search cache
13725395 11.2.0.3.BP07, 11.2.0.4, 12.1.0.0 ORA-600 [kdzhFindHeadPiece: unnewed > 1] from load into HCC table
13700577 11.2.0.3.BP07, 11.2.0.4, 12.1.0.0 PQ slave dies with ORA-600 [kdblddr_2]
12747437 12.1.0.0 ORA-600 [ktspfmdb:objdchk_kcbnew_3] after purging single consumer queue table
12582839 11.2.0.3, 12.1.0.0 ORA-8103/ORA-600 [3020] on RMAN recovered locally managed tablespace
12321309 12.1.0.0 ORA-600 / ORA-8103 UNUSABLE state of partitioned index is not carried across by TABLESPACE transport using DataPump
11937253 11.2.0.2.6, 11.2.0.2.BP11, 11.2.0.3, 12.1.0.0 A Parallel query fails with ORA-8103 on an Active Dataguard Enviroment.
11850492 11.2.0.3, 12.1.0.0 ORA-8103 ORA-600 ORA-3113 on temporary tables using INDEX FAST FULL SCAN and DIRECT read
10385812 11.2.0.3, 12.1.0.0 ORA-1410 or ORA-8103 by queries with DIRECT READ while concurrent DIRECT INSERT
10329146 11.2.0.1.BP10, 11.2.0.2.2, 11.2.0.2.BP03, 11.2.0.2.GIBUNDLE02, 11.2.0.2.GIPSU02, 11.2.0.3, 12.1.0.0 Lost write in ASM with multiple DBWs and a disk is offlined and then onlined
+ 10209232 11.1.0.7.7, 11.2.0.1.BP08, 11.2.0.2.1, 11.2.0.2.BP02, 11.2.0.2.GIBUNDLE01, 11.2.0.3, 12.1.0.0 ORA-1578 / ORA-600 [3020] Corruption. Misplaced Blocks and Lost Write in ASM
10136415 11.2.0.3, 12.1.0.0 ORA-8103 on Partitioned IOT after partition maintenance
9965085 11.2.0.3, 12.1.0.0 ORA-1578 / ORA-8103 Temporary table block corruption / space wastage from PDML
9659614 10.2.0.5.3, 11.2.0.2, 11.2.0.3.5, 11.2.0.3.BP05, 12.1.0.0 Large trace file for ORA-8103
9651350 11.2.0.2.2, 11.2.0.2.BP05, 11.2.0.3, 12.1.0.0 Large redo dump and ORA-308 might be raised due to ORA-8103
9275027 11.2.0.2, 12.1.0.0 ORA-600 [kcbnew_3] can occur after TRUNCATE / DROP
9272086 11.1.0.7.4, 11.2.0.1.2, 11.2.0.1.BP06, 11.2.0.2, 12.1.0.0 ORA-8103 by a query on DBA_EXTENTS. Trace file with Block type: 0×44=NGLOB: Extent Map
8754670 11.2.0.2, 12.1.0.0 IMP-17 / ORA-8103 transporting a large dictionary managed tablespace
8740993 11.1.0.7.8, 11.2.0.2, 12.1.0.0 ORA-1410 / ORA-8103 on ADG STANDBY during table scan after DROP/TRUNCATE/SHRINK in PRIMARY
8725282 11.2.0.1.BP08, 11.2.0.2, 12.1.0.0 Corruption from cross platform transport of tablespace with securefile objects
8716064 11.2.0.2, 12.1.0.0 Analyze Table Validate Structure fails on ADG standby with several errors
+ 8597106 11.2.0.1.BP06, 11.2.0.2, 12.1.0.0 Lost Write in ASM when normal redundancy is used
8428523 11.2.0.2, 12.1.0.0 Alter Table Rename causes wrong results/ora-8103/hangs on ADG Standby.
7710827 11.2.0.2, 12.1.0.0 Index rebuild or Merge partition causes wrong results in concurrent reads instead of ORA-8103
7519406 10.2.0.5.1, 11.2.0.1.2, 11.2.0.1.BP06, 11.2.0.2, 12.1.0.0 Larger trace than needed for ORA-8103 under kteinicnt1
P 12330911 12.1 EXADATA LSI firmware for lost writes
8876094 11.1.0.7.2, 11.2.0.2 ORA-8103 by DBA_UNDO_EXTENTS or DBMS_SPACE_ADMIN.TABLESPACE_VERIFY on Block type: 0×25
9167831 11.2.0.2 ORA-8103 instead of ORA-1410
7650993 11.1.0.7.1, 11.2.0.1 ORA-8103 in a select at ADG standby database from table stored in ASSM tablespace
7432556 11.1.0.7.1, 11.2.0.1 ORA-8103 by Parallel Query on Partitioned Tables in BIGFILE Tablespaces
7390324 11.2.0.1 ANALYZE signals OERI [kcbgtcr_12]/ORA-8103 on bitmap index
7117200 11.2.0.1 ORA-8103 after TSPITR/PLUGIN tablespace from a restored Level 1 Backup
8825048 11.1.0.7.3 ORA-308/ORA-27037 when dumping archived log for ORA-8103. Dump when event 10736 level 4 is set
6337376 11.1.0.7 OERI:kcbgcur_3 / ORA-8103 after truncating a partition table with LOBs
9711472 11.1.0.6 ORA-8103 on operations for a partitioned LOB if any different partition is dropped
5637976 10.2.0.4, 11.1.0.6 ORA-8103/ORA-1410 from concurrent INSERT / export on ASSM tables
5083393 10.2.0.4, 11.1.0.6 DBA_FREE_SPACE FILE_ID / REL_FNO may be wrong
4592596 10.2.0.4, 11.1.0.6 Corruption (ORA-1410 / ORA-8103) from multi-table insert with direct load
6864586 10.2.0.5 ORA-8103 on partitioned table with a LOB column during analyze table with concurrent add/drop partition.
3569503 9.2.0.6, 10.2.0.4 PQ may signal a false ORA-8103 under load
13618170 ORA-8103 for create index online when the fix of bug 10027403 is installed
3966709 9.2.0.7, 10.1.0.4, 10.2.0.1 Range/object reuse prematurely (ORA-8103)
3868753 9.2.0.7, 10.1.0.5, 10.2.0.1 Concurrent export / INSERT of ASSM segment can fail with ORA-1410 / ORA-8103
+ 5523799 Various OERI (eg kcbgtcr_12) using ASSM managed segments – superceded
P* 6047085 Linux x64-64: SGA corruption / crash following any ORA-7445
* 3785200 9.2.0.6, 10.1.0.2 Corruption possible in automatic space managed segments
3083560 9.2.0.5, 10.1.0.2 ORA-1410 / ORA-8103 from direct path export if concurrent DML occurs
2619867 9.2.0.3, 10.1.0.2 OERI:[KCBGTCR_12] / ORA-8103 / ORA-1410 SELECTing from bitmap managed segment
2551000 9.2.0.4, 10.1.0.2 False ORA-1410 / ORA-8103 possible from ANALYZE COMPUTE/ESTIMATE STATISTICS
2333731 9.2.0.2 ORA-8103 possible in PQ slave
2105419 9.0.1.3, 9.2.0.1 ORA-8103 possible from PQ on bitmap managed segments with concurrent inserts
1998455 8.1.7.3, 9.0.1.3, 9.2.0.1 OERI:KCBGTCR_4 possible from long running DDL if referenced object dropped/truncated
1804299 9.0.1.1, 9.2.0.1 Rollback of Direct load can corrupt BITMAP managed segments / ORA-8103
1698789 9.2.0.1 Wrong results, ORA-1410, ORA-8103, OERI:25012 on SELECT of UNSCOPED REF with ROWID
1504967 9.2.0.1 ORA-8103 possible on READ ONLY standby after TRUNCATE on primary
1400739 8.1.7.1, 9.0.1.0 Block corruption/OERI:2023 /ORA-8103 can occur if TRUNCATE is interrupted (Ctrl-C)
1283521 8.1.7.0 ORA-8103 can occur on TRUNCATED cluster table
589855 7.3.3.6, 7.3.4.1 ORA:1578 or ORA:8103 selecting invalid ROWID
P 1053863 8.0.5.2, 8.0.6.2 NCR: ORA-8103 / corrupt read possible using async IO

 

对于ORA-8103问题的更多信息可以参考:

MOS文档Note 268302.1 ORA-8103 Diagnostics and Solution 

诊断ORA-08103错误

【数据恢复】利用构造ROWID实现无备份情况下绕过ORA-1578、ORA-8103、ORA-1410等逻辑/物理坏块问题

 

Oracle数据恢复专题

恢复恢复是Oracle中永恒的话题, 只要有数据 就有备份恢复的需求。 而在国内对于备份以及备份的可用性往往被企业所忽视。这造成了再数据库恢复上存在着东西方的差异。 更多的老外DBA把经历花在对Oracle内部原理和性能优化的研究上。

oracle data block structure

而我们国内 DBA似乎必须要精通一门额外的技术==》 在没有任何备份的情况下 恢复Oracle数据库中数据的技术!  虽然这在大多数情况下是屠龙之技, 但很多时候却又变成了衡量一个DBA技术水准的标准了,(这样不好..  不好)。

 

当然也并不是说 这种无备份下的数据恢复是无技术含金量的,实际上它们很需要对Oracle数据文件、数据块及其数据结构的理解,以及对数据字典构成的了解。

 

这里我们总结Oracle数据恢复专题的专题,包括一些在无备份情况下的数据恢复:例如DUL和BBED工具恢复等技术。

 

 

【数据恢复】利用构造ROWID实现无备份情况下绕过ORA-1578、ORA-8103、ORA-1410等逻辑/物理坏块问题

【数据恢复】ORA-600[kccpb_sanity_check_2]一例

Oracle rman中set newname可能很慢

如何清除Oracle控制文件中的无用记录,例如v$archived_log中的deleted归档日志记录

如何找回被create or replace覆盖的PL/SQL对象

Archivelog Completed Before VS UNTIL TIME

ASM丢失disk header导致ORA-15032、ORA-15040、ORA-15042 Diskgroup无法mount

Overcome ORA-600[4xxx] open database

数据恢复:解决ORA-600[kghstack_free2][kghstack_err+0068]一例

清理RMAN Catalog恢复目录

如何rename datafile name中存在乱码的数据文件

11g新特性recover corruption list

解决ORA-01578错误一例

Script:收集介质恢复诊断信息

如何重建SYSAUX表空间上的对象

Oracle数据恢复:解决ORA-00600:[4000] ORA-00704: bootstrap process failure错误一例

Script:检查数据库当前是否有备份操作在执行中

数据恢复:模拟2个逻辑坏块

Script:收集Oracle备份恢复信息

Oracle备份恢复:Rman Backup缓慢问题一例

了解rman catalog的兼容性

Oracle内部错误:ORA-00600[2608]一例

使用bbed解决ORA-01189错误

Fractured block found during backing up datafile

手动递增SCN号的几种方法:How to increase System Change Number by manual

DBMS_REPAIR example

 

【数据恢复】利用构造ROWID实现无备份情况下绕过ORA-1578、ORA-8103、ORA-1410等逻辑/物理坏块问题

对于无备份情况下的ORA-1578、ORA-8103、ORA-1410等逻辑/物理坏块问题,可以通过如下PL/SQL 构造ROWID的方式挽救绝大多数非坏块的数据, 一般分成 2种情况 有索引可以用来获取ROWID, 或者 没有索引、索引不可用,必须通过dbms_rowid.ROWID_CREATE来构造ROWID的方式。

 

对于有索引的情况,可以直接使用MOS上提供的脚本:

 

 

REM Create a new table based on the table that is producing errors with no rows:

create table 
as
select *
from   
where  1=2;

REM Create the table to keep track of ROWIDs pointing to affected rows:

create table bad_rows (row_id rowid
                      ,oracle_error_code number);
set serveroutput on

DECLARE
  TYPE RowIDTab IS TABLE OF ROWID INDEX BY BINARY_INTEGER;
  CURSOR c1 IS select /*+ index(tab1) */ rowid
  from  tab1
  where  is NOT NULL;
  r RowIDTab;
  rows NATURAL := 20000;
  bad_rows number := 0 ;
  errors number;
  error_code number;
  myrowid rowid;
BEGIN
  OPEN c1;
  LOOP
   FETCH c1 BULK COLLECT INTO r LIMIT rows;
   EXIT WHEN r.count=0;
   BEGIN
    FORALL i IN r.FIRST..r.LAST SAVE EXCEPTIONS
     insert into 
     select /*+ ROWID(A) */ 
     from  A where rowid = r(i);
   EXCEPTION
   when OTHERS then
    BEGIN
     errors := SQL%BULK_EXCEPTIONS.COUNT;
     FOR err1 IN 1..errors LOOP
      error_code := SQL%BULK_EXCEPTIONS(err1).ERROR_CODE;
      if error_code in (1410, 8103, 1578) then
       myrowid := r(SQL%BULK_EXCEPTIONS(err1).ERROR_INDEX);
       bad_rows := bad_rows + 1;
       insert into bad_rows values(myrowid, error_code);
      else
       raise;
      end if;
     END LOOP;
    END;
   END;
   commit;
  END LOOP;
  commit;
  CLOSE c1;
  dbms_output.put_line('Total Bad Rows: '||bad_rows);
END;
/

 

 

 

对于没有索引或者索引损坏的情况可以使用如下方法:

 

 

创建示例数据

create table maclean_tab1 (t1 int,t2 date default sysdate) tablespace users
partition by range(t1) 
(partition p1 values less than (10000),
partition p2 values less than (20000),
partition p3 values less than (30000),
partition p4 values less than (40000),
partition p5 values less than (50000),
partition p6 values less than (60000),
partition p7 values less than (70000),
partition p8 values less than (80000),
partition p9 values less than (90000),
partition p10 values less than (100000),
partition p11 values less than (110000),
partition p12 values less than (120000),
partition p13 values less than (130000),
partition p14 values less than (140000),
partition p15 values less than (150000),
partition p16 values less than (160000))
;

insert into maclean_tab1(t1) select rownum from dual connect by level<160000;

commit;

SQL> select count(*) from maclean_tab1;

  COUNT(*)
----------
    159999

exec dbms_stats.gather_table_stats(USER,'MACLEAN_TAB1');   

SQL> alter system flush buffer_cache;

System altered.

SQL> /

System altered.

随即采样一些块来做 坏块 约涉及到10个块的数据

set linesize 200 pagesize 1400

select dbms_rowid.rowid_block_number(rowid) blkid,
       dbms_rowid.rowid_relative_fno(rowid) rfile
  from maclean_tab1 sample(0.01)
 where rownum <= 200
 group by dbms_rowid.rowid_block_number(rowid),
          dbms_rowid.rowid_relative_fno(rowid)
 order by 1;

     BLKID      RFILE
---------- ----------
    741833          4
    741850          4
    741994          4
    742030          4
    742085          4
    742141          4
    742159          4
    742172          4
    742173          4
    742179          4

 制造坏块

    [oracle@vrh8 udump]$ rman target  /

RMAN> blockrecover datafile 4 block 741833,741850,741994,742030,742085,742141,742159,742172,742173,742179 clear;

Starting blockrecover at 21-APR-13
using channel ORA_DISK_1
Finished blockrecover at 21-APR-13

SQL> select count(*) from maclean_tab1;
select count(*) from maclean_tab1
                     *
ERROR at line 1:
ORA-01578: ORACLE data block corrupted (file # 4, block # 741833)
ORA-01110: data file 4:
'/s01/oradata/G10R25/datafile/o1_mf_users_8nx5srgb_.dbf'

通过blockrecover datafile block clear 构造了一系列坏块且没有备份 ,我们通过下列脚本挽回大部分可用数据

一个实际操作过程



drop table maclean_tab_backup;

create table maclean_tab_backup 
tablespace users 
nologging compress pctfree 0 pctused 99        --可以注释掉的
 as select * from maclean_tab1 where 1=0;

drop table bad_rows;
create table bad_rows (row_id rowid,oracle_error_code varchar2(50))
tablespace users 
nologging compress pctfree 0 pctused 99        --可以注释掉的;

set serveroutput on;
set timing on;

DECLARE
  TYPE RowIDTab IS TABLE OF ROWID INDEX BY BINARY_INTEGER;
  CURSOR Crowid_info IS
    select Do.DATA_OBJECT_ID dataid,
           DE.FILE_ID        fid,
           DE.BLOCK_ID       blkid,
           DE.BLOCKS         blkcnt
      from dba_objects DO, dba_extents DE
     where DO.OBJECT_NAME = 'MACLEAN_TAB1' 
     --and DE.PARTITION_NAME='&PARTITION_NAME'          --若指定分区则取消注释
       and nvl(DO.SUBOBJECT_NAME,'-1') = nvl(DE.PARTITION_NAME,'-1')
       and DO.OBJECT_NAME = DE.SEGMENT_NAME
       and DO.owner = 'SYS'
     order by 1, 2, 3 asc;
  bad_rows   number := 0;
  errors     varchar2(500);
  error_code varchar2(500);
  myrowid    rowid;
BEGIN
  /* Maclean Liu http://www.askmaclean.com * Copy Right 2013-4-20 */ 
  execute immediate 'alter session set commit_write=''batch,nowait'' ';
  for i in Crowid_info loop
    for j in 0 .. i.blkcnt - 1 loop
      for z in 0 .. 2000 loop
        begin
          myrowid := dbms_rowid.ROWID_CREATE(1,
                                             i.dataid,
                                             i.fid,
                                             i.blkid + j,
                                             z);
          insert into maclean_tab_backup
            select /*+ ROWID(A) */
             *
              from maclean_tab1 A
             where rowid = myrowid;
        EXCEPTION
          when OTHERS then
            BEGIN
              errors     := SQLERRM;
              error_code := SQLCODE;
              if (error_code like '%1410%' or error_code like '%8103%' or  error_code like '%1578%') then
                bad_rows := bad_rows + 1;
                insert into bad_rows values (myrowid, error_code);
                commit;
              else
                raise;
              end if;
            END;
            commit;
        end;
      end loop;
    end loop;
  end loop;
  dbms_output.put_line('Total Bad Rows: ' || bad_rows);
  commit;
END;
/

Elapsed: 00:01:10.16

SQL> select count(*) from maclean_tab_backup;

  COUNT(*)
----------
    155921

 ===>损失了少量的10个块的数据

 

 

原始脚本如下:

步骤1 创建备份表

create table <new table name>
as
select *
from   <original table name>
where  1=2;

步骤2 创建bad_rows表

create table bad_rows (row_id rowid,oracle_error_code varchar2(50));

步骤3 运行下列脚本

DECLARE
  TYPE RowIDTab IS TABLE OF ROWID INDEX BY BINARY_INTEGER;
  CURSOR Crowid_info IS
    select Do.DATA_OBJECT_ID dataid,
           DE.FILE_ID        fid,
           DE.BLOCK_ID       blkid,
           DE.BLOCKS         blkcnt
      from dba_objects DO, dba_extents DE
     where DO.OBJECT_NAME = '&TABNAME' 
     --and DE.PARTITION_NAME='&PARTITION_NAME'          --若指定分区则取消注释
       and nvl(DO.SUBOBJECT_NAME,'-1') = nvl(DE.PARTITION_NAME,'-1')
       and DO.OBJECT_NAME = DE.SEGMENT_NAME
       and DO.owner = '&OWNER'
     order by 1, 2, 3 asc;
  bad_rows   number := 0;
  errors     varchar2(500);
  error_code varchar2(500);
  myrowid    rowid;
BEGIN
  /* Maclean Liu http://www.askmaclean.com * Copy Right 2013-4-20 */ 
  execute immediate 'alter session set commit_write=''batch,nowait'' ';
  for i in Crowid_info loop
    for j in 0 .. i.blkcnt - 1 loop
      for z in 0 .. 2000 loop
        begin
          myrowid := dbms_rowid.ROWID_CREATE(1,
                                             i.dataid,
                                             i.fid,
                                             i.blkid + j,
                                             z);
          insert into &backup_table
            select /*+ ROWID(A) */
             *
              from &source_table A
             where rowid = myrowid;
        EXCEPTION
          when OTHERS then
            BEGIN
              errors     := SQLERRM;
              error_code := SQLCODE;
              if (error_code like '%1410%' or error_code like '%8103%' or  error_code like '%1578%') then
                bad_rows := bad_rows + 1;
                insert into bad_rows values (myrowid, error_code);
                commit;
              else
                raise;
              end if;
            END;
            commit;
        end;
      end loop;
    end loop;
  end loop;
  dbms_output.put_line('Total Bad Rows: ' || bad_rows);
  commit;
END;
/

构造ROWID绕过ORA-1578、ORA-8103、ORA-1410脚本下载

【数据恢复】ORA-600[kccpb_sanity_check_2]一例

kccpb_sanity_check_2内核函数kernel function负责监测控制文件的健康性,该ORA-600[kccpb_sanity_check_2]一般在alter database mount阶段发生; 该ORA-600[kccpb_sanity_check_2]发生的原因一般是 控制文件controlfile 块头的seq#号大于控制文件头中的seq#,所以该监测函数认为存在控制文件逻辑不一致。

该kccpb_sanity_check_2函数是从10gR2才引入了,换句话说9i没有这样的控制文件健康性监测,引入该特性的目的是为了检测出写丢失lost write和陈旧读stale read。

 

 

该ORA-600[kccpb_sanity_check_2]一般有2个argument代码:

ARGUMENTS:
Arg [a] seq# in control block header.
Arg [b] seq# in the control file header.
Arg [c] maclean

 

 

ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [kccpb_sanity_check_2], [53672], [53643], [0x000000000], [], [], [], []
Current SQL statement for this session:
ALTER DATABASE   MOUNT
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
ksedst+001c          bl       ksedst1              000000000 ? 700000010007FE0 ?
ksedmp+0290          bl       ksedst               104C1FCD0 ?
ksfdmp+02d8          bl       03F34734             
kgerinv+00dc         bl       _ptrgl               
kgeasnmierr+004c     bl       kgerinv              1105312C0 ? 110211598 ?
                                                   000000000 ? 000015018 ?
                                                   000000000 ?
kccpb_sanity_check+  bl       kgeasnmierr          11019B7D0 ? 1104A0040 ?
013c                                               104DFF150 ? 300000003 ?
                                                   000000000 ? 00000D1A8 ?
                                                   000000000 ? 00000D18B ?
kccbmp_get+00f8      bl       kccpb_sanity_check   FFFFFFFFFFFFFFFF ?
                                                   2700000027 ?
kccsed_rbl+008c      bl       kccbmp_get           1100745A0 ? 104E02124 ?
kccocx+08fc          bl       kccsed_rbl           100000068 ?
kccocf+0140          bl       kccocx               FFFFFFFFFFEB7C8 ? 0CBC08BD8 ?
                                                   16D694B2F ? 110231D90 ?
kcfcmb+0ab4          bl       kccocf               000000000 ? 000000000 ?
                                                   000000002 ? 10041C1F4 ?
kcfmdb+0054          bl       kcfcmb               0FFFED160 ? 7000000B9FF038E ?
                                                   000000003 ? 000000000 ?
                                                   000000003 ? 000000000 ?
                                                   000000000 ? 000000000 ?
adbdrv+06c0          bl       kcfmdb               110262390 ? 7000000BCFFA050 ?
                                                   7000000BCFFA470 ? 104F38E08 ?
opiexe+2d34          bl       adbdrv               
opiosq0+1ac8         bl       opiexe               000000001 ? 000000000 ?
                                                   FFFFFFFFFFF9148 ?
kpooprx+016c         bl       opiosq0              300000020 ? 110231D90 ?
                                                   7000000BD1038F8 ?
                                                   A400011019B7D0 ? 000000000 ?
kpoal8+03cc          bl       kpooprx              FFFFFFFFFFFB954 ?
                                                   FFFFFFFFFFFB700 ?
                                                   1600000016 ? 100000001 ?
                                                   000000000 ? A40000000000A4 ?
                                                   000000000 ? 1103ABA18 ?
opiodr+0b2c          bl       _ptrgl               
ttcpip+1020          bl       _ptrgl               
opitsk+117c          bl       01FC6908             
opiino+09d0          bl       opitsk               1EFFFFD920 ? 000000000 ?
opiodr+0b2c          bl       _ptrgl               
opidrv+04a4          bl       opiodr               3C102A89D8 ? 404C72CF0 ?
                                                   FFFFFFFFFFFF8E0 ? 0102A89D0 ?
sou2o+0090           bl       opidrv               3C02A2895C ? 400000020 ?
                                                   FFFFFFFFFFFF8E0 ?
opimai_real+01bc     bl       01FC3174             
main+0098            bl       opimai_real          000000000 ? 000000000 ?
__start+0090         bl       main                 000000000 ? 000000000 ?

    last wait for 'control file sequential read' wait_time=0.000511 sec, seconds since wait started=0
                file#=0, block#=27, blocks=1
                blocking sess=0x0 seq=23
    Dumping Session Wait History
     for 'control file sequential read' count=1 wait_time=0.000511 sec
                file#=0, block#=27, blocks=1
     for 'control file sequential read' count=1 wait_time=0.000957 sec
                file#=0, block#=1, blocks=1
     for 'CSS operation: action' count=1 wait_time=0.036477 sec
                function_id=41, =0, =0
     for 'CSS operation: action' count=1 wait_time=0.000200 sec
                function_id=41, =0, =0
     for 'CSS initialization' count=1 wait_time=0.060291 sec
                =0, =0, =0
     for 'control file sequential read' count=1 wait_time=0.000547 sec
                file#=0, block#=1, blocks=1
     for 'control file sequential read' count=1 wait_time=0.003763 sec
                file#=0, block#=3, blocks=20
     for 'control file heartbeat' count=1 wait_time=3.906271 sec
                =0, =0, =0
     for 'control file sequential read' count=1 wait_time=0.020452 sec
                file#=0, block#=3, blocks=20
     for 'control file sequential read' count=1 wait_time=0.000310 sec
                file#=0, block#=1, blocks=1

 

 

 

上例是一个ORA-600[kccpb_sanity_check_2]的实战案例,虽然出现了该错误,但是由于多路复用了controlfile控制文件,通过修改参数control_files ,发现其中第二个控制文件mount时未报错,可以确信仅有1个控制文件有问题,所以只需要dd 复制一下即可。

针对这个 实例mount阶段的ORA-600[kccpb_sanity_check_2]错误,一般有几种解决方法:

1、如果多路复用了控制文件,则未必所有控制文件都坏了,修改control_files参数一个个试过来,注意当 好的控制文件和坏的控制文件都在参数control_files里时是无法mount成功的

2、从备份中restore健康的控制文件出来

3、若没有备份,则需要手动重建控制文件了

Oracle rman中set newname可能很慢

Oracle rman中set newname可能很慢, 对于该问题有几种可能:

 

1、rman 软件bug ,具体请去MOS查询

2、打开了BACKUP OPTIMIZATION on,show all命令确认下,若是则 CONFIGURE BACKUP OPTIMIZATION OFF; 暂时关掉

3、 set newname 之前不要set until time之类的设置,最好也别allocate channel xx type sbt,可以在set newname之后再设不迟

4 、使用catalog 恢复目录替代controlfile模式往往可以绕过该问题

 

具体的set newname操作涉及到  sys . dbms_backup_restore . normalizeFileName和 dbms_rcvman . translateDataFile  2个存储过程:

 

FUNCTION normalizeFileName (fname IN varchar2) RETURN varchar2;

– Normalizes the file name according to the rules of the target
— database and returns the normalized filename.

— Input parameters:
— fname
— File name to be normalized.
— Exceptions:
— NAME-TOO-LONG (ora-19704)
— The specified file name is longer than the port-specific
— maximum file name length.

– translateDataFile translates the datafile name/number into
– a datafile number and creation SCN and filename. getDataFile must
– be called to obtain the translation info, just as for the other translate
– functions.
– Unlike the other translation functions, translateDatafile by name is always
– performed relative to current time. If an until setting is in effect,
– and if the filename is ambiguous, then an exception is raised. Ambiguous
– means that the filename refers to different datafile at the until time than
– it does at the current time. This happens only when a filename has been
– reused. When fno and ckpscn are passed, the filename and other info as of
– that scn is returned.

– Input parameters:
– fname
– name of the datafile to be translated.
– The name must be a normalized filename.
– fno
– The datafile number. If the datafile number was not in use at the
– until time, then an exception is raised.
– Exceptions:
– DATAFILE_DOES_NOT_EXIST (ORA-20201)
– the datafile to be translated does not exists
– Check that the recovery catalog is current.

PROCEDURE translateDataFile(
fname IN varchar2);

PROCEDURE translateDatafile(
fno IN number);

PROCEDURE translateDatafile(
fno IN number
,ckpscn IN number);

 

如何清除Oracle控制文件中的无用记录,例如v$archived_log中的deleted归档日志记录

有这样一种需求,虽然controlfile中的archivelog和其他一些备份信息会被覆盖重用,收到参数control_file_record_keep_time的影响,但是我们没法手动触发这种重用,这导致有时候controlfile中的archivelog 记录过多,可能导致一些备份、恢复过程中的”control file sequential read”等待事件。

已知的一些解决方案,包括:

 

1. 重建控制文件, 例如 backup controlfile to trace后重建该控制文件,但要求有数据文件均存在

2. 设置control_file_record_keep_time=0 然后等待记录被重用, 太过被动了….

 

这里再介绍一种手动清除v$ARCHIVED_LOG中不管是DELETED还是available 记录的方法,注意不要在你的产品数据库上这样做:

 

SQL> select count(*) from v$archived_log;

COUNT(*)
———-
553

SQL> /

COUNT(*)
———-
553
SQL> execute sys.dbms_backup_restore.resetCfileSection( 11);

PL/SQL procedure successfully completed.

SQL> select count(*) from v$archived_log;

COUNT(*)
———-
0

 

PROCEDURE resetCfileSection(record_type IN binary_integer );

– This procedure attempts to reset the circular controlfile section.

– Input parameters:
– record_type
– The circular record type whose controlfile section is to be reset.

 

execute sys.dbms_backup_restore.resetCfileSection( 11);  ==> 清理v$ARCHIVED_LOG对应的记录

execute sys.dbms_backup_restore.resetCfileSection(28);  ==>清理v$rman_status对应的记录

 

以上是2个例子,具体的section_id可以这样获得:

 

1* select rownum-1, type from v$controlfile_record_section
SQL> /

ROWNUM-1 TYPE
———- —————————-
0 DATABASE
1 CKPT PROGRESS
2 REDO THREAD
3 REDO LOG
4 DATAFILE
5 FILENAME
6 TABLESPACE
7 TEMPORARY FILENAME
8 RMAN CONFIGURATION
9 LOG HISTORY
10 OFFLINE RANGE
11 ARCHIVED LOG
12 BACKUP SET
13 BACKUP PIECE
14 BACKUP DATAFILE
15 BACKUP REDOLOG
16 DATAFILE COPY
17 BACKUP CORRUPTION
18 COPY CORRUPTION
19 DELETED OBJECT
20 PROXY COPY
21 BACKUP SPFILE
22 DATABASE INCARNATION
23 FLASHBACK LOG
24 RECOVERY DESTINATION
25 INSTANCE SPACE RESERVATION
26 REMOVABLE RECOVERY FILES
27 RMAN STATUS
28 THREAD INSTANCE NAME MAPPING
29 MTTR
30 DATAFILE HISTORY
31 STANDBY DATABASE MATRIX
32 GUARANTEED RESTORE POINT
33 RESTORE POINT

LOG History占用的控制文件记录数可以查询v$controlfile_record_section获得:

 

SQL> select * from v$controlfile_record_section where type=’LOG HISTORY’ ;

TYPE RECORD_SIZE RECORDS_TOTAL RECORDS_USED FIRST_INDEX LAST_INDEX LAST_RECID
—————————- ———– ————- ———— ———– ———- ———-
LOG HISTORY 56 584 563 300 278 570

control file sequential read

Reading from the controlfile. This happens in many cases, e.g. while:

- making a backup of the controlfiles.

- the shared info (between instances) from the controlfile.

- reading other blocks from the controlfiles.

- reading the header block.

Wait time

The wait time is the elapse time of the read.

Parameters

file#

This identifies the controlfile that Oracle7 is reading from and with the following SQL statement one can determine the name of the controlfile:

select *

from x$kcccf

where indx = file#

block#

Blocknumber in the controlfile from where we are starting to read. The blocksize is the as the physical blocksize of the port (normally 512 bytes, some UNIX ports have 1 Kilobytes or 2 Kilobytes).

blocks

The number of blocks that we are trying to read.

Advise

If the wait time is too long (more then average I/O speed), check if the controlfiles are not a disk that is too busy. This could really impact Parallel Server performance as some of the synchronization between instances is done through the controlfiles.