Fractured block found during backing up datafile

一套AIX上的10.2.0.3系统,在rman备份期间告警日志出现如下记录:

======================= alert log record ============================
Hex dump of (file 35, block 1087687) in trace file /oracle/product/10.2.0/admin/MS/udump/ms_ora_103548.trc
Corrupt block relative dba: 0x08d098c7 (file 35, block 1087687)
Fractured block found during backing up datafile
Data in bad block:
 type: 6 format: 2 rdba: 0x08d098c7
 last change scn: 0x0006.44443e06 seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x6d910601
 check value in block header: 0xc0b0
 computed block checksum: 0x4286
Reread of blocknum=1087687, file=/dev/vx/rdsk/oradgMS/lv_ms_DB31. found valid data

=========== trace information for process 103548 ========================
Corrupt block relative dba: 0x08d098c7 (file 35, block 1087687)
Fractured block found during backing up datafile
Data in bad block:
 type: 6 format: 2 rdba: 0x08d098c7
 last change scn: 0x0006.44443e06 seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x6d910601
 check value in block header: 0xc0b0
 computed block checksum: 0x4286
Reread of blocknum=1087687, file=/dev/vx/rdsk/oradgMS/lv_ms_DB31. found valid data

可以看到该块的type为6,下面列出了Oracle 9.2中已经存在的各种块类型:

Type Description
1 KTU UNDO HEADER
2 KTU UNDO BLOCK
3 KTT SAVE UNDO HEADER
4 KTT SAVE UNDO BLOCK
5 DATA SEGMENT HEADER
6 trans data
7 Unknown
8 Unknown
9 Unknown
10 DATA SEGMENT FREE LIST BLOCK
11 Unknown
12 DATA SEGMENT HEADER WITH FREE LIST BLOCKS
13 Compatibility segment
14 KTU UNDO HEADER W/UNLIMITED EXTENTS
15 KTT SAVE UNDO HEADER W/UNLIMITED EXTENTS
16 DATA SEGMENT HEADER – UNLIMITED
17 DATA SEGMENT HEADER WITH FREE LIST BLKS – UNLIMITED
18 EXTENT MAP BLOCK
19 Unknown
20 Unknown
21 Unknown
22 DATA SEGMENT FREE LIST BLOCK WITH FREE BLOCK COUNT
23 BITMAPPED DATA SEGMENT HEADER
24 BITMAPPED DATA SEGMENT FREELIST
25 BITMAP INDEX BLOCK
26 BITMAP BLOCK
27 LOB BLOCK
28 KTU BITMAP UNDO HEADER – LIMITED EXTENTS
29 KTFB Bitmapped File Space Header
30 KTFB Bitmapped File Space Bitmap
31 TEMP INDEX BLOCK
32 FIRST LEVEL BITMAP BLOCK
33 SECOND LEVEL BITMAP BLOCK
34 THIRD LEVEL BITMAP BLOCK
35 PAGETABLE SEGMENT HEADER
36 PAGETABLE EXTENT MAP BLOCK
37 EXTENT MAP BLOCK OF SYSTEM MANAGED UNDO SEGMENT
38 KTU SMU HEADER BLOCK
39 Unknown
40 PAGETABLE MANAGED LOB BLOCK
41 Unknown
42 Unknown
43 Unknown
44 Unknown
45 Unknown
46 Unknown
47 Unknown

表和索引的块均可能为type 6的trans data;也就是说rman在备份期间读取到该数据块,并在初次读取时发现该块断裂了(Fractured),但Fractured并不代表块就真的corrupted了;从告警日志看rman在初次读取发现该块Fractured后,又再次读取时发现数据块已经恢复正常(found valid data)。所以上述告警日志并代表所列出的数据块存在讹误,很有可能是该数据块所在数据文件在备份期间发生了剧烈的IO操作,当rman读取到该数据块时可能存储正在对其进行写的操作,所以rman在第一次读取时认为该快断裂了(Fractured);之后rman对该块进行reread发现”断裂”现象已不存在,而”Corrupt block”仅仅是一种假象;针对上述问题可以对表或索引进行进一步的analyze..validate操作以确保不存在坏块。
同时上述”Corrupt block误报”现象极有可能是因为在Rman备份期间个别数据文件的IO过于活跃所致(如频繁的dml操作),建议在磁盘活跃度低的时间段运行rman备份工作。

参考文档:

Fractured Block Messages in Alert.log During RMAN Backup of Datafile
    * fact: Oracle Server - Enterprise Edition 8
    * fact: Oracle Server - Enterprise Edition 9
    * fact: Recovery Manager (RMAN)
    * symptom: Fractured block found during backup up datafile
    * symptom: Reread of blocknum found some corrupt data
    * symptom: Analyze table validate structure cascade returns no errors
    * change: NOTE ROLE: The messages are of the form Reread of blocknum=36256, 
       file=/pdscdata/pdsclive/data1/dispatch_data_large2. dbf. 
found same corrupt data *** Corrupt block relative dba: 0xfc008dc0 (file 63, block 36288) 
Fractured block found during backing up datafile Data in bad block - 
type: 0 format: 0 rdba: 0x00000000 last change scn: 0x0000.00000000 seq: 0x0 flg: 0x00 consistency 
value in tail: 0x53494e53 check value in block header: 0x0, block checksum disabled 
spare1: 0x0, spare2: 0x0, spare3: 0x0
    * cause: RMAN backups of datafile are being performed while the datafile is involved in heavy I/O. 
RMAN reads Oracle blocks from disk. If it finds that the block is fractured, which means it is being actively used, 
it performs a reread of the block. If that fails again then the block is assumed to be corrupt. 
By identifying the object that these blocks belong to by following Handling Oracle Block Corruptions in 
Oracle7/8/8i and performing an analyze .. validate structure cascade on the object involved you can 
confirm that the object is not corrupt. 

fix:
Run the backups when the tablespace has less I/O activity.

TSM配置导致RMAN备份expired

下午协助客户做异机备份恢复测试,平台是AIX 5.3使用tsm备份。在源生产机上备份了数据库和控制文件,准备从磁带库上恢复到测试机时RMAN报找到不备份的控制文件,之后crosscheck了一把:
[Read more…]

图文详解安装NetBackup 6.5备份恢复Oracle 10g rac 数据库(修订)

我们使用Linux平台进行测试,OS版本为Oracle Enterprise Linux 5.5 x86_64:
[root@nas servsoft]# cat /etc/issue
Enterprise Linux Enterprise Linux Server release 5.5 (Carthage)
Kernel \r on an \m

Netbackup 6.0仅支持2.4内核的Linux版本,2.6内核的Linux版本(主流的包括RHEL4,5 Centos等)需要使用NBU,只能安装Netbackup 6.5或以上版本。
现在我们有三台主机: rh1(rac的2号节点),rh2(rac的1号节点),nas(NBU Server)。
首先需要安装的是Netbackup Server端软件,当然你需要用到安装介质,你可以尝试在Veritas的官方网站下载到最新的版本;获取到安装介质后,我们首先要解压它:

[root@nas netbackup]# cp NetBackup_6.5_LinuxRedhat2.6.tar.gz /tmp
[root@nas tmp]# gunzip NetBackup_6.5_LinuxRedhat2.6.tar.gz
[root@nas tmp]# tar -xvf NetBackup_6.5_LinuxRedhat2.6.tar
在进入安装前确认xinetd服务正确运行着:
[root@nas tmp]# service xinetd status
xinetd (pid  2886) is running...
[root@nas NB_65_LinuxR_x86_20070723]# ./install
Do you want to install NetBackup and Media Manager files? [y,n] (y) y
NetBackup and Media Manager are normally installed in /usr/openv.
Is it OK to install in /usr/openv? [y,n] (y) y
Reading NetBackup files from /tmp/NB_65_LinuxR_x86_20070723/linuxR_x86/anb
...................
Enter the full path name to the directory where the
appropriate installics script is located followed by
a  to continue. This script will then install
the package(s).
OR
Enter q to stop this install and abort.
此时我们需要输入Netbackup ISC(Infrastructure Core Services)软件所在的目录,当然你也可以从Veritas官方网站下载到该软件包,尝试解压:
[root@nas tmp]# cp NetBackup_6.5_ICS_LinuxX86.tar.gz /tmp
[root@nas tmp]# cd /tmp
[root@nas tmp]# gunzip NetBackup_6.5_ICS_LinuxX86.tar.gz
[root@nas tmp]# tar -xvf NetBackup_6.5_ICS_LinuxX86.tar
则此时ISC安装介质位于/tmp/NB_65_ICS_1.4.37.0_LinuxX86下,在原终端窗口中输入该目录
Enter q to stop this install and abort.
/tmp/NB_65_ICS_1.4.37.0_LinuxX86
Installing VRTSpbx...
A NetBackup Server or Enterprise Server license key is needed
for installation to continue.
Enter license key:
继续安装,此时需要输入您所购买的License注册码;如果您没有购买该软件但仍想使用的话,
可以尝试下面一串字符:DEX6-23FJ-T92R-O4O4-O4O4-K777-7777-EPXP-3XO。
Enter license key: DEX6-23FJ-T92R-O4O4-O4O4-K777-7777-EPXP-3XO
DEX6-23FJ-T92R-O4O4-O4O4-K777-7777-EPXP-3XO:
NetBackup Enterprise Server Base product with all the features enabled
has been registered.
All additional keys should be added at this time.
Do you want to add additional license keys now? [y,n] (y) n
Use /usr/openv/netbackup/bin/admincmd/get_license_key
to add, delete or list license keys at a later time.
Installing NetBackup Enterprise Server version: 6.5
If this machine will be using a different network interface than the
default (nas), the name of the preferred interface should be used
as the configured server name.  If this machine will be part of a
cluster, the virtual name should be used as the configured server name.
Would you like to use "nas" as the configured
name of the NetBackup server? [y,n] (y) y
Is nas the master server? [y,n] (y) y
Do you have any media servers? [y,n] (n) n
Checking /etc/services for the needed NetBackup and Media Manager services.
Copying original /etc/services file to /etc/services.NBU_062910.14:27:41
Editing /etc/services to update NetBackup and Media Manager services.
/etc/services will be updated to add the following entries for
NetBackup/Media Manager.
bpjobd  13723/tcp       bpjobd
vmd     13701/tcp       vmd
acsd    13702/tcp       acsd
tl8cd   13705/tcp       tl8cd
tldcd   13711/tcp       tldcd
odld    13706/tcp       odld
tl4d    13713/tcp       tl4d
tshd    13715/tcp       tshd
tlmd    13716/tcp       tlmd
tlhcd   13717/tcp       tlhcd
rsmd    13719/tcp       rsmd
...................

好了Netbackup Server端软件已经在NAS主机上安装完成,接下来我们需要进一步配置备份策略。

将/usr/openv/netbackup/bin路径加入到你的用户环境变量PATH中,以方便调用相关执行文件;

并在具有X11 forwarding功能的软件中(譬如Xmanager)中输入jnbSA命令,

您可能遭遇java.lang.UnsatisfiedLinkError: /usr/openv/java/jre/lib/i386/libawt.so: libXp.so的错误,

一般是由于没有安装libXp(i386和x86_64版本的都装一下)包所导致的。

正确安装的话输入jnbSA命令可以看到以下界面:

接着我们需要定义存储单元(Storage Unit),如果你同我一样没有真实的磁带机的话那么我们可以定义普通Disk

类型的存储单元, 选择Netbackup Management->Storage -> Storage Unit,在右边分隔栏右键点击New Storage Unit,

为你的存储单元起一个名字,并输入相关存储目录:

接下来点击NetBackup Management->Policies 选项定义Oracle备份使用到的备份策略,启用Backup Policy Configuration Wizard,并选择Oracle为备份策略类型:

在客户端列表(client list)中加入需要备份2台RAC所在主机,分别为rh1,rh2;硬件与操作系统选择Linux,Redhat2.6

好了,server端的配置完成了,接下来我们安装client端软件,安装前确认你已经获得了相关安装介质,以NBU6.5举例来说你需要有:NetBackup_6.5_CLIENTS2.tar.gz和NetBackup_6.5_UnixOptions.tar.gz 分别为Client端和Oracle Agent软件。

[root@rh2 tmp]# gunzip  NetBackup_6.5_UnixOptions.tar.gz
[root@rh2 tmp]# gunzip  NetBackup_6.5_CLIENTS2.tar.gz
[root@rh2 tmp]# tar -xvf NetBackup_6.5_UnixOptions.tar
[root@rh2 tmp]# tar -xvf NetBackup_6.5_CLIENTS2.tar
[root@rh2 tmp]# cd NB_65_CLIENTS2_20070723/
[root@rh2 NB_65_CLIENTS2_20070723]# ./install
Symantec Installation Script
Copyright 1993 - 2007 Symantec Corporation, All Rights Reserved.
Installing NetBackup Client Software
NOTE:  To install NetBackup Server software, insert the appropriate
NetBackup Server cdrom.
Do you wish to continue? [y,n] (y) y
Do you want to install the NetBackup client software for this client? [y,n] (y) y
This package will install Linux/RedHat2.6 client.
This package will install NetBackup client 6.5.
Enter the name of the NetBackup server : nas
Would you like to use "rh2" as the configured
name of the NetBackup client? [y,n] (y) y
........................
File /usr/openv/tmp/install_trace.10994 contains a trace of this install.
That file can be deleted after you are sure the install was successful.
[root@rh2 tmp]# cd NB_65_UOptions_20070723/
root@rh2 NB_65_UOptions_20070723]# ./install
Symantec Installation Script
Copyright 1993 - 2007 Symantec Corporation, All Rights Reserved.
Installation Options
1 NetBackup Add-On Product Software
2 NetBackup Database Agent Software
q To quit from this script
Choose an option [default: q]: 2
**********
There are two ways to install database agent software.
1.  Remote Installation:  Loads the software on a server with
the intent of pushing database software out to affected clients.
2.  Local Installation:   Loads and installs the software only to this
local machine.
**********
Do you want to do a local installation? [y,n] (n) y
**********
NetBackup Database Agent Installation
Choose the Database Agents you wish to install
one at a time or select Install All Database Agents.
1)  NetBackup for DB2
2)  NetBackup for Informix
3)  NetBackup for Lotus Notes
4)  NetBackup for Oracle
5)  NetBackup for SAP
6)  NetBackup for Sybase
7)  Install All Database Agents
q)  Done Selecting Agents
x)  Exit from this Script
Choose an option: 4
Choose an option: q
You have chosen to install these Database Agents:
NetBackup for Oracle
Is this list correct? [y,n] (y) y
**********
Of the agents selected, the following are supported
on this platform and will be installed:
Oracle
Loading the Database Agent packages into the
/usr/openv/netbackup/dbext directory and installing.
**********
Installing NetBackup for Oracle
Installing NetBackup for Oracle...
..........................
NetBackup for Oracle installation completed.
完成NBU客户端和Netbackup for Oracle Agent安装后,我们还需要对MML介质库文件进行链接,使用dba或oinstall组账户执行/usr/openv/netbackup/bin/oracle_link文件:
[root@rh2 NB_65_UOptions_20070723]# su - maclean
[maclean@rh2 ~]$ cd /usr/openv/netbackup/bin/
[maclean@rh2 bin]$ ./oracle_link
Tue Jun 29 19:22:28 EDT 2010
All Oracle instances should be shutdown before running this script.
Please log into the Unix system as the Oracle owner for running this script
Do you want to continue? (y/n) [n]
[maclean@rh2 bin]$ echo $ORACLE_HOME
/s01/rac10g
[maclean@rh2 bin]$ ./oracle_link
Tue Jun 29 19:22:35 EDT 2010
All Oracle instances should be shutdown before running this script.
Please log into the Unix system as the Oracle owner for running this script
Do you want to continue? (y/n) [n] y
LIBOBK path: /usr/openv/netbackup/bin
ORACLE_HOME: /s01/rac10g
Oracle version: 10.2.0.5.0
Platform type: x86_64
Linking LIBOBK:
ln -s /usr/openv/netbackup/bin/libobk.so64 /s01/rac10g/lib/libobk.so
Done

接下来在rh2主机上进行备份测试:

[maclean@rh2 bin]$ rman target /
Recovery Manager: Release 10.2.0.5.0 - Production on Tue Jun 29 19:26:00 2010
Copyright (c) 1982, 2007, Oracle.  All rights reserved.
connected to target database: RACDB (DBID=720516428)
RMAN> run
2> { allocate channel c1 type sbt parms="ENV=(NB_ORA_SERV=nas,NB_ORA_POLICY=racdb,NB_ORA_CLIENT=rh2)";
3> backup current controlfile;
4> release channel c1;
5> }
RMAN> run
2> { allocate channel c1 type sbt parms="ENV=(NB_ORA_SERV=nas,NB_ORA_POLICY=racdb,NB_ORA_CLIENT=rh2)";
3> backup current controlfile;
4> release channel c1;
5> }
using target database control file instead of recovery catalog
allocated channel: c1
channel c1: sid=136 instance=racdb1 devtype=SBT_TAPE
channel c1: Veritas NetBackup for Oracle - Release 6.5 (2007072323)
Starting backup at 29-JUN-10
channel c1: starting full datafile backupset
channel c1: specifying datafile(s) in backupset
including current control file in backupset
channel c1: starting piece 1 at 29-JUN-10
channel c1: finished piece 1 at 29-JUN-10
piece handle=03lhfi11_1_1 tag=TAG20100629T192729 comment=API Version 2.0,MMS Version 5.0.0.0
channel c1: backup set complete, elapsed time: 00:00:37
Finished backup at 29-JUN-10
released channel: c1
如上所示成功备份了当前控制文件。
RMAN> run
2> {
3> allocate channel c1 type sbt parms="ENV=(NB_ORA_SERV=nas,NB_ORA_POLICY=racdb)";
4> backup archivelog all delete input;
5> release channel c1;
6> }
using target database control file instead of recovery catalog
allocated channel: c1
channel c1: sid=136 instance=racdb1 devtype=SBT_TAPE
channel c1: Veritas NetBackup for Oracle - Release 6.5 (2007072323)
Starting backup at 29-JUN-10
channel c1: starting archive log backupset
channel c1: specifying archive log(s) in backup set
input archive log thread=1 sequence=1 recid=2 stamp=722901460
input archive log thread=1 sequence=2 recid=4 stamp=722901476
input archive log thread=1 sequence=3 recid=5 stamp=722901499
input archive log thread=1 sequence=4 recid=6 stamp=722904852
input archive log thread=2 sequence=1 recid=1 stamp=722901426
input archive log thread=2 sequence=2 recid=3 stamp=722901470
input archive log thread=2 sequence=3 recid=7 stamp=722904852
channel c1: starting piece 1 at 29-JUN-10
channel c1: finished piece 1 at 29-JUN-10
piece handle=06lhfjqr_1_1 tag=TAG20100629T195819 comment=API Version 2.0,MMS Version 5.0.0.0
channel c1: backup set complete, elapsed time: 00:00:46
channel c1: deleting archive log(s)
archive log filename=/arch/1_1_722899663.dbf recid=2 stamp=722901460
archive log filename=/arch/1_2_722899663.dbf recid=4 stamp=722901476
archive log filename=/arch/1_3_722899663.dbf recid=5 stamp=722901499
archive log filename=/arch/1_4_722899663.dbf recid=6 stamp=722904852
archive log filename=/arch/2_1_722899663.dbf recid=1 stamp=722901426
archive log filename=/arch/2_2_722899663.dbf recid=3 stamp=722901470
archive log filename=/arch/2_3_722899663.dbf recid=7 stamp=722904852
Finished backup at 29-JUN-10
Starting Control File and SPFILE Autobackup at 29-JUN-10
piece handle=c-720516428-20100629-01 comment=API Version 2.0,MMS Version 5.0.0.0
Finished Control File and SPFILE Autobackup at 29-JUN-10
released channel: c1

11g Release 2 enhanced Tablespace Point In Time Recovery

11g release 2中引入了针对被dropped掉的表空间的表空间时间点恢复,这是一种十分有用的新特性。TSPITR(TablesSpace Point In Time Recovery)在10g中就能做到自动创建辅助实例以恢复表空间到某个时间点,但在10g中是无法恢复一个已经被drop掉的表空间的。如同10g中一样11g仍旧可以利用全自动的TSPITR恢复被drop的表空间;Oracle会自动创建并启动辅助实例,且仅仅还原那些恢复所需的控制文件,system,sysaux,undo表空间及目标表空间,这些工作都将在用户指定的辅助目的地’Auxiliary Destination’中完成;之后Oracle将进一步使用辅助实例recover目标表空间到指定的时间点,并将其中的数据以Data Pump传输表空间的形式倒回到原数据库当中。
接下来我们要具体测试这一新特性,我们会创建一个示例表空间并在该表空间上产生少量数据,之后我们将对数据库进行备份,drop目标示例表空间,并在RMAN中使用TSPITR的方式将已经被drop掉的表空间恢复回来。在正式drop表空间前我们当然需要留意时间点或者当时的scn号,以保证正常恢复,同时在测试时使用recovery catalog恢复目录,虽然我们同样可以不用。
[Read more…]

rman hang on SQL*Net message from client

有这样一个问题,平台为HP-UX(B.11.31 U ia64),Oracle版本为10.2.0.4 single instance,RMAN自动备份autobackup controlfile时出现hang症状,等待事件为SQL*Net message from client, 这还仅仅是使用最简单的NOCATALOG+ Disk Device的情况,没有MML层面的活动。

针对该问题对RMAN服务进程做了后台TRACE,发现stack call总是hang在_read_sys=>KERNEL内核态函数上,感觉与ORACLE的关系不大,应当是HP-UX C函数调用造成的问题,例如:

10046 trace:

PARSING IN CURSOR #3 len=278 dep=0 uid=0 oct=3 lid=0 tim=9513124502191 hv=3071086789 ad=’bd7447d0′
select sofar, context, start_time from v$session_longops where (start_time > nvl(:1, sysdate-100) or start_time = nvl(:2, sysdate+100)) and sid = :3 and serial# = :4 and opname like ‘RMAN:%’ order by start_time desc, context desc
END OF STMT
PARSE #3:c=0,e=3005,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,tim=9513124502186
WAIT #3: nam=’SQL*Net message to client’ ela= 1 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=9513124502275
WAIT #3: nam=’SQL*Net message from client’ ela= 234 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=9513124502527
BINDS #3:
kkscoacd
Bind#0
oacdty=12 mxl=07(07) mxlc=00 mal=00 scl=00 pre=00
oacflg=01 fl2=1000000 frm=00 csi=00 siz=64 off=0
kxsbbbfp=9fffffffbf3d73b0 bln=07 avl=00 flg=05
Bind#1
oacdty=12 mxl=07(07) mxlc=00 mal=00 scl=00 pre=00
oacflg=01 fl2=1000000 frm=00 csi=00 siz=0 off=8
kxsbbbfp=9fffffffbf3d73b8 bln=07 avl=00 flg=01
Bind#2
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=01 fl2=1000000 frm=00 csi=00 siz=0 off=16
kxsbbbfp=9fffffffbf3d73c0 bln=22 avl=03 flg=01
value=1875
Bind#3
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=01 fl2=1000000 frm=00 csi=00 siz=0 off=40
kxsbbbfp=9fffffffbf3d73d8 bln=22 avl=04 flg=01
value=50006
EXEC #3:c=10000,e=2859,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,tim=9513124505442
FETCH #3:c=0,e=6030,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,tim=9513124511493
WAIT #3: nam=’SQL*Net message to client’ ela= 1 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=9513124511560
WAIT #3: nam=’SQL*Net message from client’ ela= 56 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=9513124511690
BINDS #3:
kkscoacd
Bind#0
oacdty=12 mxl=07(07) mxlc=00 mal=00 scl=00 pre=00
oacflg=01 fl2=1000000 frm=00 csi=00 siz=64 off=0
kxsbbbfp=9fffffffbf3d6360 bln=07 avl=00 flg=05
Bind#1
oacdty=12 mxl=07(07) mxlc=00 mal=00 scl=00 pre=00
oacflg=01 fl2=1000000 frm=00 csi=00 siz=0 off=8
kxsbbbfp=9fffffffbf3d6368 bln=07 avl=00 flg=01
Bind#2
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=01 fl2=1000000 frm=00 csi=00 siz=0 off=16
kxsbbbfp=9fffffffbf3d6370 bln=22 avl=03 flg=01
value=1875
Bind#3
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=01 fl2=1000000 frm=00 csi=00 siz=0 off=40
kxsbbbfp=9fffffffbf3d6388 bln=22 avl=04 flg=01
value=50006
EXEC #3:c=0,e=275,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,tim=9513124512027
FETCH #3:c=10000,e=5688,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,tim=9513124517733
WAIT #3: nam=’SQL*Net message to client’ ela= 1 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=9513124517765
WAIT #3: nam=’SQL*Net message from client’ ela= 35 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=9513124517853
STAT #3 id=1 cnt=0 pid=0 pos=1 obj=0 op=’SORT ORDER BY (cr=0 pr=0 pw=0 time=11729 us)’
STAT #3 id=2 cnt=0 pid=1 pos=1 obj=0 op=’FIXED TABLE FULL X$KSULOP (cr=0 pr=0 pw=0 time=11697 us)’
WAIT #0: nam=’SQL*Net message to client’ ela= 0 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=9513124517949
WAIT #0: nam=’SQL*Net message from client’ ela= 1629803 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=9513126147781
BINDS #2:
kkscoacd
Bind#0
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=01 fl2=1000000 frm=00 csi=00 siz=72 off=0
kxsbbbfp=9fffffffbf3d7ef8 bln=22 avl=03 flg=05
value=1782
Bind#1
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=01 fl2=1000000 frm=00 csi=00 siz=0 off=24
kxsbbbfp=9fffffffbf3d7f10 bln=22 avl=04 flg=01
value=62996
Bind#2
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=01 fl2=1000000 frm=00 csi=00 siz=0 off=48
kxsbbbfp=9fffffffbf3d7f28 bln=22 avl=02 flg=01
value=2

………….

RMAN DEBUG

DBGRCVMAN: resetAll
DBGRCVMAN: setRAflags kindMask=255 containerMask=15 actionMask=63
DBGRCVMAN: ENTERING setComputeRecoveryActionMasks
DBGRCVMAN: EXITING setComputeRecoveryActionMasks
DBGRCVMAN: allIncarnations is set to FALSE
DBGRCVMAN: reset transclause
DBGRCVMAN: *****BsRecCache Statistics*****
DBGRCVMAN: Cache size=0 hit=0
DBGRCVMAN: rcvRecBackupAge= 0
DBGRCVMAN: thisBackupAge= 0
DBGRCVMAN: ENTERING setRedoLogDeletionPolicy
DBGRCVMAN: setRedoLogDeletionPolicy with policy = TO NONE
DBGRCVMAN: setRedoLogDeletionPolicy with alldest = 0
DBGRCVMAN: ENTERING validateStandbyConfig1
DBGRCVMAN: policy = TO NONE
DBGRCVMAN: alldest = 0
DBGRCVMAN: EXITING validateStandbyConfig1 with target = NULLwith status = 1
DBGRCVMAN: EXITING setRedoLogDeletionPolicy with policy = TO NONE with target = NULL with bind = MANDATORY
DBGMISC: krmknmtr: the parse tree after name translation is: [13:13:53.118]
DBGMISC: EXITED krmknmtr with status Control File and SPFILE Autobackup [13:13:53.118] elapsed time [00:00:00:00.021]
DBGMISC: krmkdps: this_reset_scn=56498483682 [13:13:53.118]
DBGMISC: krmkdps: this_reset_time=02-SEP-09 [13:13:53.119]
DBGMISC: krmkdps: untilSCN= [13:13:53.119]
DBGMISC: krmkdps: untilTime= [13:13:53.119]
DBGMISC: krmkdps: getRA_completedAfter= [13:13:53.119]
DBGMISC: krmkdps: getRA_completedBefore= [13:13:53.119]
DBGMISC: krmkdps: getRA_likePattern= [13:13:53.120]
DBGMISC: krmkdps: getRA_containerMask=15 [13:13:53.120]
DBGMISC: krmkdps: getRA_actionMask=63 [13:13:53.120]
DBGMISC: krmkdps: computeRA_allRecords0 [13:13:53.120]
DBGMISC: krmkdps: computeRA_fullBackups=1 [13:13:53.120]
DBGMISC: krmkdps: allIncarnations=0 [13:13:53.120]
DBGMISC: ENTERED krmkbaut [13:13:53.121]
DBGMISC: Entering krmkgconf [13:13:53.121]

DBGSQL: EXEC SQL AT TARGET select controlfile_type ,controlfile_change# ,controlfile_sequence# ,controlfile_created ,decode(offr.records_used,0,0,((offr.last_recid-offr.records_used)+1)) into :b1,:b2,:b3,:b4,:b5 from v$database ,v$controlfile_record_section offr where offr.type=’OFFLINE RANGE’ [13:13:53.121]
DBGSQL: sqlcode=0 [13:13:53.125]
DBGSQL: :b1 = “CURRENT”
DBGSQL: :b2 = 152792315080
DBGSQL: :b3 = 7538564
DBGSQL: :b4 = “09-AUG-09″
DBGSQL: :b5 = 0
DBGMISC: krmkgconf: Configuration is ENCRYPTION FOR DATABASE [13:13:53.126]

DBGSQL: EXEC SQL AT TARGET declare first boolean ; conf# number ; begin if ( :first > 0 ) then first := TRUE ; else first := FALSE ; end if ; dbms_rcvman . getConfig ( conf# , :name:name_i , :value:value_i , first ) ; end ; [13:13:53.126]
DBGSQL: sqlcode=1403 [13:13:53.127]
DBGMISC: Function kmkgconf created the following tree: [13:13:53.128]
DBGMISC: EXIT
DBGMISC: Exiting krmkgconf [13:13:53.128]
DBGMISC: Configured NoEncrypt at db level [13:13:53.128]
DBGMISC: No Encrypt backup init(60000000) for cfile [13:13:53.128]
DBGMISC: EXITED krmkbaut [13:13:53.128] elapsed time [00:00:00:00.007]
DBGMISC: EXITED krmkomp [13:13:53.128] elapsed time [00:00:00:00.032]
DBGPLSQL: the compiled command tree is: [13:13:53.129] (krmicomp)
DBGPLSQL: 1 CMD type=Control File and SPFILE Autobackup id=1 status=NOT STARTED
DBGPLSQL: 1 STEP id=1 status=NOT STARTED
DBGPLSQL: 1 TEXTNOD = declare
DBGPLSQL: 2 TEXTNOD = ncopies number;
DBGPLSQL: 3 TEXTNOD = copyno number;
DBGPLSQL: 4 TEXTNOD = handle varchar2(512);
DBGPLSQL: 5 TEXTNOD = comment varchar2(80);
DBGPLSQL: 6 TEXTNOD = media varchar2(80);
DBGPLSQL: 7 TEXTNOD = lcfaudate date;
DBGPLSQL: 8 TEXTNOD = lsequence binary_integer;
DBGPLSQL: 9 TEXTNOD = lbautfmt varchar2(512);
DBGPLSQL: 10 TEXTNOD = rsid number;
DBGPLSQL: 11 TEXTNOD = rsts number;
DBGPLSQL: 12 TEXTNOD = p1 binary_integer := 0;
DBGPLSQL: 13 TEXTNOD = p2 binary_integer;
DBGPLSQL: 14 TEXTNOD = p3 binary_integer;
DBGPLSQL: 15 TEXTNOD = p4 binary_integer;
DBGPLSQL: 16 TEXTNOD = p5 binary_integer;
DBGPLSQL: 17 TEXTNOD = t1 varchar2(1025);
DBGPLSQL: 18 TEXTNOD = t2 varchar2(1);
DBGPLSQL: 19 TEXTNOD = t3 varchar2(1);
DBGPLSQL: 20 TEXTNOD = begin
DBGPLSQL: 21 TEXTNOD =
DBGPLSQL: 22 PRMVAL = lsequence := null;
DBGPLSQL: 23 TEXTNOD =
DBGPLSQL: 24 PRMVAL = lcfaudate := null;
DBGPLSQL: 25 TEXTNOD =
DBGPLSQL: 26 PRMVAL = lbautfmt := null;
DBGPLSQL: 27 TEXTNOD =
DBGPLSQL: 28 PRMVAL = rsid := 5708; rsts := 791817232;
DBGPLSQL: 29 TEXTNOD = setBackupParams(FALSE);
DBGPLSQL: 30 TEXTNOD = if (krmicd.getParams(1, p2, p3, p4, p5, t1, t2, t3)) then
DBGPLSQL: 31 TEXTNOD = p1 := 1;
DBGPLSQL: 32 TEXTNOD = end if;
DBGPLSQL: 33 TEXTNOD = sys.dbms_backup_restore.setRmanStatusRowId(rsid=>rsid, rsts=>rsts);
DBGPLSQL: 34 TEXTNOD = sys.dbms_backup_restore.DoAutobackup(ncopies => ncopies,
DBGPLSQL: 35 TEXTNOD = cfaudate => lcfaudate,
DBGPLSQL: 36 TEXTNOD = seq => lsequence,
DBGPLSQL: 37 TEXTNOD = format => lbautfmt,
DBGPLSQL: 38 TEXTNOD = p1 => p1,
DBGPLSQL: 39 TEXTNOD = p2 => p2,
DBGPLSQL: 40 TEXTNOD = p3 => p3,
DBGPLSQL: 41 TEXTNOD = p4 => t1);
DBGPLSQL: 42 TEXTNOD = copyno := 0;
DBGPLSQL: 43 TEXTNOD = loop
DBGPLSQL: 44 TEXTNOD = exit when copyno=ncopies;
DBGPLSQL: 45 TEXTNOD = sys.dbms_backup_restore.backupPieceCrtDupGet(copyno,
DBGPLSQL: 46 TEXTNOD = handle,
DBGPLSQL: 47 TEXTNOD = comment,
DBGPLSQL: 48 TEXTNOD = media);
DBGPLSQL: 49 TEXTNOD = if comment is null then comment := ‘NONE’; end if;
DBGPLSQL: 50 TEXTNOD = krmicd.writeMsg(8503, handle, comment);
DBGPLSQL: 51 TEXTNOD = copyno := copyno + 1;
DBGPLSQL: 52 TEXTNOD = end loop;
DBGPLSQL: 53 TEXTNOD = sys.dbms_backup_restore.setRmanStatusRowId(rsid=>0, rsts=>0);
DBGPLSQL: 54 TEXTNOD = end;
DBGMISC: executing command Control File and SPFILE Autobackup [13:13:53.138]
DBGRPC: krmxpoq: xc=6917529027645961080, action=”0000003 FINISHED66″, col_l=18, ind=0, sid=1475
DBGRPC: krmxr: xc=6917529027645961080 chid=default rpc count=3
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000021 FINISHED129″, col_l=19, ind=0, sid=1870
DBGRPC: krmxr: xc=6917529027645974080 chid=ORA_DISK_1 rpc count=21
DBGRPC: ENTERED krmqgns
DBGRPC: krmqgns: looking for work for channel default (krmqgns)
DBGRPC: krmqgns: commands remaining to be executed: (krmqgns)
DBGRPC: CMD type=Control File and SPFILE Autobackup id=1 status=NOT STARTED
DBGRPC: 1 STEP id=1 status=NOT STARTED
DBGRPC: krmqgns: no work found for channel default (krmqgns)
DBGRPC: (krmqgns)
DBGRPC: EXITED krmqgns with status 1
DBGRPC: ENTERED krmqgns
DBGRPC: krmqgns: looking for work for channel ORA_DISK_1 (krmqgns)
DBGRPC: krmqgns: commands remaining to be executed: (krmqgns)
DBGRPC: CMD type=Control File and SPFILE Autobackup id=1 status=NOT STARTED
DBGRPC: 1 STEP id=1 status=NOT STARTED
DBGRPC: krmqgns: channel ORA_DISK_1 assigned step 1 (krmqgns)
DBGRPC: (krmqgns)
DBGRPC: EXITED krmqgns with status 0
DBGRPC: krmxcic: xc=6917529027645974080 chid=ORA_DISK_1 calling peidcs
DBGRPC: krmxcis: xc=6917529027645974080 chid=ORA_DISK_1 calling pcicmp
DBGRPC: krmxr: xc=6917529027645974080 chid=ORA_DISK_1 calling peicnt
DBGMISC: ENTERED krmzgparms [13:13:53.152]
DBGMISC: Step id = 1; Code = 1 [13:13:53.152] (krmzgparms)
DBGMISC: EXITED krmzgparms with status 0 (FALSE) [13:13:53.152] elapsed time [00:00:00:00.000]
DBGMISC: ENTERED krmzgparms [13:13:53.152]
DBGMISC: Step id = 1; Code = 1 [13:13:53.152] (krmzgparms)
DBGMISC: EXITED krmzgparms with status 0 (FALSE) [13:13:53.152] elapsed time [00:00:00:00.000]
DBGRPC: krmxrpc: xc=6917529027645974080 kpurpc2 rc=0 db=target proc=DBMS_BACKUP_RESTORE.SETRMANSTATUSROWID
DBGRPC: krmxrpc: xc=6917529027645974080 chid=ORA_DISK_1 rpc count(non-blocking)=22
DBGRPC: krmxrpc: xc=6917529027645974080 RPC #22 completed immediately
DBGRPC: krmxrpc: xc=6917529027645974080 kpurpc2 rc=3123 db=target proc=DBMS_BACKUP_RESTORE.DOAUTOBACKUP
DBGRPC: krmxrpc: xc=6917529027645974080 chid=ORA_DISK_1 rpc count=23
DBGRPC: krmxrpc: xc=6917529027645974080 starting longrunning RPC #23 to target: DBMS_BACKUP_RESTORE.DOAUTOBACKUP
DBGRPC: krmxr: xc=6917529027645974080 started long running rpc
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000022 FINISHED129″, col_l=19, ind=0, sid=1870
DBGRPC: krmxr: callback returned TRUE, skipping sleep
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 1 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 2 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 4 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 8 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 10 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 10 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 10 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 10 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 10 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 10 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 10 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 10 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 10 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 10 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 10 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111″, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 10 seconds
DBGRPC: krmxpoq: xc=6917529027645974080, action=”0000023 STARTED111”, col_l=18, ind=0, sid=1870
DBGRPC: krmxr: sleeping for 10 seconds

对RMAN相关进程做stack call的追踪

select action from gv$session where sid=1782 and serial#=62996 and inst_id=2

rman.log

-rw-r—– 1 oracle oinstall 56475 Aug 20 13:13 mac_ora_29364.trc
-rw-r—– 1 oracle oinstall 439863 Aug 20 13:13 mac_ora_28147.trc

mac> select spid from v$process where addr in ( select paddr from v$session where module like ‘rman%’);

SPID
————
27957
26914

mac>
mac>
mac> oradebug setospid 27957;
Oracle pid: 275, Unix process pid: 27957, image: oracle@mfs138 (TNS V1-V3)
mac> oradebug short_stack;
ksdxfstk()+48<-ksdxcb()+5776<-sspuser()+640<-<-_read_sys()+48<-_read()+224<-sntpread()+80<-ntprd()+192<-nsprecv()+736<-nsrdr()+272<-nsdo()+12512<-nsbrecv()+80<-nioqrc()+704<-opikndf2()+1216<-opitsk()+1280<-opiino()+1840<-opiodr()+2128<-opidrv()+1088<-sou2o()+336<-opimai_real()+224<-main()+368 oradebug short_stack;
ksdxfstk()+48<-ksdxcb()+5776<-sspuser()+640<-<-_read_sys()+48<-_read()+224<-sntpread()+80<-ntprd()+192<-nsprecv()+736<-nsrdr()+272<-nsdo()+12512<-nsbrecv()+80<-nioqrc()+704<-opikndf2()+1216<-opitsk()+1280<-opiino()+1840<-opiodr()+2128<-opidrv()+1088<-sou2o()+336<-opimai_real()+224<-main()+368 oradebug short_stack;
ksdxfstk()+48<-ksdxcb()+5776<-sspuser()+640<-<-_read_sys()+48<-_read()+224<-sntpread()+80<-ntprd()+192<-nsprecv()+736<-nsrdr()+272<-nsdo()+12512<-nsbrecv()+80<-nioqrc()+704<-opikndf2()+1216<-opitsk()+1280<-opiino()+1840<-opiodr()+2128<-opidrv()+1088<-sou2o()+336<-opimai_real()+224<-main()+368
mac> oradebug setospid 26914
Oracle pid: 463, Unix process pid: 26914, image: oracle@mfs138 (TNS V1-V3)
mac> oradebug short_stack
ksdxfstk()+48<-ksdxcb()+5776<-sspuser()+640<-<-_read_sys()+48<-_read()+224<-sntpread()+80<-ntprd()+192<-nsprecv()+736<-nsrdr()+272<-nsdo()+12512<-nsbrecv()+80<-nioqrc()+704<-opikndf2()+1216<-opitsk()+1280<-opiino()+1840<-opiodr()+2128<-opidrv()+1088<-sou2o()+336<-opimai_real()+224<-main()+368 oradebug short_stack
ksdxfstk()+48<-ksdxcb()+5776<-sspuser()+640<-<-_read_sys()+48<-_read()+224<-sntpread()+80<-ntprd()+192<-nsprecv()+736<-nsrdr()+272<-nsdo()+12512<-nsbrecv()+80<-nioqrc()+704<-opikndf2()+1216<-opitsk()+1280<-opiino()+1840<-opiodr()+2128<-opidrv()+1088<-sou2o()+336<-opimai_real()+224<-main()+368 oradebug short_stack
ksdxfstk()+48<-ksdxcb()+5776<-sspuser()+640<-<-_read_sys()+48<-_read()+224<-sntpread()+80<-ntprd()+192<-nsprecv()+736<-nsrdr()+272<-nsdo()+12512<-nsbrecv()+80<-nioqrc()+704<-opikndf2()+1216<-opitsk()+1280<-opiino()+1840<-opiodr()+2128<-opidrv()+1088<-sou2o()+336<-opimai_real()+224<-main()+368<-main_opd_entry()+80

oradebug dump errorstack 3;
Statement processed.

oradebug tracefile_name
/apps/oracle/admin/mac/udump/mac_ora_26914.trc

pstack 27957
27957: /apps/oracle/product/10.2.0/bin/oracle

——————————– lwpid : 5899967 ——————————-

0: c000000000405690 : _read_sys() + 0x30 (/usr/lib/hpux64/libc.so.1)
1: c000000000419f00 : _read() + 0xe0 (/usr/lib/hpux64/libc.so.1)
2: 40000000020d7ad0 : sntpread() + 0x50 (/apps/oracle/product/10.2.0/bin/oracle)
3: 40000000020d7a20 : ntprd() + 0xc0 (/apps/oracle/product/10.2.0/bin/oracle)
4: 400000000aa3b7c0 : nsprecv() + 0x2e0 (/apps/oracle/product/10.2.0/bin/oracle)
5: 4000000002307bb0 : nsrdr() + 0x110 (/apps/oracle/product/10.2.0/bin/oracle)
6: 4000000002300180 : nsdo() + 0x30e0 (/apps/oracle/product/10.2.0/bin/oracle)
7: 40000000022fcfa0 : nsbrecv() + 0x50 (/apps/oracle/product/10.2.0/bin/oracle)
8: 4000000002aff570 : nioqrc() + 0x2c0 (/apps/oracle/product/10.2.0/bin/oracle)
9: 4000000002bd3c80 : opikndf2() + 0x4c0 (/apps/oracle/product/10.2.0/bin/oracle)
10: 4000000002d9f3f0 : opitsk() + 0x500 (/apps/oracle/product/10.2.0/bin/oracle)
11: 400000000268c3f0 : opiino() + 0x730 (/apps/oracle/product/10.2.0/bin/oracle)
12: 4000000002cd6ec0 : opiodr() + 0x850 (/apps/oracle/product/10.2.0/bin/oracle)
13: 400000000260fd80 : opidrv() + 0x440 (/apps/oracle/product/10.2.0/bin/oracle)
14: 400000000260f390 : sou2o() + 0x150 (/apps/oracle/product/10.2.0/bin/oracle)
15: 40000000025fbfc0 : opimai_real() + 0xe0 (/apps/oracle/product/10.2.0/bin/oracle)
16: 400000000250c880 : main() + 0x170 (/apps/oracle/product/10.2.0/bin/oracle)
17: c000000000032f90 : main_opd_entry() + 0x50 (/usr/lib/hpux64/dld.so)

pstack 26914
26914: /apps/oracle/product/10.2.0/bin/oracle

——————————– lwpid : 5898924 ——————————-

0: c000000000405690 : _read_sys() + 0x30 (/usr/lib/hpux64/libc.so.1)
1: c000000000419f00 : _read() + 0xe0 (/usr/lib/hpux64/libc.so.1)
2: 40000000020d7ad0 : sntpread() + 0x50 (/apps/oracle/product/10.2.0/bin/oracle)
3: 40000000020d7a20 : ntprd() + 0xc0 (/apps/oracle/product/10.2.0/bin/oracle)
4: 400000000aa3b7c0 : nsprecv() + 0x2e0 (/apps/oracle/product/10.2.0/bin/oracle)
5: 4000000002307bb0 : nsrdr() + 0x110 (/apps/oracle/product/10.2.0/bin/oracle)
6: 4000000002300180 : nsdo() + 0x30e0 (/apps/oracle/product/10.2.0/bin/oracle)
7: 40000000022fcfa0 : nsbrecv() + 0x50 (/apps/oracle/product/10.2.0/bin/oracle)
8: 4000000002aff570 : nioqrc() + 0x2c0 (/apps/oracle/product/10.2.0/bin/oracle)
9: 4000000002bd3c80 : opikndf2() + 0x4c0 (/apps/oracle/product/10.2.0/bin/oracle)
10: 4000000002d9f3f0 : opitsk() + 0x500 (/apps/oracle/product/10.2.0/bin/oracle)
11: 400000000268c3f0 : opiino() + 0x730 (/apps/oracle/product/10.2.0/bin/oracle)
12: 4000000002cd6ec0 : opiodr() + 0x850 (/apps/oracle/product/10.2.0/bin/oracle)
13: 400000000260fd80 : opidrv() + 0x440 (/apps/oracle/product/10.2.0/bin/oracle)
14: 400000000260f390 : sou2o() + 0x150 (/apps/oracle/product/10.2.0/bin/oracle)
15: 40000000025fbfc0 : opimai_real() + 0xe0 (/apps/oracle/product/10.2.0/bin/oracle)
16: 400000000250c880 : main() + 0x170 (/apps/oracle/product/10.2.0/bin/oracle)
17: c000000000032f90 : main_opd_entry() + 0x50 (/usr/lib/hpux64/dld.so)

uname -a
HP-UX B.11.31 U ia64 unlimited-user license

Workaround:

没有找到直接解决该问题的方法, (从HP-UX上迁移走似乎是一种不可行的方案),可以通过configure autobackup off绕过该问题,在RMAN备份脚本中加入backup current controlfile可以起来类似的作用。

使用logminer日志挖掘技术

使用logminer日志挖掘技术

 

如果自己搞不定可以找诗檀软件专业ORACLE数据库修复团队成员帮您恢复!

诗檀软件专业数据库修复团队

服务热线 : 13764045638   QQ号:47079569    邮箱:service@parnassusdata.com

[gview file=”https://www.askmaclean.com/wp-content/uploads/2014/11/less11_Flashback.pdf”]

使用Oracle闪回flashback技术

使用Oracle闪回flashback技术

 

如果自己搞不定可以找诗檀软件专业ORACLE数据库修复团队成员帮您恢复!

诗檀软件专业数据库修复团队

服务热线 : 13764045638   QQ号:47079569    邮箱:service@parnassusdata.com

 

 

[gview file=”https://www.askmaclean.com/wp-content/uploads/2014/11/less11_Flashback.pdf”]

手动递增SCN号的几种方法:How to increase System Change Number by manual

 

 

手动递增SCN号的几种方法

 

除去下面几种,还有一种方法直接修改 实例的Global Lamport SCN,在SGA中由kcsgscn变量存储,对于一个实例来说这是唯一的源SCN,所有其他的SCN均由这个source scn所驱动。 这种递增方式是直接用oradebug 修改该Global Lamport SCN kcsgscn

 

 

如果自己搞不定可以找诗檀软件专业ORACLE数据库修复团队成员帮您恢复!

诗檀软件专业数据库修复团队

服务热线 : 13764045638   QQ号:47079569    邮箱:service@parnassusdata.com

 

 

SQL> oradebug setmypid
Statement processed.

 

SQL> select to_char(current_scn,’XXXXXXXXXXXX’) from v$database;

TO_CHAR(CURRE
————-
D3E1E

SQL>
SQL> oradebug DUMPvar SGA kcsgscn
kcslf kcsgscn_ [060012658, 060012688) = 000D3E1E 00000000 00000000 00000000 0000162D 00000000 00000000 00000000 00000000 00000000 60012338 00000000

 

ORADEBUG POKE 0x060012658 4 0xfffff

poke 命令的语法

<address> <length> <value>” allows you to modify a given region of memory (length of memory is limited to size of scalar C types)

 

SQL> select current_scn from v$database;

CURRENT_SCN
———–
1048583

 

SQL> select to_char(current_scn,’XXXXXXXXXXXX’) from v$database;

TO_CHAR(CURRE
————-
10000B

SQL> oradebug DUMPvar SGA kcsgscn
kcslf kcsgscn_ [060012658, 060012688) = 00100010 00000000 00000000 00000000 0000004E 00000000 00000000 00000000 00000000 00000000 60012338 00000000

 

 

 

 

 

How to jump SCN  by manual ,  this could be a problem:

1. We can bump up the SCN by using the procedure from Note: 386830.1

 

Bump the system SCN on the primary database to fix any metadata index corruptions for both
primary and physical standby databases. Set the following parameters in the init.ora and restart
the database in restricted mode to bump the system SCN of the primary database.
For Real Application Clusters, perform the steps on only one node of the cluster. 
Init.ora syntax:
*._allow_error_simulation = TRUE
*._smu_debug_mode = 268435456
If using an spfile, Oracle recommends creating a temporary init.ora using the CREATE PFILE SQL command.
SQL> create pfile='/tmp/initTMP.ora' from spfile='';
Then add the parameters to this temporary file.
To use the temporary init.ora file when starting the instance, include the 'PFILE' clause with the STARTUP SQL command i.e.
SQL> startup restrict pfile=
WARNING: These parameters should only be used for this fix and must be removed immediately afterwards in step 2.
If the above parameters are used through multiple database restarts, a complete database rebuild will be required.
To know the system SCN has been bumped, monitor the instance's alert.log for the following message:
advance SCN to wrap base xxxx
Where xxx represents the new wrap SCN.

 

 

 

 

2.EVENT: ADJUST_SCN – Quick Reference (Doc ID 30681.1)

 

 

 

WORKAROUND:
-----------
Searched in webiv (ora-1555, ora-604 see note:1063408.6 ) suggests workaround
is to adjust serial number using event
Will try : (30681.1)
alter session set events 'IMMEDIATE trace name ADJUST_SCN level 1';
ora-600 [2256][0][1073741824][1][293672646]
ERROR:
ORA-600 [2256][a][b][c][d][e]
VERSIONS:
versions 7.3.X, 8.0.X, 8.1.X
DESCRIPTION:
This exception indicates that you attempted to ADJUST_SCN but the level
supplied would be less that the current SCN.
ARGUMENTS:
a.  Requested SCN WRAP
b.  Requested SCN BASE
c.  Current SCN WRAP
d.  Current SCN BASE = [293672646]
*4 = 1174690584
will try level 2
ora-600 2256[0][2147483648][1][293672646]
level 3
ora-600 2256[0][3221225472][1][293672646]
Now, we increase the SCN on DST7 by using the ADJUST_SCN event -
(Note: do not use this event outside of your test environment)
set the following hidden parameter in init.ora on DST7 database and bounce
the database.
_allow_error_simulation=true
Now, the SCN is increased by doing -
alter system set events 'immediate trace name adjust_scn level ';
where  translates to (n*0x40000000) as the target SCN value. ie. n = 1
will set the SCN as 0x0000.40000000. For the testcase pick a value that is
larger than the current SCN on both databases by atleast 100000 SCNs (our
earlier SCN adjust threshold value).
alter system set events 'immediate trace name adjust_scn level 2';
select to_char(current_scn, 'xxxxxxxxxxxx') from v$database;

 

 

 

3.Note 552438.1 How To Adjust the SCN using parameter _MINIMUM_GIGA_SCN

 

 

 

 

Parameter: MINIMUM_GIGA_SCN
~~~~~~~~~~~~~~~~~~~~~~~~~~~
@Oracle8i:	HIDDEN
@Identifier:	kcmmsn
@Versions:	See <IVERS.MINIMUM_GIGA_SCN>
Values:
Related:
Description:	Minimum SCN to start with in 2^30 units
~~~~~~~~~~~~
Articles:	
<Event:ADJUST_SCN>
Overview of Init.Ora Parameter Reference notes

终极方案:
使用bbed修改datafile header实现修改scn

主要是修改system01.dbf datafile header kcvfh.kcvfhckp.kcvcpscn

helpbkup_us@oracle.com
helpkern_us@oracle.com
SQL> select * from v$version;
BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bi
PL/SQL Release 10.2.0.1.0 - Production
CORE    10.2.0.1.0      Production
TNS for Linux: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 - Production
1* select name,CHECKPOINT_CHANGE# from v$datafile
SQL> /
/s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_system_9f2flf09_.dbf                  3905523
/s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_undotbs1_9f2flf52_.dbf                3905523
/s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_sysaux_9f2flf2v_.dbf                  3905523
/s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_users_9f2flf5g_.dbf                   3905523
/s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_example_9f2fmfto_.dbf                 3905523
/s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_youchuan_9f2fqwr4_.dbf                3905523
/s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_guobao_9f2fwkkt_.dbf                  3905523
/s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_tsoa_d_bjx6j57v_.dbf                  3905523
SQL>  select to_char(3905523,'XXXXXXXXXXXXX') from dual;
TO_CHAR(390552
--------------
3B97F3
SQL> oradebug tracefile_name
/s01/oracle/product/10.2.0/db_1/admin/MACLEAN1/udump/maclean1_ora_9468.trc
[oracle@vrh8 ~]$ cp /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_system_9f2flf09_.dbf  /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_system_9f2flf09_.dbf.bak
[oracle@vrh8 ~]$ bbed filename=/s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_system_9f2flf09_.dbf password=blockedit
BBED: Release 2.0.0.0.0 - Limited Production on Tue Mar 24 16:50:39 2015
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
************* !!! For Oracle Internal Use only !!! ***************
BBED> set mode edit
MODE            Edit
BBED>  set blocksize 8192
BLOCKSIZE       8192
BBED> set block 1
BLOCK#          1
BBED> map
File: /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_system (0)
Block: 1                                     Dba:0x00000000
------------------------------------------------------------
Data File Header
struct kcvfh, 676 bytes                    @0       
ub4 tailchk                                @8188    
BBED> p kcvfh
struct kcvfh, 676 bytes                     @0       
struct kcvfhbfh, 20 bytes                @0       
ub1 type_kcbh                         @0        0x0b
ub1 frmt_kcbh                         @1        0xa2
ub1 spare1_kcbh                       @2        0x00
ub1 spare2_kcbh                       @3        0x00
ub4 rdba_kcbh                         @4        0x00400001
ub4 bas_kcbh                          @8        0x00000000
ub2 wrp_kcbh                          @12       0x0000
ub1 seq_kcbh                          @14       0x01
ub1 flg_kcbh                          @15       0x04 (KCBHFCKV)
ub2 chkval_kcbh                       @16       0x4fc6
ub2 spare3_kcbh                       @18       0x0000
struct kcvfhhdr, 76 bytes                @20      
ub4 kccfhswv                          @20       0x00000000
ub4 kccfhcvn                          @24       0x0a200100
ub4 kccfhdbi                          @28       0x157f2927
text kccfhdbn[0]                      @32      M
text kccfhdbn[1]                      @33      A
text kccfhdbn[2]                      @34      C
text kccfhdbn[3]                      @35      L
text kccfhdbn[4]                      @36      E
text kccfhdbn[5]                      @37      A
text kccfhdbn[6]                      @38      N
text kccfhdbn[7]                      @39      1
ub4 kccfhcsq                          @40       0x00000675
ub4 kccfhfsz                          @44       0x00010900
s_blkz kccfhbsz                       @48       0x00
ub2 kccfhfno                          @52       0x0001
ub2 kccfhtyp                          @54       0x0003
ub4 kccfhacid                         @56       0x00000000
ub4 kccfhcks                          @60       0x00000000
text kccfhtag[0]                      @64       
text kccfhtag[1]                      @65       
text kccfhtag[2]                      @66       
text kccfhtag[3]                      @67       
text kccfhtag[4]                      @68       
text kccfhtag[5]                      @69       
text kccfhtag[6]                      @70       
text kccfhtag[7]                      @71       
text kccfhtag[8]                      @72       
text kccfhtag[9]                      @73       
text kccfhtag[10]                     @74       
text kccfhtag[11]                     @75       
text kccfhtag[12]                     @76       
text kccfhtag[13]                     @77       
text kccfhtag[14]                     @78       
text kccfhtag[15]                     @79       
text kccfhtag[16]                     @80       
text kccfhtag[17]                     @81       
text kccfhtag[18]                     @82       
text kccfhtag[19]                     @83       
text kccfhtag[20]                     @84       
text kccfhtag[21]                     @85       
text kccfhtag[22]                     @86       
text kccfhtag[23]                     @87       
text kccfhtag[24]                     @88       
text kccfhtag[25]                     @89       
text kccfhtag[26]                     @90       
text kccfhtag[27]                     @91       
text kccfhtag[28]                     @92       
text kccfhtag[29]                     @93       
text kccfhtag[30]                     @94       
text kccfhtag[31]                     @95       
ub4 kcvfhrdb                             @96       0x00400179
struct kcvfhcrs, 8 bytes                 @100     
ub4 kscnbas                           @100      0x00000008
ub2 kscnwrp                           @104      0x0000
ub4 kcvfhcrt                             @108      0x221e01bf
ub4 kcvfhrlc                             @112      0x31dd7868
struct kcvfhrls, 8 bytes                 @116     
ub4 kscnbas                           @116      0x00080634
ub2 kscnwrp                           @120      0x0000
ub4 kcvfhbti                             @124      0x00000000
struct kcvfhbsc, 8 bytes                 @128     
ub4 kscnbas                           @128      0x00000000
ub2 kscnwrp                           @132      0x0000
ub2 kcvfhbth                             @136      0x0000
ub2 kcvfhsta                             @138      0x2000 (NONE)
struct kcvfhckp, 36 bytes                @484     
struct kcvcpscn, 8 bytes              @484     
ub4 kscnbas                        @484      0x003b97f3
ub2 kscnwrp                        @488      0x0000
ub4 kcvcptim                          @492      0x342a9561
ub2 kcvcpthr                          @496      0x0001
union u, 12 bytes                     @500     
struct kcvcprba, 12 bytes          @500     
ub4 kcrbaseq                    @500      0x00000052
ub4 kcrbabno                    @504      0x00013fc0
ub2 kcrbabof                    @508      0x0010
ub1 kcvcpetb[0]                       @512      0x02
ub1 kcvcpetb[1]                       @513      0x00
ub1 kcvcpetb[2]                       @514      0x00
ub1 kcvcpetb[3]                       @515      0x00
ub1 kcvcpetb[4]                       @516      0x00
ub1 kcvcpetb[5]                       @517      0x00
ub1 kcvcpetb[6]                       @518      0x00
ub1 kcvcpetb[7]                       @519      0x00
ub4 kcvfhcpc                             @140      0x00000087
ub4 kcvfhrts                             @144      0x3427f1a8
ub4 kcvfhccc                             @148      0x00000086
struct kcvfhbcp, 36 bytes                @152     
struct kcvcpscn, 8 bytes              @152     
ub4 kscnbas                        @152      0x00000000
ub2 kscnwrp                        @156      0x0000
ub4 kcvcptim                          @160      0x00000000
ub2 kcvcpthr                          @164      0x0000
union u, 12 bytes                     @168     
struct kcvcprba, 12 bytes          @168     
ub4 kcrbaseq                    @168      0x00000000
ub4 kcrbabno                    @172      0x00000000
ub2 kcrbabof                    @176      0x0000
ub1 kcvcpetb[0]                       @180      0x00
ub1 kcvcpetb[1]                       @181      0x00
ub1 kcvcpetb[2]                       @182      0x00
ub1 kcvcpetb[3]                       @183      0x00
ub1 kcvcpetb[4]                       @184      0x00
ub1 kcvcpetb[5]                       @185      0x00
ub1 kcvcpetb[6]                       @186      0x00
ub1 kcvcpetb[7]                       @187      0x00
ub4 kcvfhbhz                             @312      0x00000000
struct kcvfhxcd, 16 bytes                @316     
ub4 space_kcvmxcd[0]                  @316      0x00000000
ub4 space_kcvmxcd[1]                  @320      0x00000000
ub4 space_kcvmxcd[2]                  @324      0x00000000
ub4 space_kcvmxcd[3]                  @328      0x00000000
word kcvfhtsn                            @332      0
ub2 kcvfhtln                             @336      0x0006
text kcvfhtnm[0]                         @338     S
text kcvfhtnm[1]                         @339     Y
text kcvfhtnm[2]                         @340     S
text kcvfhtnm[3]                         @341     T
text kcvfhtnm[4]                         @342     E
text kcvfhtnm[5]                         @343     M
text kcvfhtnm[6]                         @344      
text kcvfhtnm[7]                         @345      
text kcvfhtnm[8]                         @346      
text kcvfhtnm[9]                         @347      
text kcvfhtnm[10]                        @348      
text kcvfhtnm[11]                        @349      
text kcvfhtnm[12]                        @350      
text kcvfhtnm[13]                        @351      
text kcvfhtnm[14]                        @352      
text kcvfhtnm[15]                        @353      
text kcvfhtnm[16]                        @354      
text kcvfhtnm[17]                        @355      
text kcvfhtnm[18]                        @356      
text kcvfhtnm[19]                        @357      
text kcvfhtnm[20]                        @358      
text kcvfhtnm[21]                        @359      
text kcvfhtnm[22]                        @360      
text kcvfhtnm[23]                        @361      
text kcvfhtnm[24]                        @362      
text kcvfhtnm[25]                        @363      
text kcvfhtnm[26]                        @364      
text kcvfhtnm[27]                        @365      
text kcvfhtnm[28]                        @366      
text kcvfhtnm[29]                        @367      
ub4 kcvfhrfn                             @368      0x00000001
struct kcvfhrfs, 8 bytes                 @372     
ub4 kscnbas                           @372      0x00000000
ub2 kscnwrp                           @376      0x0000
ub4 kcvfhrft                             @380      0x00000000
struct kcvfhafs, 8 bytes                 @384     
ub4 kscnbas                           @384      0x00000000
ub2 kscnwrp                           @388      0x0000
ub4 kcvfhbbc                             @392      0x00000000
ub4 kcvfhncb                             @396      0x00000000
ub4 kcvfhmcb                             @400      0x00000000
ub4 kcvfhlcb                             @404      0x00000000
ub4 kcvfhbcs                             @408      0x00000000
ub2 kcvfhofb                             @412      0x000a
ub2 kcvfhnfb                             @414      0x000a
ub4 kcvfhprc                             @416      0x221e01a8
struct kcvfhprs, 8 bytes                 @420     
ub4 kscnbas                           @420      0x00000001
ub2 kscnwrp                           @424      0x0000
struct kcvfhprfs, 8 bytes                @428     
ub4 kscnbas                           @428      0x00000000
ub2 kscnwrp                           @432      0x0000
ub4 kcvfhtrt                             @444      0x00000000
BBED> p kcvfhckp
struct kcvfhckp, 36 bytes                   @484     
struct kcvcpscn, 8 bytes                 @484     
ub4 kscnbas                           @484      0x003b97f3
ub2 kscnwrp                           @488      0x0000
ub4 kcvcptim                             @492      0x342a9561
ub2 kcvcpthr                             @496      0x0001
union u, 12 bytes                        @500     
struct kcvcprba, 12 bytes             @500     
ub4 kcrbaseq                       @500      0x00000052
ub4 kcrbabno                       @504      0x00013fc0
ub2 kcrbabof                       @508      0x0010
ub1 kcvcpetb[0]                          @512      0x02
ub1 kcvcpetb[1]                          @513      0x00
ub1 kcvcpetb[2]                          @514      0x00
ub1 kcvcpetb[3]                          @515      0x00
ub1 kcvcpetb[4]                          @516      0x00
ub1 kcvcpetb[5]                          @517      0x00
ub1 kcvcpetb[6]                          @518      0x00
ub1 kcvcpetb[7]                          @519      0x00
BBED> set offset 484
OFFSET          484
BBED> modify /x 0xF397
File: /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_system (0)
Block: 1                Offsets:  484 to  995           Dba:0x00000000
------------------------------------------------------------------------
f397f4f3 00000000 61952a34 01000000 52000000 c03f0100 1000a865 02000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
0d000d00 0d000100 00000000 00000000 00000000 02004000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
<32 bytes per line>
BBED> set offset 486
OFFSET          486
BBED> modify /x 0x4B00
File: /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_system (0)
Block: 1                Offsets:  486 to  997           Dba:0x00000000
------------------------------------------------------------------------
4b000000 00006195 2a340100 00005200 0000c03f 01001000 a8650200 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000d00 
0d000d00 01000000 00000000 00000000 00000200 40000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
<32 bytes per line>
BBED> p kcvfhckp
struct kcvfhckp, 36 bytes                   @484     
struct kcvcpscn, 8 bytes                 @484     
ub4 kscnbas                           @484      0x004b97f3
ub2 kscnwrp                           @488      0x0000
ub4 kcvcptim                             @492      0x342a9561
ub2 kcvcpthr                             @496      0x0001
union u, 12 bytes                        @500     
struct kcvcprba, 12 bytes             @500     
ub4 kcrbaseq                       @500      0x00000052
ub4 kcrbabno                       @504      0x00013fc0
ub2 kcrbabof                       @508      0x0010
ub1 kcvcpetb[0]                          @512      0x02
ub1 kcvcpetb[1]                          @513      0x00
ub1 kcvcpetb[2]                          @514      0x00
ub1 kcvcpetb[3]                          @515      0x00
ub1 kcvcpetb[4]                          @516      0x00
ub1 kcvcpetb[5]                          @517      0x00
ub1 kcvcpetb[6]                          @518      0x00
ub1 kcvcpetb[7]                          @519      0x00
BBED> sum
Check value for File 0, Block 1:
current = 0x4fc6, required = 0x4fb6
BBED> sum apply
Check value for File 0, Block 1:
current = 0x4fb6, required = 0x4fb6
BBED> verify
DBVERIFY - Verification starting
FILE = /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_system_9f2flf09_.dbf
BLOCK = 1
DBVERIFY - Verification complete
Total Blocks Examined         : 1
Total Blocks Processed (Data) : 0
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
SQL> select file#,name,checkpoint_change# from v$datafile;
1 /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_system_9f2flf09_.dbf                  3905523
2 /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_undotbs1_9f2flf52_.dbf                3905523
3 /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_sysaux_9f2flf2v_.dbf                  3905523
4 /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_users_9f2flf5g_.dbf                   3905523
5 /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_example_9f2fmfto_.dbf                 3905523
6 /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_youchuan_9f2fqwr4_.dbf                3905523
7 /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_guobao_9f2fwkkt_.dbf                  3905523
8 /s01/oracle/product/10.2.0/db_1/oradata/MACLEAN1/datafile/o1_mf_tsoa_d_bjx6j57v_.dbf                  3905523
8 rows selected.
SQL> select file#,FUZZY,CHECKPOINT_CHANGE#        from v$datafile_header;
1 NO             4954099
2 NO             3905523
3 NO             3905523
4 NO             3905523
5 NO             3905523
6 NO             3905523
7 NO             3905523
8 NO             3905523
8 rows selected.
SQL> alter database open;
Database altered.
SQL> select file#,FUZZY,CHECKPOINT_CHANGE#        from v$datafile_header;
1 YES            4954100
2 YES            4954100
3 YES            4954100
4 YES            4954100
5 YES            4954100
6 YES            4954100
7 YES            4954100
8 YES            4954100
8 rows selected.
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup;
ORACLE instance started.
Total System Global Area  591396864 bytes
Fixed Size                  2022536 bytes
Variable Size             180355960 bytes
Database Buffers          402653184 bytes
Redo Buffers                6365184 bytes
Database mounted.
Database opened.

Oracle 8i 备份/恢复(Recovery Manager)

本文永久地址:https://www.askmaclean.com/archives/oracle-8i-备份恢复(recovery-manager).html

说明

Recovery Manager(RMAN)是从Oracle8 R8.0之后追加的管理oracle数据库备份/储存/恢复的工具。在此,我将解说oracle8i的RMAN中扩展的新功能。

 

  • RMAN基本功能的扩展
  • 媒介管理软件相关功能的扩展
  • OPS相关功能的扩展

在此对于RMAN命令的详细语法方面的内容,请参考《Oracle8i备份、恢复指南》

 

RMAN基本功能的扩展

1-1.复制备份set

考虑到Oracle的备份时,类似于归档日志以及控制文件的案例,某个备份受损可能造成毁灭性的故障。Oracle8i的RMAN中,可以一次制成多个类似的的备份。因此,在存储时某个备份无法使用的话,用于其他备份就OK了。

为了制成多个备份,需要通过RMAN命令的SET DUPLEX命令来指定复制数,执行备份。SET DUPLEX命令如下所示。

 

SET DUPLEX = {ON|OFF|1|2|3|4};

 

关键词ON意味着指定为2 。DUPLEX最多可以制成4个拷贝。以下是使用SET DUPLEX命令的范例。

RUN {

SET DUPLEX =2;

ALLOCATE CHANNEL ch1 TYPE ‘sbt_tape’;

BACKUP

FORMAT ‘arc_%s_%p_%c’ FILESPERSET 1

ARCHIVELOG ALL DELETE INPUT;

RELEASE CHANNEL ch1;

}

SET DUPLEX命令需要比分配channel(ALLOCATE CHANNEL命令)更早执行。另外,执行SET DUPLEX命令后,在分配到的所有channel中执行备份复制。
在制成tape装置的备份是,需要指定初始化参数BACKUP_TAPE_IO_SLAVES = TRUE。
执行复制备份时,用指定备份名的FORMAT句,可以用替换字符串“%”来指定。在上述例子中,可以用替换字符串“%”来指定。在上述例子中,备份set编号为10,备份piece为1时制成 ‘arc_10_1_1’、‘arc_10_1_2’两个备份。

通过RMAN制成克隆数据库

Oracle8i的RMAN中,使用RMAN的备份,在远程主机或本地主机中,制成数据库的拷贝(克隆数据库)。因为克隆数据库与真正的数据库结构完全相同,所以并不会对正式环境的数据库产生影响。可以作为测试专用数据库。

左:正式数据库、克隆数据库、制成克隆数据库 本地主机

右:克隆数据库、远程主机

通过RMAN制成克隆数据库是,以下操作可以使用RMAN完成。

  • 克隆数据库中所使用的数据库文件的还原。在远程主机中制成克隆数据库时,是经过Net8来传送文件,还原文件的。
  • 直到执行命令时,需要执行应用了REDO的不完整恢复
  • 启动指定不完整恢复之后的RESETLOGS选项的数据库
  • 对新数据库重命名
  • 制成控制文件

通过RMAN制成克隆数据库时,伴随着指定了RESETLOGS的数据库的启动,就无法在制成stand by数据库时使用。

 

以下命令是远程主机制成克隆数据库的例子。为了完成这个操作,需要实现在远程主机中进行以下操作。

 

  • 收集远程主机中的ora参数文件拷贝
  • 设定需要连接远程主机的net8
  • 编辑ora、tnsnames.ora
  • 启动listener

 

完成这些设定之后,请使用以下命令来制成克隆数据库。下例是通过Net8连接描述符来指定在本地设备中的克隆数据库,制成远程主机中的克隆数据库。

 

 

# 启动RMAN,这时请指定克隆数据库的连接描述符。
%rman connect target / auxiliary sys/passwd@clonedb catalog rman/rman/@rcat
# 以下是制成克隆数据库的RMAN命令。
RMAN>STARTUP NOMOUNT;
RMAN> RUN{
2>      ALLOCATE AUXILIARY CHANNEL ch1 TYPE DISK;
3>      DUPLICATE TARGET DATABASE TO CLONEDB 
4>         LOGFILE ‘/Oracle/dbs/log1.f’ SIZE 200K,
5>                 ‘/Oracle/dbs/log2.f’ SIZE 200K 
6>         NOFILECHECK;
7>    }

DUPLICATE命令中,需要原数据库与克隆数据库中所使用的数据库文件名不重复。因此,在同样的目录结构中,在远程主机中制成克隆数据库时,需要设置为指定NOFILECHECK选项而不执行这个检查。
具体的克隆数据库制作顺序请参考《执行例与结果》。

通过RMAN来制成克隆数据库的详细内容请参考「Oracle8i备份、恢复指南」。

启动/关闭数据库

Oracle8 R8.0中为了对数据库进行冷备份以及恢复,需要启动、关闭数据库时,需要使用Server Manager执行这些操作。因此通过RMAN来执行修复时,就会组合SVRMGR与RMAN来一起使用。一些脚本由此就会变得复杂。

在Oracle8i的RMAN中,作为RMAN的组合命令支撑数据库的STARTUP、SHUTDOWN、ALTER DATABASE命令。由此就可以仅凭RMAN启动、关闭数据库。

 

追加命令 功能
STARTUP … 启动数据库。可以指定SQL*Plus的STARTUP命令以及同样的选项。
SHUTDOWN … 关闭数据库。可以指定SQL*Plus的SHUTDOWN命令以及相关选项。
ALTER DATABASE … 将数据库从MOUNT状态变更为OPEN状况。

 

在以下RMAN命令中,获得启动中终止的数据库的冷备份。重启数据库。

SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
RUN{
ALLOCATE CHANNEL ch1 TYPE 'SBT_TAPE';
BACKUP
FORMAT 'db_%s_%p'
(DATABASE);
}
ALTER DATABASE OPEN;

 

扩展LIST命令

LIST命令是展示至今为止获得的备份的信息的命令。Oracle8i的RMAN中,LIST命令已经被扩展为执行时会展示各自的备份片的名称、所有的数据文件的名称,控制文件信息等。以下是展示表区域bo1的备份set的一览的例子。

 

 

RMAN> LIST BACKUPSET OF TABLESPACE bol;
RMAN-03022: compiling command: list
List of Backup Sets
Key     Recid      Stamp      LV Set Stamp  Set Count  Completion Time
------- ---------- ---------- -- ---------- ---------- ----------------------
1       1          365975325  0  365975324  2          20-APR-99
List of Backup Pieces
Key     Pc# Cp# Status      Completion Time        Piece Name
------- --- --- ----------- ---------------------- -----------------------
2       1   2   AVAILABLE   20-APR-99              /bkup/bol_02at0los_1_2
1       1   1   AVAILABLE   20-APR-99              /bkup/bol_02at0los_1_1
List of Datafiles Included
File Name                                  LV Type Ckp SCN    Ckp Time
---- ------------------------------------- -- ---- ---------- ------------
4    /oradata/je815/dbs/bolje815.dbf       0  Full 578102     20-APR-99

 

LIST命令中,仅会展示备份的统计数据的平均值。展示deleted、unavailable状态的备份信息时,请使用LIST ALL … 命令。

通过RMAN制成、更新恢复目录

Oracle8 R8.0中,需要使用SQL*Plus来制成、更新、删除、储存RMAN使用的信息的恢复目录的脚本。

Oracle8i中,变更为将这些操作作为RMAN的命令来执行了。这些命令是用RMAN启动时的选项RCVCAT来指定的,对于用户的架构来执行的。

 

追加命令 功能
CREATE CATALOG 制成现在连接的架构中的恢复目录。可以用TABLESPACE语句来指定制成目录的表区域。
UPDATE CATALOG 连接到旧版本的目录时,执行目录的更新。UPDATE CATALOG命令为了确认,连续执行两次,于是就会进行更新。
DROP CATALOG 删除现在连接的架构中的目录。

DROP CATALOG命令连续执行两次时,就会删除目录。

执行对于目录的操作的用户,需要重新赋予RECOVERY_CATALOG_OWNER权限。

另外,Oracle8i中为了制成目录,而附属为SQL的脚本catrman.sql。必须要用到RMAN的CREATE CATALOG命令。

 

以下的命令是执行目录更新的例子。

 

RMAN> UPGRADE CATALOG
RMAN-06435: recovery catalog owner is rman
RMAN-06442: enter UPGRADE CATALOG command again to confirm catalog upgrade
RMAN> UPGRADE CATALOG
RMAN-06408: recovery catalog upgraded to version 08.01.05

 

媒介管理软件相关功能的扩展

2-1. 恢复目录以及媒介管理软件目录的项目检查

使用RMAN中的tape装置,执行备份时,oracle并不需要媒介管理软件。因为一般媒介管理软件中,是通过目录来管理tape的,作为系统来考虑时,就会存在RMAN以及媒介管理软件两个目录。

Oracle8 R8.0的RMAN需要用户管理两个目录的同步。比如重新使用(覆盖)媒介管理软件中的tape时,RMAN的目录中包含这些tape的备份信息就会被留下。因此,用户使用RMAN的目录管理命令,需要从恢复目录中删除这些备份信息。

Oracle8i的RMAN,就会执行恢复目录以及媒介管理软件中的目录同步命令。

 

 

追加命令 功能
ALLOCATE CHANNEL …

FOR MAINTENANCE ;

为了执行目录维护,需要分配将要使用的channel。以下展示的命令在执行前,需要用这个命令来分配channel。
CROSSCHECK … ;

CHANGE… CROSSCHECK ;

这些命令用于确认指定的备份是否存在(可以从RMAN进行访问)。执行命令时,RMAN无法访问备份的话,那个备份就会被标记以及EXPIRED ,被判断为无效。这个备份可以用LIST命令以及REPORT命令来确认。

以下是使用CROSSCHECK命令的脚本例。

 

ALLOCATE CHANNEL FOR MAINTENANCE TYPE ‘sbt_tape’;

CROSSCHECK BACKUPSET OF “/Oracle/dbs/tbs_8.f”;

没有连接恢复目录时,无法使用CROSSCHECK命令。这时,请使用CHANGE … CROSSCHECK命令。CHANGE … CROSSCHECK命令没有连接恢复目录时,也可以通过控制文件中的信息以及每集管理软件的信息来进行交互检测。

EXPIRED以及被标记的备份,可以在执行下次的CHANGE … CROSSCHECK命令时,访问备份时,会被标记为AVAILABLE。类似的功能就不会再用CROSSCHECK命令来执行。这是为了确认CROSSCHECK命令是否将状态变为AVAILABE。

CROSSCHECK命令,CHANGE … CROSSCHECK命令中,需要检测是否可以访问。因此,检测备份破损时,请使用VALIDATE命令。

2-2. 使用了媒介管理软件的固有的功能的数据库备份
(PROXY COPY)

使用Oracle8 R8.0的RMAN获得的备份,对数据文件的I/O,是由oracle的服务器进程来执行。因此媒介管理软件会执行oracle的服务器进程以及数据流的对策。

Oracle8i中媒介管理软件以及接口中,媒介管理软件自身对于数据库的文件追加了执行I/O的API。通过使用API,使用媒介管理软件的固有功能,就可以高速执行备份以及还原。Oracle中这个功能称为PROXY COPY。

 

通过这个API执行备份时,oracle将对象变更为备份模式之后,将包含在对象中的文件列表移交给媒介管理软件。媒介管理软件根据被移交的列表,使用各个公司自带的功能来启动数据库。

媒介管理软件对应PROXY COPY功能时,执行备份命令时,通过指定PRXY语句,可以执行使用了媒介管理软件固有功能的备份。

 

 

RUN{
ALLOCATE CHANNEL ch1 TYPE ‘SBT_TAPE’;
BACKUP
PROXY
(TABLESPACE SYSTEM FORMAT ‘system_%s_%p’); 
} 

为了使用PROXY COPY功能需要对应PROXY COPY的媒介管理软件。不对应PROXY COPY时,指定PROXY语句时也可以通过API来执行备份。但是PROXY COPY功能无法使用时就会报错。

指定备份池

在大部分的内存管理软件中,可以在tape装置以及那个装置中使用的tape组作为“池”的概念来管理。换言之,各个tape装置以及tape装置都属于某个池,对其备份时进行选择时,可以选择想使用的tape装置以及tape。

Oracle8 R8.0中,因为可以在备份时选择池,媒介管理软件所参考的环境变量中,采取重新指定池的方法以及指定RMAN的ALLOCATE CHANNEL命令PARMS语句中的环境变量的方法。

Oracle8i中通过RMAN的BACKUP命令,可以直接指定池。通过BACKUP命令的POOL语句可以指定对应数字进行设定。

 

RUN{
ALLOCATE CHANNEL ch1 TYPE ‘SBT_TAPE’;
BACKUP 
POOL 2
(TABLESPACE SYSTEM FORMAT ‘system_%S_%P’);
} 

能否使用POOL语句来指定依赖于媒介管理软件。详细内容请参考媒介管理软件手册。

3. Oracle Prallel Server(OPS)相关的功能扩展

 

备份时的磁盘亲和性(Disk Affinity)以及分散负荷

MPP结构的OPS中,因为会连接到各个设备的各个磁盘中,访问速度并不是相等的。因此,为了使得各个设备中的磁盘I/O最优化,需要尽可能对连接到本地的设备执行I/O。

Oracle8 R8.0的OPS环境中,使用RMAN执行备份时,为了执行磁盘I/O的最优化,重要的是,在用户端中对各个文件分割合适的节点的channel(执行备份的服务器进程)。以下是RMAN的命令例,在由4个节点构成的OPS环境中,对各节点连接到本地设备上的数据文件进行备份。

 

 

 

RUN{
ALLOCATE CHANNEL ch1 TYPE ‘SBT_TAPE’ CONNECT ‘sys/passwd@NODE1’;
ALLOCATE CHANNEL ch2 TYPE ‘SBT_TAPE’ CONNECT ‘sys/passwd@NODE2’;
ALLOCATE CHANNEL ch3 TYPE ‘SBT_TAPE’ CONNECT ‘sys/passwd@NODE3’;
ALLOCATE CHANNEL ch4 TYPE ‘SBT_TAPE’ CONNECT ‘sys/passwd@NODE4’;
BACKUP 
FILESPERSET 1 FORMAT ‘DBFILE_%s_%p’
(DATAFILE ‘/dev/rdsk/lv1’ CHANNEL ch1)
(DATAFILE ‘/dev/rdsk/lv2’ CHANNEL ch2)
(DATAFILE ‘/dev/rdsk/lv3’ CHANNEL ch3)
(DATAFILE ‘/dev/rdsk/lv4’ CHANNEL ch4);
}

Oracle8i的RMAN中,在执行备份时,以设备亲和性为基础,对备份对象数据文件进行集群化(group化)这时,备份对象的数据文件所属的节点中如果分配到channel的话,通过channel就可以进行备份。节点中没有分配到channel时,节点用于的数据文件,特别是不需要考虑到亲和性。对于可以使用的channel依次分配,执行备份。
以下的例子是设备亲和性有效的环境中,对之前页面的列表执行相同的备份的命令的例子。(数据文件 /dev/rdsk/lv1 ~ lv4包含于表领域tbs1中的情况)

 

RUN{
ALLOCATE CHANNEL ch1 TYPE ‘SBT_TAPE’ CONNECT ‘sys/passwd@NODE1’;
ALLOCATE CHANNEL ch2 TYPE ‘SBT_TAPE’ CONNECT ‘sys/passwd@NODE2’;
ALLOCATE CHANNEL ch3 TYPE ‘SBT_TAPE’ CONNECT ‘sys/passwd@NODE3’;
ALLOCATE CHANNEL ch4 TYPE ‘SBT_TAPE’ CONNECT ‘sys/passwd@NODE4’;
BACKUP
FORMAT ‘tbs1_%s_%p’ 
FILESPERSET 1
(TABLESPACE tbs1);
}

另外在磁盘亲和性有效的环境中,指定包含在备份set中的磁盘数的最小值,可以对磁盘进行数据分散。这个设定可以通过指定RMAN的BACKUP命令中的DISKRATIO参数来设定。(在DISKRATIO参数中指定整数)

RUN{
ALLOCATE CHANNEL ch1 TYPE ‘SBT_TAPE’ CONNECT ‘sys/passwd@NODE1’;
ALLOCATE CHANNEL ch2 TYPE ‘SBT_TAPE’ CONNECT ‘sys/passwd@NODE2’;
ALLOCATE CHANNEL ch3 TYPE ‘SBT_TAPE’ CONNECT ‘sys/passwd@NODE3’;
ALLOCATE CHANNEL ch4 TYPE ‘SBT_TAPE’ CONNECT ‘sys/passwd@NODE4’;
BACKUP 
FORMAT ‘%d_%s_%p’
DISKRATIO 3
(DATABASE);
}

在DISKRATIO中指定整数n时,各个备份set中所包含的数据文件的总计块数为了不超过n分之一,对各个数执行数据文件分割。
比如DISKRATIO指定为3时,就会如此调整备份set:对一个备份set,从一个备份set从一个磁盘读入的块数,就会小于整体块数的三分之一。(被分散到4个磁盘中)。
这是考虑到磁盘的亲和性的备份,支持MPP平台固有功能。您如果想查看自己使用的平台是否支持磁盘亲和性的话,请参考各个平台的管理者手册。

Oracle8i的OPS中不会考虑磁盘亲和性,日志归档以及复制数据文件的备份时,不会考虑到Load Balancing。

【Oracle数据恢复】ORA-01115、ORA-01110、ORA-27091、ORA-27070、OSD-04006、O/S-Error

某用户windows 2003上的数据库由于存储故障导致系统表空间system.dbf出现IO问题,当打开数据库OPEN database时报错:

 

 

ORA-01115: IO error reading block from file 15 
ORA-01110: data file … 
ORA-27091: unable to queue I/O 
ORA-27070: async read/write failed 
OSD-04006: ReadFile() failure, unable to read from file 
O/S-Error: (OS 121) The semaphore timeout period has expired.

以上ORA-01115、ORA-01110、ORA-27091、ORA-27070、OSD-04006、O/S-Error 这堆报错本质与Oracle数据库层面没有关系,问题的根本原因是Windows上对应磁盘驱动器下的文件无法读取出来,这可能是OS bug 也可能就是对应磁盘出现了坏道或其他物理故障,所以对于该问题有限考虑在OS层面解决文件的读取问题, 如果确实发现无法从OS层面或从备份解决,那么可以考虑特殊的恢复手段。‘

 

如果自己搞不定可以找诗檀软件专业ORACLE数据库修复团队成员帮您恢复!

诗檀软件专业数据库修复团队

服务热线 : 13764045638 QQ号:47079569 邮箱:service@parnassusdata.com

 

 

Error: OSD 4006
Text: ReadFile() failure, unable to read from file
—————————————————————————
Cause: Unexpected return from Windows NT system service ReadFile()
Action: Check OS error code and consult Windows NT documentation

This is due to a problem in Windows such that when Oracle attempted to access the data file on that device, it could not because the device timed out. This suggests that Windows has run out of asynchronous I/O buffers or there is a communications delay on the device.

There is nothing you’re going to be able to do at the database level to resolve this error, unless you move the data files to another drive. Ask the O/S system administrator to run diagnostics tools to check for possible faulty hardware and disk corruption on the disk device where the error is showing in the loader log. If the error persists, then log a call with Microsoft Support.

Oracle processes may encounter various (OS 1117) errors on a Windows 2003 Server. The text of the (OS 1117) error can be seen as follows:

C:\>net helpmsg 1117
The request could not be performed because of an I/O device error.
This error may manifest itself in different ways, depending on which Oracle process encounters the error:

Oracle RDBMS Instance Encounters (OS 1117) error

1. If an Oracle RDBMS instance encounters the error, you may see messages such as the following in the alert log for the RDBMS instance:

==========================================================================

Fri Jul 13 01:21:33 2007
Errors in file d:\oracle\db\product\admin\mydb\bdump\mydb1_lmon_4608.trc:
ORA-27091 : unable to queue I/O
ORA-27070 : async read/write failed
OSD-04006: ReadFile() failure, unable to read from file
O/S-Error: (OS 1117) The request could not be performed because of an I/O device error.
==========================================================================

Oracle ASM Instance Encounters (OS 1117) errors

2. If an Oracle ASM instance encounters the error, you may see similar errors in the ASM instance’s Alert log:

============================================
Fri Jul 13 01:22:10 2007
Errors in file d:\oracle\asm\product\admin\+asm\bdump\+asm1_gmon_3836.trc:
ORA-27091 : unable to queue I/O
ORA-27070 : async read/write failed
OSD-04016: Error queuing an asynchronous I/O request.
O/S-Error: (OS 1117) The request could not be performed because of an I/O device error.
============================================

CRS Daemon (crsd.exe) encounters 1117 errors
3. If you are running in an Oracle Clusterware environment, then you may also see errors in the crsd.log and/or certain resource logs, indicating a problem accessing the OCR (Oracle Cluster Registry). An example of those errors would be:

================================================
2007-07-13 01:21:51.766: [ OCROSD][4272]utwrite:4: Problem writing the buffer phy offset 184320 and oserror 1117
2007-07-13 01:21:51.766: [ OCROSD][4352]utwrite:4: Problem writing the buffer phy offset 184320 and oserror 1117
2007-07-13 01:21:51.766: [ OCRRAW][4352]beginlog: problem 26 clearing the log metadata buffer
2007-07-13 01:21:51.766: [ OCRRAW][4352]proprdkey: Problem in begin log
2007-07-13 01:21:51.766: [ OCRRAW][4352]proprseterror: Error in accessing physical storage [26] Marking context invalid.
================================================

CSS Daemon (ocssd.exe) encounters 1117 errors

4. Also, in an Oracle Clusterware environment, the Cluster Synchronization Services daemon (ocssd.exe) may experience problems accessing the voting disk. If this occurs, you will see an error in the ocssd.log similar to the following:

============================================
[ CSSD]2007-07-13 01:22:12.501 [4052] >ERROR: Internal Error Information:
Category: 1234
Operation: scls_block_write
Location: WriteFile
Other: unable to write block(s)
Dep: 1117

[ CSSD]2007-07-13 01:22:12.501 [4052] >ERROR: clssnmvReadBlocks: read failed 1 at offset 533 of \\.\votedsk2
[ CSSD]2007-07-13 01:22:12.501 [4052] >TRACE: clssnmDiskStateChange: state from 4 to 3 disk (1/\\.\votedsk2)
[ CSSD]2007-07-13 01:22:12.501 [2200] >TRACE: clssnmDiskPMT: disk offline (1/\\.\votedsk2)
[ CSSD]2007-07-13 01:22:12.501 [2200] >ERROR: clssnmDiskPMT: Aborting, 1 of 2 voting disks unavailable
[ CSSD]2007-07-13 01:22:12.501 [2200] >ERROR: ###################################
[ CSSD]2007-07-13 01:22:12.501 [2200] >ERROR: clssscExit: CSSD aborting
[ CSSD]2007-07-13 01:22:12.501 [2200] >ERROR: ###################################
==============================================

5. When you are running in an Oracle Clusterware environment, if the ocssd process encounters an I/O error when accessing the Voting Disk, the CSS daemon will evict the node from the cluster. This is done by signalling the Oracle Fence Driver (OraFencedrv.sys) to reboot the machine. When the fence driver reboots the machine, this will be seen as a bugcheck with stop code 0x0000ffff. You will be able to see this in the System Log with a message such as:

The computer has rebooted from a bugcheck.
The bugcheck was: 0x0000ffff (0x0000000000000000, 0x0000000000000000,
0x0000000000000000, 0x0000000000000000).
A dump was saved in: C:\WINDOWS\MEMORY.DMP.

Note that the bugcheck is expected behavior when ocssd.exe (the Cluster Synchornization Services daemon) encounters an I/O error when accessing the voting disk. The node experiencing the I/O error is intentionally rebooted to avoid a split-brain and possible data corruption when access to the voting disk is lost.
CHANGES

You may encounter this error after upgrading the Microsoft Storport driver to version 5.2.3790.4021 or later.

CAUSE

Reference Microsoft KB article#932755, available at the following URL:

http://support.microsoft.com/default.aspx?scid=kb;EN-US;932755

Per that article, one of the changes introduced in this version of the Storport driver is the following:

=========================================================
If a target returns a SCSI status of BUSY or Task Set Full, the port driver retries the command immediately. Storport retries the command an unlimited number of times. Therefore, if the busy status continues, the system could eventually experience problems.

This update configures the following behavior:

• It limits the number of retries. The default is 20.

• If the target returns a status of BUSY, the Storport driver performs a time-based pause before the Storport driver retries the command.

• If the target returns a status of Task Set Full, the Storport driver performs an I/O completion-based pause before the Storport driver retries the command.
=========================================================

Therefore, prior to upgrading the Storport driver, if a storage path had become saturated, the Storport driver would immediately continue to retry – indefinitely. This would result in slow I/O and perhaps a hang or spin scenario, but no error would be returned.

With the later version of the Storport driver, the retries are limited to 20 retries by default, with a pause between each retry. After 20 failures with a device busy status, the (OS 1117) error is returned to applications waiting on I/O. For more information on changes to the Storport driver, you must contact Microsoft.

SOLUTION

This is an I/O performance problem. You will need to increase the performance/capacity of the storage system to avoid the prolonged BUSY status. Specific solutions will vary, depending on your storage vendor, so the storage vendor may need to be contacted to assist with tuning the storage. One potential solution includes implementing multi-pathing technology to improve the throughput of the storage.

沪ICP备14014813号

沪公网安备 31010802001379号