admin
2010-08-18
Hdr: 5960580 10.2.0.2 RDBMS 10.2.0.2 RAC PRODID-5 PORTID-23 5181297
Abstract: LMON CRASHING WITH ORA-481 ERROR
PROBLEM:
--------
LMON crashed with ora-481 on Mon Mar 26 18:53:47 2007. Ct said that a
particular job which was completing in 3.5 hours on Friday March 23, was
taking 5 hours on Saturday March 24 and after the crash (March 26) is taking
12 to 15 hours.
I am not sure whether the crash and the performance are related. The crash
did NOT happen when the job was running.
Let us consider this bug for only the LMON crash.
DIAGNOSTIC ANALYSIS:
--------------------
alert_PFNR1011.log:
==========================
Mon Mar 26 18:53:36 2007
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 26 18:53:36 2007
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 26 18:53:36 2007
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 26 18:53:36 2007
WARNING: inbound connection timed out (ORA-3136)
Mon Mar 26 18:53:36 2007
???????
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_lmon_1349.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
LMON: terminating instance due to error 481
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_lck0_1767.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_dbw2_1485.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_lms0_1354.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_lmd0_1352.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_pmon_1341.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_lms2_1363.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_lms4_1410.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_lms6_1425.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
System state dump is made for local instance
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_lms1_1359.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_lms3_1397.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_lms5_1419.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_lms7_1432.trc:
ORA-481: LMON process terminated with error
System State dumped to trace file
/oracle/g01/admin/PFNR1011/bdump/pfnr1011_diag_1344.trc
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_ckpt_1513.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_j002_5622.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_dbw3_1491.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:47 2007
Errors in file /oracle/g01/admin/PFNR1011/bdump/pfnr1011_j003_5637.trc:
ORA-481: LMON process terminated with error
Mon Mar 26 18:53:52 2007
Instance terminated by LMON, pid = 1349
Mon Mar 26 18:54:50 2007
Starting ORACLE instance (normal)
Mon Mar 26 18:55:29 2007
Reviewing pfnr1011_lmon_1349.trc:
===================================
*** 18:53:27.122
kjfcdrmrfg: SYNC TIMEOUT (275372, 274471, 900), step 31
Submitting asynchronized dump request [28]
kjctseventdump-end tail 205 heads 0 @ 0 205 @ 1047039906
sync() timed out - lmon exiting
kjfsprn: sync status inst 0 tmout 900 (sec)
kjfsprn: sync propose inc 8 level 396
kjfsprn: sync inc 8 level 396
waiting for 'ges remote message' blocking sess=0x0 seq=4510 wait_time=0
seconds since wait started=442
waittime=40, loop=0, p3=0
Dumping Session Wait History
for 'ges remote message' count=1 wait_time=746343
waittime=40, loop=0, p3=0
WORKAROUND:
-----------
no
RELATED BUGS:
-------------
similar to bug 5399702 - base bug 5181297
could be bug 4947571 - base bug 4940890
REPRODUCIBILITY:
----------------
no
TEST CASE:
----------
no
STACK TRACE:
------------
*** 18:53:27.159
Dumping diagnostic information for ospid 1352:
OS pid = 1352
loadavg : 1.44 1.39 1.39
swap info: free_mem = 34099.71M rsv = 27064.04M
alloc = 26321.46M avail = 95169.25 swap_free = 95911.83M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY
TIME CMD
0 S oracle 1352 1 0 40 20 ? 2646619 ? Mar 23
console 22:22 ora_lmd0_PFNR1011
1352: ora_lmd0_PFNR1011
----------------- lwp# 1 / thread# 1 --------------------
ffffffff7a8cde8c ioctl (9, 373f, 105e6ba10)
ffffffff7fffc910) + 15f4
710
0000000100e662e0 ksliwat (105c00, 2, 8, 91a000160, 0, 0) + b60
0000000100e66820 kslwaitns (8, 1, 32, 0, 40, 0) + 20
00000001010d61f4 kskthbwt (8, 1, 32, 0, 40, 0) + d4
0000000100e667dc kslwait (8, 32, 0, 15e, 190, 0) + 5c
00000001010e8344 ksxprcv (104fb4, 105d68558, 104fb4128, 1468, 105d68,
104fb4000) + 364
0000000101591254 kjctr_rksxp (40, 403fe88f8, 0, ffffffff7fffd978, 14,
ffffffff7fffd974) + 1f4
0000000101592e24 kjctrcv (ffffffff79626208, 403fe88f8, 105e992c0,
ffffffff7fffe1bc, 40, 32) + 164
000000010157f6a0 kjcsrmg (ffffffff796261f0, 0, 40, 32, 0, 105d71) + 60
00000001015dc098 kjmdm (a, 44, 4097ab030, 8, 4097ab030, 0) + 2ff8
0000000101002a60 ksbrdp (105d6b, 380007774, 380000, 38000e, 105c00,
1015d90a0) + 380
00000001024219f8 opirip (105d75000, 105c00, 105d7d, 380007000, 105d75,
105df1ae0) + 338
00000001002fe790 opidrv (105d77d18, 1, 32, 0, 32, 105c00) + 4b0
00000001002f8e30 sou2o (ffffffff7ffff468, 32, 4, ffffffff7ffff490,
1056ac000, 1056ac) + 50
00000001002bc2ec opimai_real (3, ffffffff7ffff568, 0, 0, 1e42ee4, 14400) +
10c
00000001002bc118 main (1, ffffffff7ffff678, 0, ffffffff7ffff570,
ffffffff7ffff680, ffffffff7aa00140) + 98
00000001002bc03c _start (0, 0, 0, 0, 0, 0) + 17c
SUPPORTING INFORMATION:
-----------------------
will be uploaded
24 HOUR CONTACT INFORMATION FOR P1 BUGS:
----------------------------------------
Viral Shah - customer - 267-467-6950. This is a sev2 p1 SR.
DIAL-IN INFORMATION:
--------------------
IMPACT DATE:
------------
Ct is on pre-production. Planning to go production next week. Ct saidd he
cannot upgrade to 10.2.0.3.