Help: Domino server keep crashing (or process hang) during transaction log media recovery (in 6.0.1 and 6.5, but not in 6.0.2 CF2)

HCL Domino Domino Forum
1

Hello, gurus:

I have a db backup copy from 6.0.1, my Domino server transactional logging was in ARCHIVE mode, and all logs are still on disk.

When I used the C API backup/restore tool try to recover it, the program hang after output showing latest changes updated. (NSFRecoverDatabases() not returning, by using NSD, the calling stack is shown as below) Domino server seems to be fine at this moment, but it will crash when I try to shutdown it. (some msg about “PANIC: a waiting thread already teminated”) After I killed all the notes processes and restart Domino server, the recovered database can be opened and the latest changes were back.

############################################################

FP=00124604, PC=77f94091, SP=001245e0, stkbase=00030000, stksize=278528

############################################################

[ 1] 0x77f94091 ntdll.NtWaitForSingleObject+11 (2bc,ffffffff,0,600a6127)

[ 2] 0x7c4f1b1b KERNEL32.WaitForSingleObject+15 (1d0,7673db5,60bd15e0,0)

@[ 3] 0x600f9216 nnotes._WaitOnNativeSemaphoreCounted@12+22 (1d0,7673db5,0,1249ec)

@[ 4] 0x60008567 nnotes._OSLockWriteFRWSemInt@12+279 (7673db3,1,0,1246bc)

@[ 5] 0x60008440 nnotes._OSLockWriteFRWSem@4+16 (7673db3,124774,6071fde1,7673db1)

@[ 6] 0x601a08ef nnotes._OSWaitFairEvent@8+15 (7673db1,0,4,5)

@[ 7] 0x6071fde1 nnotes._LkDumpLockAndRequests@4+8801 (1249ec,124820,20005,ffffffff)

@[ 8] 0x6071c1d1 nnotes.__LkLock@24+33 (1249ec,124820,20005,ffffffff)

@[ 9] 0x6071cf80 nnotes._LkConvert@16+432 (1249ec,5,5,ffffffff)

@[10] 0x600b8572 nnotes._ConvertDbWriteSemExclusive@4+130 (1249ec,0,3b,124d14)

@[11] 0x6003bd23 nnotes._DbClose@20+3187 (1249ec,0,1,1)

@[12] 0x6003d7b0 nnotes._NSFDbCloseExtended3@16+96 (3b,1,0,1)

@[13] 0x6003d746 nnotes._NSFDbCloseExtended2@12+22 (3b,1,0,124a60)

@[14] 0x6003d722 nnotes._NSFDbCloseExtended@8+18 (3b,1,0,1bb0000)

@[15] 0x60796990 nnotes.RmDbTable::CloseAllDbs+192 (0,89a166,4a3560,0)

@[16] 0x607982de nnotes._NSFRecoverDatabases@20+1710 (12b8198,4039a0,20,124d88)

Then I upgraded my Domino server to 6.0.2 CF2, the above issue was gone. Everything looked normal.

But after I upgraded the Domino server to 6.0.5, the Domino server began to crash during media recovery: the output screen showing the following before crash:

Media Recovery Replay (153 MB): 10% 20%

From the NSD file created, the crashing process is nserver.exe:

############################################################

thread 63/71: [ nserver:0d24: 3588]

FP=074dea3c, PC=77f94091, SP=074dea18, stkbase=073e0000, stksize=262144

############################################################

[ 1] 0x77f94091 ntdll+82065 (f0,36ee80,0,601a0993)

[ 2] 0x7c4f1b1b KERNEL32._WaitForSingleObject@8+15 (74df1d4,7c4ed360,0,74df724)

@[ 3] 0x6019faf0 nnotes._OSFaultCleanup@12+1504 (0,0,0,74df724)

@[ 4] 0x601a9509 nnotes._OSNTUnhandledExceptionFilter@4+185 (74df724,7c4ed360,0,74dffdc)

[ 5] 0x7c51bdbd KERNEL32._UnhandledExceptionFilter@4+146 (74df724,7c4ff107,74df72c,0)

[ 6] 0x7c50aab8 KERNEL32._wcscat+127 (0,0,0,0)

############################################################

FATAL THREAD 63/71 [ nserver:0d24: 3588]

FP=0x074dfadc, PC=0x60836f2a, SP=0x074dfad8, stksize=4

EAX=0x0e3310f8, EBX=0x074dfc84, ECX=0x0e3310f8, EDX=0x00000000

ESI=0x074dfc84, EDI=0x0e3310f8, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x00000038, GS=0x00000000 Flags=0x00010246

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x60836f2a nnotes.TRANENTRY::dbListLookupIndex+10 (23520e8,74dfc84,0,74dfb58)

@[ 2] 0x6083771b nnotes.TRANENTRY::dbListPhysUnlock+107 (74dfc84,74dfb58,23520e8,74dfc84)

@[ 3] 0x607fc90d nnotes._RmTransactionEndCond@8+509 (74dfc84,74dfb58,74dfc84,23520e8)

@[ 4] 0x60004205 nnotes._UnlockDbReadSemCtx@4+117 (0,1,13a3,74dfcc4)

@[ 5] 0x6000438b nnotes._UnlockDbRO@8+27 (74dfc84,108,9713000,92470d8)

@[ 6] 0x600cade6 nnotes._BITMAPWritePage@36+710 (3de7,1,0,1000)

@[ 7] 0x607a211e nnotes._UBMProcessIOThread@0+942 (0,32,74dff94,100016cb)

@[ 8] 0x1000a664 nserverl._UBMIOTask@8+260 (18770037,5f900001,3,f8d)

@[ 9] 0x100016cb nserverl._Scheduler@4+763 (0,f8d,0,0)

@[10] 0x60114924 nnotes._ThreadWrapper@4+212 (0,0,f8d,0)

[11] 0x7c4e987c KERNEL32._BaseThreadStart@8+82

Is it possible that some transaction logging related bug was fixed in 6.0.2 CF2 but not in 6.5 ?

Thanks a lot !

2

Subject: Help: Domino server keep crashing (or process hang) during transaction log media recovery (in 6.0.1 and 6.5, but not in 6.0.2 CF2)

Hi

I have similar experience with 6.0.1, 6.0.1CF1 and 6.5. The Domino server crashed during the apply of transaction log if the amount of logs to be applied is over 15MB or so.

There is a SPR for this case (SPR PROE5JWUN2) which is claimed to be fixed in 6.0.1CF2.

I will try to test it against 6.0.1CF2, 6.0.2CF1 and CF2 and 6.0.3 if I have time.

Anyway it’s ridiculous and I wonder IBM need to update the test case for transaction apply!

thx,

BG

3

Subject: RE: Help: Domino server keep crashing (or process hang) during transaction log media recovery (in 6.0.1 and 6.5, but not in 6.0.2 CF2)

Hi, bc,

Do you know if Lotus is planning to fix it ASAP or it’s already fixed? (I mean the issue with Domino 6.5)

Thanks a lot!

4

Subject: RE: Help: Domino server keep crashing (or process hang) during transaction log media recovery (in 6.0.1 and 6.5, but not in 6.0.2 CF2)

Here is my new result for this issue: 6.0.2CF2 - good

6.0.3 - bad

6.5 - bad

If you bought IBM support, you can ask for a hot fix for SPR JCHN5QVL3E. This SPR has been included in 6.5.1 but I have not tested yet.

-BG.