Server Crashes

HCL Domino Domino Forum
1

Server crash started 4 days ago. The only thing that I can relate it to is that it seems to crash a few minutes after a database backup completes. Also, since the first crash I’m receiving errors in replication for a dozen or so mail files:Attempting to write invalid timedate to replication history of F:\Lotus\Domino\Data\mail\xxx.nsf: 12/10/**** 10:24:59 PM

Could someone shed a little light on what might be causing these crashes?

8/9/09:

############################################################

thread 1/4: [nReplica: 07a0: 07a4] FATAL THREAD (Panic)

FP=0x00125aac, PC=0x7c82860c, SP=0x00125a3c

stkbase=0x00130000, total stksize=77824, used stksize=42436

EAX=0x00000001, EBX=0x0012667c, ECX=0x00000001, EDX=0x7c82860c

ESI=0x00000600, EDI=0x00000000, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00000297

############################################################

[ 1] 0x7c82860c ntdll.KiFastSystemCallRet+0 (600,493e0,0,12625c)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (600,493e0,1269f0,126694)

@[ 3] 0x601a2e75 nnotes.FRSendCommandToService+789 (12627c,12667c,126694,5)

@[ 4] 0x601a3aaf nnotes.OSRunExternalScript@8+1055 (12c,1)

@[ 5] 0x601a402f nnotes.FRTerminateWindowsResources+975 (1,1010,1,0)

@[ 6] 0x601a43f8 nnotes.OSFaultCleanupExt@24+888 (b36a14,1010,0,0,0,126d18)

@[ 7] 0x601a447a nnotes.OSFaultCleanup@12+26 (0,1010,0)

8/10/09:

############################################################

thread 1/3: [ nUpdAll: 0ea8: 0ffc] FATAL THREAD (Panic)

FP=0x00129610, PC=0x7c82860c, SP=0x001295a0

stkbase=0x00130000, total stksize=77824, used stksize=27232

EAX=0x003e0000, EBX=0x0012a1e0, ECX=0x003f0b00, EDX=0x003e0000

ESI=0x000003cc, EDI=0x00000000, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00000297

############################################################

[ 1] 0x7c82860c ntdll.KiFastSystemCallRet+0 (3cc,493e0,0,129dc0)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (3cc,493e0,12a554,12a1f8)

@[ 3] 0x601a2e75 nnotes.FRSendCommandToService+789 (129de0,12a1e0,12a1f8,5)

@[ 4] 0x601a3aaf nnotes.OSRunExternalScript@8+1055 (12c,1)

@[ 5] 0x601a402f nnotes.FRTerminateWindowsResources+975 (1,1010,1,0)

@[ 6] 0x601a43f8 nnotes.OSFaultCleanupExt@24+888 (b36a14,1010,0,0,0,12a87c)

@[ 7] 0x601a447a nnotes.OSFaultCleanup@12+26 (0,1010,0)

8/11/09:

############################################################

thread 14/41: [ nRouter: 0cb8: 0e64] FATAL THREAD (Panic)

FP=0x078d31fc, PC=0x7c82860c, SP=0x078d318c

stkbase=0x078e0000, total stksize=262144, used stksize=52852

EAX=0x00000000, EBX=0x078d3dcc, ECX=0x0025060c, EDX=0x00250608

ESI=0x00000768, EDI=0x00000000, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00000297

############################################################

[ 1] 0x7c82860c ntdll.KiFastSystemCallRet+0 (768,493e0,0,78d39ac)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (768,493e0,78d4140,78d3de4)

@[ 3] 0x601a2e75 nnotes.FRSendCommandToService+789 (78d39cc,78d3dcc,78d3de4,5)

@[ 4] 0x601a3aaf nnotes.OSRunExternalScript@8+1055 (12c,1)

@[ 5] 0x601a402f nnotes.FRTerminateWindowsResources+975 (1,1010,1,0)

@[ 6] 0x601a43f8 nnotes.OSFaultCleanupExt@24+888 (ba6a14,1010,0,0,0,78d4468)

@[ 7] 0x601a447a nnotes.OSFaultCleanup@12+26 (0,1010,0)

8/12/09:

############################################################

thread 27/36: [ nRouter: 078c: 0644] FATAL THREAD (Panic)

FP=0x1eddec84, PC=0x7c82860c, SP=0x1eddec14

stkbase=0x1edf0000, total stksize=262144, used stksize=70636

EAX=0x00150640, EBX=0x1eddf854, ECX=0x1edddf7c, EDX=0x00002000

ESI=0x00000714, EDI=0x00000000, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00000297

############################################################

[ 1] 0x7c82860c ntdll.KiFastSystemCallRet+0 (714,493e0,0,1eddf434)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (714,493e0,1eddfbc8,1eddf86c)

@[ 3] 0x601a2e75 nnotes.FRSendCommandToService+789 (1eddf454,1eddf854,1eddf86c,5)

@[ 4] 0x601a3aaf nnotes.OSRunExternalScript@8+1055 (12c,1)

@[ 5] 0x601a402f nnotes.FRTerminateWindowsResources+975 (1,1010,1,0)

@[ 6] 0x601a43f8 nnotes.OSFaultCleanupExt@24+888 (ba6a14,1010,0,0,0,1eddfef0)

@[ 7] 0x601a447a nnotes.OSFaultCleanup@12+26 (0,1010,0)

2

Subject: Need the full call stacks to get a better idea…

First thing I would suggest is upgrade Domino Server to Release 8.5 FP1 - You can check out the Fixlist and it does have a few fixes for Domino Server Crashes…

jpaganet@us.ibm.com

3

Subject: Installed update this morning

Will have to wait & see if she still crashes!

I’m still receiving errors on the mail files.

I’ve run fixup on them - didn’t work.

4

Subject: Can you post the FATAL THREAD section of the NSD

Hi

Can you post the FATAL THREAD section of the NSD files with the few lines under it

For example:

############################################################

FATAL THREAD 1/2 [ domdsm:107a8:67076]

FP=0x0012a34c, PC=0x60069a86, SP=0x0012a2f0, stksize=92

EAX=0x01a3088c, EBX=0x01a48650, ECX=0x0000bffe, EDX=0x01a325e6

ESI=0x00004000, EDI=0x0032ff78, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010206

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x60069a86 nnotes._fdDelete@4+118 (4000,12a368,608770ef,4000)

@[ 2] 0x60069a0c nnotes._OSFileClose@4+12 (4000,12adc0,12a380,60872ea7)

@[ 3] 0x608770ef nnotes._sqloclose+15 (4000,12b59c,5010,0)

@[ 4] 0x60872ea7 nnotes._sqlpgclf+71 (12adc0,1aaa2cb,70,1aaa2f3)

@[ 5] 0x60885458 nnotes._sqlpgfdl+616 (12b9a4,1aaa2cb,41,12be44)

@[ 6] 0x60876a6f nnotes._sqlpgcdl+927 (12bf44,12b9a4,0,12be44).

Date

Topic

Frequently asked question - How to analyze Notes/Domino NSDs? (Jean-Yves Riverin)

call IBM support

JYR