NSD after shutdown/restart Domino 8.5 server on Windows 2008

HCL Domino Domino Forum
1

We have installed Domino 8.5 HF1 on Windows 2008.Domino Server works fine until we restart (res ser) of shutdown the server. We get a NSD report. Restart doesn’t work.

NSD info W2000-D8.5 :

console_TEST-SRVR01_2009_02_16@10_43_23.log

16-02-2009 10:43:51 Schedule Manager shutdown complete

16-02-2009 10:43:51 Administration Process shutdown

16-02-2009 10:43:51 AMgr: Executive ‘1’ shutting down. Process id ‘3840’

16-02-2009 10:43:51 Database Replicator shutdown

16-02-2009 10:43:52 Index update process shutdown

16-02-2009 10:43:52 Agent Manager shutdown complete

16-02-2009 10:43:53 Router: Mail Router shutdown

16-02-2009 10:43:55 Event Monitor shutdown

16-02-2009 10:44:08 Server shutdown complete

2-2009 10:43:28 SchedMgr: Done validating schedule database

16-02-2009 10:43:28 Router: Mail Router started for domain TEST_NL

16-02-2009 10:43:28 Router: Internet SMTP host TEST-SRV01 in domain TEST.NL

16-02-2009 10:43:28 Database Server started

quit

Quiting16-02-2009 10:43:51 Router: Shutdown is in progress

16-02-2009 10:43:51 Calendar Connector shutdown

16-02-2009 10:43:51 Rooms and Resources Manager shutdown complete

nsd_W32I_TEST-SRVR01_2009_02_16@10_44_08.log

<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

Section: Summary (Time 10:44:22)

<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

Successfully detached from all processes

Started at: Mon Feb 16 10:44:08 2009

Ended at: Mon Feb 16 10:44:22 2009

Time Spent: 00:00:14

Generated Messages:

INFO (0): failed to match all of the Notes processes

INFO (0): Generating core dump for [ nserver: 075c] (coreflags=0, exp=1d1ca30, dbgver=6.8.0004.0)

INFO (0): Generated core dump file C:\Program Files\IBM\Lotus\Domino\data\IBM_TECHNICAL_SUPPORT\core_nserver_W32I_TEST-SRVR01_2009_02_16@10_44_08.dmp

INFO (0): Memcheck running stand-alone

INFO (0): Remote Drive query disabled. Refer to help for more information.

INFO (0): terminated process [ nserver: 075c]

INFO (0): Deleting pid.nbf

WARNING (3): can’t get process 0x93c status

WARNING (3): can’t get process 0xee4 status

WARNING (3): can’t get process 0x47c status

WARNING (3): can’t get process 0xfb4 status

Please send the following files:

            NSD Log : C:\Program Files\IBM\Lotus\Domino\data\IBM_TECHNICAL_SUPPORT\nsd_W32I_TEST-SRVR01_2009_02_16@10_44_08.log

          Core Dump : C:\Program Files\IBM\Lotus\Domino\data\IBM_TECHNICAL_SUPPORT\core_nserver_W32I_TEST-SRVR01_2009_02_16@10_44_08.dmp

NSD info W2008-D8.5 after hf1

console_TEST-SRVR01_2009_02_16@13_43_05.log

16-02-2009 13:43:46 Administration Process shutdown

16-02-2009 13:43:47 Calendar Connector shutdown

16-02-2009 13:43:47 Rooms and Resources Manager shutdown complete

16-02-2009 13:43:47 Database Replicator shutdown

16-02-2009 13:43:47 Agent Manager shutdown complete

16-02-2009 13:43:49 Router: Mail Router shutdown

16-02-2009 13:43:50 Index update process shutdown

16-02-2009 13:43:50 Event Monitor shutdown

16-02-2009 13:43:55 Server shutdown complete

9 13:43:14 RnRMgr: Done validating schedule database

16-02-2009 13:43:14 Router: Mail Router started for domain TEST_NL

16-02-2009 13:43:14 Router: Internet SMTP host TEST-SRV01 in domain TEST.NL

16-02-2009 13:43:14 SchedMgr: Done validating schedule database

quit

Quiting16-02-2009 13:43:46 Schedule Manager shutdown complete

16-02-2009 13:43:46 Router: Shutdown is in progress

16-02-2009 13:43:46 AMgr: Executive ‘1’ shutting down. Process id ‘680’

nsd_W32I_TEST-SRVR01_2009_02_16@10_44_08.log

<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

Section: Summary (Time 10:44:22)

<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

Successfully detached from all processes

Started at: Mon Feb 16 10:44:08 2009

Ended at: Mon Feb 16 10:44:22 2009

Time Spent: 00:00:14

Generated Messages:

INFO (0): failed to match all of the Notes processes

INFO (0): Generating core dump for [ nserver: 075c] (coreflags=0, exp=1d1ca30, dbgver=6.8.0004.0)

INFO (0): Generated core dump file C:\Program Files\IBM\Lotus\Domino\data\IBM_TECHNICAL_SUPPORT\core_nserver_W32I_TEST-SRVR01_2009_02_16@10_44_08.dmp

INFO (0): Memcheck running stand-alone

INFO (0): Remote Drive query disabled. Refer to help for more information.

INFO (0): terminated process [ nserver: 075c]

INFO (0): Deleting pid.nbf

WARNING (3): can’t get process 0x93c status

WARNING (3): can’t get process 0xee4 status

WARNING (3): can’t get process 0x47c status

WARNING (3): can’t get process 0xfb4 status

Please send the following files:

            NSD Log : C:\Program Files\IBM\Lotus\Domino\data\IBM_TECHNICAL_SUPPORT\nsd_W32I_TEST-SRVR01_2009_02_16@10_44_08.log

          Core Dump : C:\Program Files\IBM\Lotus\Domino\data\IBM_TECHNICAL_SUPPORT\core_nserver_W32I_TEST-SRVR01_2009_02_16@10_44_08.dmp

Any suggestions ?

2

Subject: Seems to be related to a network thing - Wine

############################################################

FATAL THREAD 1/3 [ nserver: 0f18: 0f1c]

FP=0x0006f0a8, PC=0x775e962a, SP=0x0006f09c

stkbase=00070000, total stksize=49152, used stksize=3940

EAX=0x00000001, EBX=0x001b4140, ECX=0x800c6016, EDX=0x00002000

ESI=0x00000000, EDI=0x00000000, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010246

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

[ 1] 0x775e962a ntdll.RtlDestroyHeap+1009 (0,2,1b4070,1b37f0)

[ 2] 0x775e96a1 ntdll.RtlDestroyHeap+1128 (0,2,1,6f148)

[ 3] 0x775e96a1 ntdll.RtlDestroyHeap+1128 (0,2,77635684,1b08f8)

[ 4] 0x775e9723 ntdll.RtlDestroyHeap+1258 (73b50000,6f1ac,776356c0,1b08f8)

[ 5] 0x775ee8e2 ntdll.LdrUnloadDll+70 (73b50000,0,6f1f4,76e09524)

[ 6] 0x76d60979 kernel32.FreeLibrary+129 (73b50000,1b1588,76e094f1,1b1588)

[ 7] 0x76e09524 WS2_32.WSAEnumProtocolsW+691 (1,1b08f8,76e093e0,1b08f8)

[ 8] 0x76e094da WS2_32.WSAEnumProtocolsW+617 (1,199b70,76e09857,76d2e875)

[ 9] 0x76e09906 WS2_32.WahEnumerateHandleContexts+805 (1,202,0,196eb8)

[10] 0x76e097e1 WS2_32.WahEnumerateHandleContexts+512 (1,202,0,181940)

[11] 0x76e096fe WS2_32.WahEnumerateHandleContexts+285 (6197159d,6f2cc,166c0001,3cd0ef4)

@[12] 0x6197123f nNTCP.ProcessTerminateWinSock+79 (166c0001,202,0,0)

@[13] 0x60b7b462 nnotes.Drvr_Free@4+114 (166c0001)

@[14] 0x60b78ad6 nnotes.Process_FreeAllDrvrs@4+166 (11900001)

@[15] 0x60b78bad nnotes.Process_Free@4+13 (11900001)

@[16] 0x60b78de9 nnotes.Root_FreeProcess@4+89 (11900001)

@[17] 0x60b79235 nnotes.nti_cleanup@0+37 ()

@[18] 0x60789a50 nnotes.NetTerm@4+80 (0)

@[19] 0x6017305d nnotes.OSTerm@0+861 ()

@[20] 0x1000152c nserverl.ServerMain@16+924 (2,1a0fe14,0,0)

RtlDestroyHeap

http://www.google.ca/search?hl=fr&q=RtlDestroyHeap&btnG=Rechercher&meta=

WS2_32.WSAEnumProtocolsW

http://source.winehq.org/WineAPI/WSAEnumProtocolsW.html

3

Subject: Could you either post or send the nsd

to me. I can take a look.

swatts@us.ibm.com

–Steve

4

Subject: Can you post the FATAL THREAD section of the NSD…

Hi Louise

Can you post the FATAL THREAD section of the NSD files with the few lines under it

For example:

############################################################

FATAL THREAD 1/2 [ domdsm:107a8:67076]

FP=0x0012a34c, PC=0x60069a86, SP=0x0012a2f0, stksize=92

EAX=0x01a3088c, EBX=0x01a48650, ECX=0x0000bffe, EDX=0x01a325e6

ESI=0x00004000, EDI=0x0032ff78, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010206

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x60069a86 nnotes._fdDelete@4+118 (4000,12a368,608770ef,4000)

@[ 2] 0x60069a0c nnotes._OSFileClose@4+12 (4000,12adc0,12a380,60872ea7)

@[ 3] 0x608770ef nnotes._sqloclose+15 (4000,12b59c,5010,0)

@[ 4] 0x60872ea7 nnotes._sqlpgclf+71 (12adc0,1aaa2cb,70,1aaa2f3)

@[ 5] 0x60885458 nnotes._sqlpgfdl+616 (12b9a4,1aaa2cb,41,12be44)

@[ 6] 0x60876a6f nnotes._sqlpgcdl+927 (12bf44,12b9a4,0,12be44).

Date

Topic

Frequently asked question - How to analyze Notes/Domino NSDs? (Jean-Yves Riverin)

JYR

5

Subject: Fatal Thread

fatal thread 1/3:

############################################################

thread 1/3: [ nserver: 0f18: 0f1c]

FP=0x0006d118, PC=0x77609a94, SP=0x0006d0a8

stkbase=0x00070000, total stksize=49152, used stksize=12120

############################################################

[ 1] 0x77609a94 ntdll.KiFastSystemCallRet+0 (1ec,493e0,0,6d8c8)

[ 2] 0x76d6c1b2 kernel32.WaitForSingleObject+18 (1ec,493e0,6e05c,6dd12)

@[ 3] 0x601a2e75 nnotes.FRSendCommandToService+789 (6d8e8,6dce8,6dd12,3)

@[ 4] 0x601a3aaf nnotes.OSRunExternalScript@8+1055 (12c,1)

@[ 5] 0x601a402f nnotes.FRTerminateWindowsResources+975 (1,1000,1,0)

@[ 6] 0x601a43f8 nnotes.OSFaultCleanupExt@24+888 (1f46a14,1000,0,0,0,6e384)

@[ 7] 0x601a447a nnotes.OSFaultCleanup@12+26 (0,1000,0)

@[ 8] 0x601af924 nnotes.OSNTUnhandledExceptionFilter@4+276 (6ecc0)

[ 9] 0x76dbf4be kernel32.UnhandledExceptionFilter+309 (6ecc0,0,0,0)

@[10] 0x004174be nserver._XcptFilter+351 (0,6ecc0,414651,6ecc8)

@[11] 0x00411f29 nserver.WinMainCRTStartup+428 (7ffdd000,6ffd4,775ee4b6,7ffdd000)

[12] 0x76d64911 kernel32.BaseThreadInitThunk+18 (7ffdd000,776358d8,0,0)

[13] 0x775ee4b6 ntdll.RtlInitializeExceptionChain+99 (411d7d,7ffdd000,ffffffff,776575de)

[14] 0x775ee489 ntdll.RtlInitializeExceptionChain+54 (411d7d,7ffdd000,0,78746341)

INFO (0): Generating core dump for [ nserver: 0f18] (coreflags=0, exp=1c1ca30, dbgver=6.8.0004.0)

INFO (0): Generated core dump file C:\Program Files\IBM\Lotus\Domino\data\IBM_TECHNICAL_SUPPORT\core_nserver_W32I_TEST-SRVR01_2009_02_16@13_43_55.dmp

############################################################

FATAL THREAD 1/3 [ nserver: 0f18: 0f1c]

FP=0x0006f0a8, PC=0x775e962a, SP=0x0006f09c

stkbase=00070000, total stksize=49152, used stksize=3940

EAX=0x00000001, EBX=0x001b4140, ECX=0x800c6016, EDX=0x00002000

ESI=0x00000000, EDI=0x00000000, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010246

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

[ 1] 0x775e962a ntdll.RtlDestroyHeap+1009 (0,2,1b4070,1b37f0)

[ 2] 0x775e96a1 ntdll.RtlDestroyHeap+1128 (0,2,1,6f148)

[ 3] 0x775e96a1 ntdll.RtlDestroyHeap+1128 (0,2,77635684,1b08f8)

[ 4] 0x775e9723 ntdll.RtlDestroyHeap+1258 (73b50000,6f1ac,776356c0,1b08f8)

[ 5] 0x775ee8e2 ntdll.LdrUnloadDll+70 (73b50000,0,6f1f4,76e09524)

[ 6] 0x76d60979 kernel32.FreeLibrary+129 (73b50000,1b1588,76e094f1,1b1588)

[ 7] 0x76e09524 WS2_32.WSAEnumProtocolsW+691 (1,1b08f8,76e093e0,1b08f8)

[ 8] 0x76e094da WS2_32.WSAEnumProtocolsW+617 (1,199b70,76e09857,76d2e875)

[ 9] 0x76e09906 WS2_32.WahEnumerateHandleContexts+805 (1,202,0,196eb8)

[10] 0x76e097e1 WS2_32.WahEnumerateHandleContexts+512 (1,202,0,181940)

[11] 0x76e096fe WS2_32.WahEnumerateHandleContexts+285 (6197159d,6f2cc,166c0001,3cd0ef4)

@[12] 0x6197123f nNTCP.ProcessTerminateWinSock+79 (166c0001,202,0,0)

@[13] 0x60b7b462 nnotes.Drvr_Free@4+114 (166c0001)

@[14] 0x60b78ad6 nnotes.Process_FreeAllDrvrs@4+166 (11900001)

@[15] 0x60b78bad nnotes.Process_Free@4+13 (11900001)

@[16] 0x60b78de9 nnotes.Root_FreeProcess@4+89 (11900001)

@[17] 0x60b79235 nnotes.nti_cleanup@0+37 ()

@[18] 0x60789a50 nnotes.NetTerm@4+80 (0)

@[19] 0x6017305d nnotes.OSTerm@0+861 ()

@[20] 0x1000152c nserverl.ServerMain@16+924 (2,1a0fe14,0,0)

@[21] 0x004024f2 nserver.WinMain@16+674 (400000,0,3,a)

@[22] 0x00411f01 nserver.WinMainCRTStartup+388 (7ffdd000,6ffd4,775ee4b6,7ffdd000)

[23] 0x76d64911 kernel32.BaseThreadInitThunk+18 (7ffdd000,776358d8,0,0)

[24] 0x775ee4b6 ntdll.RtlInitializeExceptionChain+99 (411d7d,7ffdd000,ffffffff,776575de)

[25] 0x775ee489 ntdll.RtlInitializeExceptionChain+54 (411d7d,7ffdd000,0,78746341)

############################################################

PASS 2 : FATAL THREAD with STACK FRAMES 1/3 [ nserver: 0f18: 0f1c]

FP=0006f0a8, PC=775e962a, SP=0006f09c

stkbase=00070000, total stksize=49152, used stksize=3940

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

Disassembly of c. 10 instructions before and after faulting address 775e962a:



    775e9615 0bca             or      ecx,edx

    775e9617 894834           mov     [eax+0x34],ecx            ds:01c1ab17=00000000

    775e961a 53               push    ebx

    775e961b 8d5860           lea     ebx,[eax+0x60]            ds:01c1ab17=00000000

    775e961e 57               push    edi

    775e961f 8b3b             mov     edi,[ebx]                 ds:001b4140=001b37c0

    775e9621 3bfb             cmp     edi,ebx

    775e9623 742a             jz      775e964f

    775e9625 56               push    esi

    775e9626 83650800         and     dword ptr [ebp+0x8],0x0   ss:01c89bbe=00000000

FAULT ->775e962a 837f0c00 cmp dword ptr [edi+0xc],0x0 ds:01c1ab16=00000000

    775e962e 8b7708           mov     esi,[edi+0x8]             ds:01c1ab16=00000000

    775e9631 7615             jbe     775e9648

    775e9633 0fb74638         movzx   eax,word ptr [esi+0x38]       ds:01c1ab17=0000

    775e9637 6683f8ff         cmp     ax,0xff

    775e963b 7518             jnz     775e9655

    775e963d ff4508           inc     dword ptr [ebp+0x8]       ss:01c89bbe=00000000

    775e9640 8b4508           mov     eax,[ebp+0x8]             ss:01c89bbe=00000000

    775e9643 3b470c           cmp     eax,[edi+0xc]             ds:01c1ab16=00000000

    775e9646 72eb             jb      775e9733

    775e9648 8b3f             mov     edi,[edi]                 ds:00000000=????????

    775e964a 3bfb             cmp     edi,ebx

thread 2/3:

############################################################

thread 2/3: [ nserver: 0f18: 04d8]

FP=0x08f7fd60, PC=0x77609a94, SP=0x08f7fcf8

stkbase=0x08f80000, total stksize=262144, used stksize=776

############################################################

[ 1] 0x77609a94 ntdll.KiFastSystemCallRet+0 (64,0,8f7fd80,60090d9a)

[ 2] 0x76d21c7a kernel32.Sleep+15 (64,6,8f7fed4,10003697)

@[ 3] 0x60090d9a nnotes.OSDelayThread@4+42 (64)

@[ 4] 0x10003697 nserverl.ShutdownMonitorTask@8+231 (2,0)

@[ 5] 0x10001a2e nserverl.Scheduler@4+750 (0)

@[ 6] 0x600fe6ef nnotes.ThreadWrapper@4+175 (0)

[ 7] 0x76d64911 kernel32.BaseThreadInitThunk+18 (0,7f9258d8,0,0)

[ 8] 0x775ee4b6 ntdll.RtlInitializeExceptionChain+99 (600fe640,0,ffffffff,776575de)

[ 9] 0x775ee489 ntdll.RtlInitializeExceptionChain+54 (0,0,0,0)

thread 3/3:

############################################################

thread 3/3: [ nserver: 0f18: 0b28]

FP=0x0016ff88, PC=0x775f7dff, SP=0x0016ff5c

stkbase=0x00170000, total stksize=16384, used stksize=164

############################################################

[ 1] 0x775f7dff ntdll.DbgBreakPoint+1 (0,16ffd4,775ee4b6,0)

[ 2] 0x76d64911 kernel32.BaseThreadInitThunk+18 (0,777358d8,0,0)

[ 3] 0x775ee4b6 ntdll.RtlInitializeExceptionChain+99 (7763d094,0,ffffffff,776575de)

[ 4] 0x775ee489 ntdll.RtlInitializeExceptionChain+54 (0,0,0,0)

<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

Section: Notes Process Info → SYM Files (Time 13:44:02)

<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

SymFile Idx SymSrc #Lkup #Refs #Syms SYMFSize SysMod ImgTime SymTime Module

ntdll.dll — IMAGE 31 1 1824 1208320 Yes 01/19/2008@08:32:54 same C:\Windows\system32\ntdll.dll

kernel32.dll — IMAGE 12 1 1217 897024 Yes 01/19/2008@08:31:57 same C:\Windows\system32\kernel32.dll

nnotes.SYM — NDSYM 29 1 98751 5255212 No 12/29/2008@18:29:36 same C:\Program Files\IBM\Lotus\Domino\nnotes.dll

LotusDomino.sym 127 NDSYM 8 1 1697 11715595 No 12/06/2008@08:46:24 same C:\Program Files\IBM\Lotus\Domino\nserver.exe

WS2_32.dll — IMAGE 15 1 167 184320 Yes 01/19/2008@08:32:40 same C:\Windows\system32\WS2_32.dll

LotusDomino.sym 106 NDSYM 3 1 1231 11715595 No 12/06/2008@08:45:35 same C:\Program Files\IBM\Lotus\Domino\nNTCP.DLL

LotusDomino.sym 128 NDSYM 5 1 5994 11715595 No 12/06/2008@08:46:25 same C:\Program Files\IBM\Lotus\Domino\nserverl.dll

SymFile SymSrc #Syms #ImgSyms #Pools PSize #Alloc #PAlloc MemUsage

ntdll.dll IMAGE 1824 1824 16 120 1824 1920 86.3K

kernel32.dll IMAGE 1217 1217 16 80 1217 1280 57.5K

nnotes.SYM NDSYM 98751 108525 37 2688 98751 99456 4.4M

LotusDomino.sym NDSYM 1697 545810 2 1352 1697 2704 121.5K

WS2_32.dll IMAGE 167 500 6 32 167 192 8.6K

LotusDomino.sym NDSYM 1231 545810 1 1352 1231 1352 60.7K

LotusDomino.sym NDSYM 5994 545810 5 1352 5994 6760 303.7K

Totals ------ 110881 1749496 83 6976 110881 113664 5.0M

Total Syms = 110881, Total Alloc = 113664, overhead = 2783 (125.0K)