Panic. Urgent!

hcl-bot · May 27, 2010, 2:07am

Please help.

Crash Domino server. Versin 8.5.1 (64 bit) on Windows server 2008 R2, 64 bit.

Part of NSD:

Host Name : SERVER

User Name : SYSTEM

Date : Thu May 27 08:47:54 2010

Windows Dir : C:\Windows

Arguments : “D:\lotus\Domino\nsd.exe” -dumpandkill -termstatus 1 -panicdirect -crashpid 7256 -crashtid 7260 -runtime 600 -ini “D:\lotus\Domino\notes.ini” -svcreq 128

NSD Version : 8.5.10.9243 (Release 8.5.1)

OS Version : Windows/2008 R2 6.1 [64-bit] (Build 7600), PlatID=2, (8 Processors)

Build time : Tue Sep 29 02:03:55 2009

Latest file mod : Tue Sep 01 22:08:33 2009

Domino Version : Release 8.5.1 (64-bit server)

<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

Section: System Data → OS Process Table (Time 08:47:56)

<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

<@@ ------ System Data → Active Users (Time 08:47:56) ------ @@>

UID Name

0  SYSTEM

500 Administrator

1131 McAfeePostgresUser

<@@ ------ System Data → Processes (Time 08:47:56) ------ @@>

INFO PID PPID UID STIME COMMAND

    0000 0000     0            ??? [[System Process]:  0000]

    012c 0004     0 05/27 08:39:45 [\SystemRoot\System32\smss.exe:  012c]

    01b0 01a8     0 05/27 08:39:50 [C:\Windows\system32\csrss: %SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,768 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllIn...:  01b0]

    0210 01a8     0 05/27 08:39:51 [C:\Windows\system32\winlogon: winlogon.exe:  0210]

    03d0 0210     0 05/27 08:39:58   [C:\Windows\system32\LogonUI: "LogonUI.exe" /flags:0x0:  03d0]

    0894 087c  1131 05/27 08:40:33 [C:\Program Files (x86)\McAfee\Security for Lotus Domino\PostgreSQL\bin\postmaster: "C:/Program Files (x86)/McAfee/Security for Lotus Domino/PostgreSQL/bin/postmaster.exe"  -D "C:/Program Files (x86)/McAfee/Security for Lotus Domino/Data/MSDWDat...:  0894]

    0970 0894  1131 05/27 08:40:35   [C:\Program Files (x86)\McAfee\Security for Lotus Domino\PostgreSQL\bin\postgres: "C:/Program Files (x86)/McAfee/Security for Lotus Domino/PostgreSQL/bin/postgres.exe" "-forkboot" "4864" "-x3" "-p" "template1":  0970]

    097c 0894  1131 05/27 08:40:35   [C:\Program Files (x86)\McAfee\Security for Lotus Domino\PostgreSQL\bin\postgres: "C:/Program Files (x86)/McAfee/Security for Lotus Domino/PostgreSQL/bin/postgres.exe" "-forkbuf" "4848" "C:/Program Files (x86)/McAfee/Security for Lotus Domino/Po...:  097c]

    0990 097c  1131 05/27 08:40:35     [C:\Program Files (x86)\McAfee\Security for Lotus Domino\PostgreSQL\bin\postgres: "C:/Program Files (x86)/McAfee/Security for Lotus Domino/PostgreSQL/bin/postgres.exe" "-forkcol" "384" "C:/Program Files (x86)/McAfee/Security for Lotus Domino/Pos...:  0990]

    22343800001D80 0894  1131 05/27 08:41:20   [C:\Program Files (x86)\McAfee\Security for Lotus Domino\PostgreSQL\bin\postgres: "C:/Program Files (x86)/McAfee/Security for Lotus Domino/PostgreSQL/bin/postgres.exe" "-forkbackend" "4868":  1d80]

    726F6600001250 121c     0 05/27 08:40:45 [C:\hp\hpsmh\bin\rotatelogs.exe  -f C:/hp/hpsmh/logs/error_log 5M:  1250]

 -> 726F6600001A38 1770     0 05/27 08:41:10 [D:\lotus\Domino\nevent.EXE:  1a38]

 -> 726F6600000138 1770     0 05/27 08:41:12 [D:\lotus\Domino\nReplica.EXE :  0138]

 -> 726F6600000244 1770     0 05/27 08:41:12 [D:\lotus\Domino\nRouter.EXE :  0244]

 -> 726F66000018F4 1770     0 05/27 08:41:12 [D:\lotus\Domino\nUpdate.EXE :  18f4]

 -> 726F660000190C 1770     0 05/27 08:41:12 [D:\lotus\Domino\nAMgr.EXE -s:  190c]

 -> 1cd8 190c     0 05/27 08:41:16   [D:\lotus\Domino\namgr.EXE -e 1:  1cd8]

 -> 726F660000191C 1770     0 05/27 08:41:12 [D:\lotus\Domino\nSched.EXE :  191c]

 -> 726F6600001798 1770     0 05/27 08:41:12 [D:\lotus\Domino\nCalConn.EXE :  1798]

 -> 726F6600001974 1770     0 05/27 08:41:13 [D:\lotus\Domino\nRnRMgr.EXE :  1974]

 -> 726F6600000150 1770     0 05/27 08:41:13 [D:\lotus\Domino\nIMAP.EXE :  0150]

 -> 726F660000118C 1770     0 05/27 08:41:13 [D:\lotus\Domino\nPOP3.EXE :  118c]

 -> 726F660000000C 1770     0 05/27 08:41:13 [D:\lotus\Domino\nhttp.EXE :  000c]

 -> 726F6600001A24 1770     0 05/27 08:41:13 [D:\lotus\Domino\nSMTP.EXE :  1a24]

 -> 726F6600001C58 1770     0 05/27 08:41:14 [D:\lotus\Domino\nprocmon.EXE 6000:  1c58]

    25a8 1c58     0 05/27 08:47:54   ["D:\lotus\Domino\nsd.exe" -svcddlg -svcreq 128 -ini "D:\lotus\Domino\notes.ini":  25a8]

    F6E696D00002310 2308     0 05/27 08:41:39 [C:\Windows\system32\csrss: %SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,768 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllIn...:  2310]

    F6E696D00002328 2308     0 05/27 08:41:39 [C:\Windows\system32\winlogon: winlogon.exe:  2328]

    F6E696D000023A0 224c   500 05/27 08:41:52 [C:\Windows\Explorer.EXE:  23a0]

    27b8 23a0   500 05/27 08:43:12   ["D:\lotus\Domino\jconsole.exe" "=D:\lotus\Domino\data\Dconsole.ini":  27b8]

    C3A44220000227C 2188   500 05/27 08:42:01 ["C:\Program Files (x86)\McAfee\VirusScan Enterprise\shstat.exe" /STANDALONE:  227c]

    C3A442200002244 2188   500 05/27 08:42:02 ["C:\Program Files\Eaton\IntelligentPowerManager\mc2.exe" -systray:  2244]

    C3A442200000F04 2188   500 05/27 08:42:02 ["C:\Program Files (x86)\Common Files\McAfee\TalkBack\TBMon.exe" :  0f04]

    C3A442200002640 2618   500 05/27 08:42:49 [C:\Program Files (x86)\IBM\Lotus\Notes\NLNOTES: NLNOTES.EXE admin "=C:\Program Files (x86)\IBM\Lotus\Notes\notes.ini":  2640]

    26f0 2640   500 05/27 08:43:02   ["C:\Program Files (x86)\IBM\Lotus\Notes\nEvent.EXE" -D:  26f0]

    2788 2640   500 05/27 08:43:07   ["C:\Program Files (x86)\IBM\Lotus\Notes\ntaskldr.EXE" C:\Program Files (x86)\IBM\Lotus\Notes\ntaskldr.EXE:  2788]

<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

Section: System Data → Notes Process Summary (Time 08:47:56)

<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

<@@ ------ Instances → Instance1 (current) (Time 08:47:56) ------ @@>

            Ini File   : D:\lotus\Domino\notes.ini

            Data Dir   : D:\Lotus\Domino\data

            Prog Dir   : D:\lotus\Domino

            Notes Build: Release 8.5.1 September 28, 2009             

        Proc List:

              D:\lotus\Domino\nevent.EXE (1a38)

              D:\lotus\Domino\nReplica.EXE (138)

              D:\lotus\Domino\nRouter.EXE (244)

              D:\lotus\Domino\nUpdate.EXE (18f4)

              D:\lotus\Domino\nAMgr.EXE (190c)

              D:\lotus\Domino\nSched.EXE (191c)

              D:\lotus\Domino\nCalConn.EXE (1798)

              D:\lotus\Domino\nRnRMgr.EXE (1974)

              D:\lotus\Domino\nIMAP.EXE (150)

              D:\lotus\Domino\nPOP3.EXE (118c)

              D:\lotus\Domino\nhttp.EXE (c)

              D:\lotus\Domino\nSMTP.EXE (1a24)

              D:\lotus\Domino\nprocmon.EXE (1c58)

              D:\lotus\Domino\nAMgr.EXE (1cd8)

<@@ ------ Notes Processes Not Associated With An Instance (Time 08:47:56) ------ @@>

D:\lotus\Domino\nservice.exe (468)

<@@ ------ System Data → Memory Usage (Time 08:47:56) ------ @@>

Total Physical Memory: 6.0G

Avail Physical Memory: 3.7G

Memory Usage : 37%

Total Paging File : 12.0G

Avail Paging File : 8.9G

Total Virtual Memory: 8.0T

Avail Virtual Memory: 8.0T

Avail Extended Virtual Memory: 0.0K

<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

Section: Notes Process Info (Time 08:47:56)

<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

<@@ ------ Notes Process Info → Load Modules for Process :: [nprocmon: 1c58] (Time 08:47:56) ------ @@>

Base End Size Module

0x00400000 0x00412000 73728 D:\lotus\Domino\nprocmon.EXE

0x77ab0000 0x77c5b000 1748992 C:\Windows\SYSTEM32\ntdll.dll

0x77890000 0x779af000 1175552 C:\Windows\system32\kernel32.dll

0x000007FEFDD30000 0x000007FEFDD9B000 438272 C:\Windows\system32\KERNELBASE.dll

0x10000000 0x134a0000 55181312 D:\lotus\Domino\nnotes.dll

0x000007FEFDB20000 0x000007FEFDC86000 1466368 C:\Windows\system32\CRYPT32.dll

0x000007FEFFBF0000 0x000007FEFFC8F000 651264 C:\Windows\system32\msvcrt.dll

0x000007FEFDAB0000 0x000007FEFDABF000 61440 C:\Windows\system32\MSASN1.dll

0x00420000 0x00b11000 7278592 D:\lotus\Domino\nxmlproc.dll

0x000007FEF3040000 0x000007FEF30D6000 614400 C:\Windows\system32\msvcp60.dll

0x00b20000 0x00ea5000 3690496 D:\lotus\Domino\NLSCCSTR.DLL

0x000007FEFE7D0000 0x000007FEFE8AB000 897024 C:\Windows\system32\ADVAPI32.dll

0x000007FEFE150000 0x000007FEFE16F000 126976 C:\Windows\SYSTEM32\sechost.dll

0x000007FEFE020000 0x000007FEFE14E000 1236992 C:\Windows\system32\RPCRT4.dll

0x000007FEFEB40000 0x000007FEFEC17000 880640 C:\Windows\system32\OLEAUT32.dll

0x000007FEFDE10000 0x000007FEFE011000 2101248 C:\Windows\system32\ole32.dll

0x000007FEFE3D0000 0x000007FEFE437000 421888 C:\Windows\system32\GDI32.dll

0x779b0000 0x77aaa000 1024000 C:\Windows\system32\USER32.dll

0x000007FEFFDB0000 0x000007FEFFDBE000 57344 C:\Windows\system32\LPK.dll

0x000007FEFFC90000 0x000007FEFFD5A000 827392 C:\Windows\system32\USP10.dll

0x000007FEFFD60000 0x000007FEFFDAD000 315392 C:\Windows\system32\WS2_32.dll

0x000007FEFDE00000 0x000007FEFDE08000 32768 C:\Windows\system32\NSI.dll

0x000007FEFC4B0000 0x000007FEFC4CE000 122880 C:\Windows\system32\USERENV.dll

0x000007FEFDA10000 0x000007FEFDA1F000 61440 C:\Windows\system32\profapi.dll

0x001d0000 0x001e0000 65536 D:\lotus\Domino\ndgts.dll

0x000007FEFE7A0000 0x000007FEFE7CE000 188416 C:\Windows\system32\IMM32.DLL

0x000007FEFE8B0000 0x000007FEFE9B9000 1085440 C:\Windows\system32\MSCTF.dll

0x0dbc0000 0x0dcdd000 1167360 D:\lotus\Domino\NSTRINGS.DLL

0x00220000 0x00230000 65536 D:\lotus\Domino\namhook.DLL

0x00230000 0x0024a000 106496 D:\lotus\Domino\McAfeeEMX64.DLL

0x755c0000 0x756c9000 1085440 C:\Windows\WinSxS\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4927_none_88dce9872fb18caf\MSVCP80.dll

0x756f0000 0x757b9000 823296 C:\Windows\WinSxS\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.4927_none_88dce9872fb18caf\MSVCR80.dll

0x64000000 0x640ad000 708608 C:\Program Files (x86)\McAfee\Security for Lotus Domino\bin\CommonX64.dll

0x000007FEFA7C0000 0x000007FEFA7D6000 90112 C:\Windows\system32\netapi32.dll

0x000007FEFD400000 0x000007FEFD40C000 49152 C:\Windows\system32\netutils.dll

0x000007FEFD580000 0x000007FEFD5A3000 143360 C:\Windows\system32\srvcli.dll

0x000007FEFA730000 0x000007FEFA745000 86016 C:\Windows\system32\wkscli.dll

0x000007FEFA490000 0x000007FEFA4A4000 81920 C:\Windows\system32\SAMCLI.DLL

0x00270000 0x0027f000 61440 C:\Program Files (x86)\McAfee\Security for Lotus Domino\bin\boost_thread-vc80-mt-1_64.dll

0x0ede0000 0x0f07a000 2727936 C:\Program Files (x86)\McAfee\Security for Lotus Domino\bin\xerces-c_2_vc80_X64.dll

0x000007FEFEDC0000 0x000007FEFFB46000 14180352 C:\Windows\system32\SHELL32.dll

0x000007FEFECA0000 0x000007FEFED11000 462848 C:\Windows\system32\SHLWAPI.dll

0x002b0000 0x002cd000 118784 C:\Program Files (x86)\McAfee\Security for Lotus Domino\bin\CommonFWX64.dll

0x154a0000 0x169df000 22278144 D:\lotus\Domino\nsdhelp.dll

0x000007FEFC660000 0x000007FEFC66C000 49152 C:\Windows\system32\VERSION.dll

0x000007FEFAE30000 0x000007FEFAE48000 98304 C:\Windows\system32\MPR.dll

0x000007FEFD8A0000 0x000007FEFD8C5000 151552 C:\Windows\system32\SspiCli.dll

0x000007FEFD990000 0x000007FEFD9E7000 356352 C:\Windows\system32\apphelp.dll

<@@ ------ Notes Process Info → Call Stack for Process :: [nprocmon: 1c58] (Time 08:47:56) ------ @@>

############################################################

thread 1/3: [nprocmon: 1c58: 1c5c] FATAL THREAD (Panic)

FP=0x001271a8, PC=0x77affefa, SP=0x001271a8

stkbase=0x00130000, total stksize=53248, used stksize=36440

EAX=0x00000041, EBX=0x00000000, ECX=0x00126192, EDX=0x7FEFD89D4D7

ESI=0x000927c0, EDI=0x00000218, CS=0x00000033, SS=0x7FE0000002B

DS=0x00000000, ES=0x00000000, FS=0x00000000, GS=0x7FE00000000 Flags=0x00000287

############################################################

[ 1] 0x77affefa ntdll.ZwWaitForSingleObject+10 (FD679775946F,11fe9be9,3,131112a0)

[ 2] 0x7FEFDD310AC KERNELBASE.WaitForSingleObjectEx+156 (0,0,0,218)

@[ 3] 0x10089987 nnotes.FRSendCommandToService+1931 (12c810,12cb28,0,10086284)

@[ 4] 0x1008d190 nnotes.OSRunExternalScript+5824 (365636F00000003,0,12060124,3)

@[ 5] 0x100870b1 nnotes.FRTerminateWindowsResources+2277 (80,80,0,300230)

@[ 6] 0x1008fe36 nnotes.OSFaultCleanupExt+622 (0,1008644b,150014,12063c20)

@[ 7] 0x10090849 nnotes.OSFaultCleanup+29 (80,10052d1c,0,d21f4e0)

@[ 8] 0x100a8ee5 nnotes.OSNTUnhandledExceptionFilter+677 (12faa0,12047a30,12fb08,4d)

@[ 9] 0x1001f0ce nnotes.Panic+682 (409502,409600,17da,ff)

[10] 0x00401172 nprocmon+4466 (2,101224c9,0,0)

[11] 0x0040123e nprocmon+4670 (302030203020302,302030203020302,0,0)

[12] 0x00401345 nprocmon+4933 (3,0,0,0)

[13] 0x00401968 nprocmon+6504 (0,0,0,0)

[14] 0x778af56d kernel32.BaseThreadInitThunk+13 (0,0,0,0)

[15] 0x77ae3281 ntdll.RtlUserThreadStart+33 (0,0,0,0)

############################################################

PASS 2 : FATAL THREAD (Panic) with STACK FRAMES [nprocmon: 1c58: 1c5c]

Koos van Harmelen

hcl-bot · May 27, 2010, 11:46am

Subject: FPs

Some things to do/try:

Install the latest FP (fix packs). Please install FP2 and the IF1 for FP2.
What else is running on the server?

hcl-bot · May 28, 2010, 1:28am

Subject: Tried to install FP2

I already tried to install FP2 but Domino did not start at all. So I uninstalled Domino software, reinstalled, installed FP2, no success. Again reinstalled, without FP2, running oke.Opened Service request at IBM. Answer: Sorry, can’t help. Windows 2008 R2 64 nit is not supported. Great Windows or great IBM?

hcl-bot · May 28, 2010, 11:01am

Subject: No FPs?

Do you’re running 8.5.1 without FP1 or FP2? If it runs well I’d wait until IBM confirms support for 2008 R2 so they’ll support you. Are you using DAOS? FP1/FP2/IF1 address many issues with DAOS so I’d leave that feature disabled until you’re able to install the FPs/IF.

hcl-bot · May 31, 2010, 1:39am

Subject: No FPs

Yes, I was running 8.5.1 without any FP. Because of a crash (Domino did not start at all anymore)I tried to install FP2, w/o success. I did not realise Windows 2008 R2 was not yet supported. So I reinstalled 8.5.1 and I already decided to wait for 8.5.2 before upgrading. I my post may 28 I asked for start of support for 2008 R2, possible in 8.5.2.No DAOS.

Koos

hcl-bot · August 10, 2010, 11:46pm

Subject: 8.5.1 FP1

Even I am also facing similar problem with Domino 8.5.1 FP1.For me the crash happens throwing this error:FATAL Thread (panic) with parameter data [nHTTP :1180 : 0a6c].I have changed the HTTPJVMMaxHeapSize=256M in the notes.ini to 64M and it still did not make a difference

hcl-bot · November 7, 2010, 3:24pm

Subject: McAfee ?!?

I have seen similar issues on a customer environment. It seamed that the McAfee is crashing the server. I can see from your NSD that you have added EXTMGR_ADDINS=McAfeeEMX64 to the notes.ini.

The customer removed that line from the notes.ini and the server is starting then without any problems. BUT McAfee is then no longer working … So its not the fault from the OS or from IBM, its more a McAfee issue I think.

Can you check if your server is starting if you remove the line from the notes.ini entry? In that case you need to check with McAfee why there software isn´t running correctly on a domino server with fixpack.