Server crashes with odd JVM error every day unless we restart server nightly

HCL Domino Domino Forum
1

We have an application that involves the running of Java agents via the Lotusscript agent.runonserver method. The agents are run a lot, and they do various things such as parse XML, consume Web Services, and modify Notes design elements. We get this error when we do not restart our server each evening:

JVMCL052: Cannot allocate memory in initializeHeap for heap segment

The documentation on this says:

The C-runtime heap of the process (not the Java object heap) is full. Increase the heap if that is possible in your environment.

Has anyone seen\solved this?

We are recycling everything.

The bulk of our code resides in jars on the server filesystem (because we have heard that importing it into agents or Libraries can cause memory issues).

JavaMaxHeapsize is at 256MB (though we have tried it at 128 and 384, and we still get the same crashes)

Any thoughts would be greatly appreciated.

2

Subject: Server crashes with odd JVM error every day unless we restart server nightly

Hi,

Post the FATAL THREAD section of the NSD files with the few lines under it

For example:

############################################################

FATAL THREAD 1/2 [ domdsm:107a8:67076]

FP=0x0012a34c, PC=0x60069a86, SP=0x0012a2f0, stksize=92

EAX=0x01a3088c, EBX=0x01a48650, ECX=0x0000bffe, EDX=0x01a325e6

ESI=0x00004000, EDI=0x0032ff78, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010206

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x60069a86 nnotes._fdDelete@4+118 (4000,12a368,608770ef,4000)

@[ 2] 0x60069a0c nnotes._OSFileClose@4+12 (4000,12adc0,12a380,60872ea7)

@[ 3] 0x608770ef nnotes._sqloclose+15 (4000,12b59c,5010,0)

@[ 4] 0x60872ea7 nnotes._sqlpgclf+71 (12adc0,1aaa2cb,70,1aaa2f3)

@[ 5] 0x60885458 nnotes._sqlpgfdl+616 (12b9a4,1aaa2cb,41,12be44)

@[ 6] 0x60876a6f nnotes._sqlpgcdl+927 (12bf44,12b9a4,0,12be44).

Frequently asked question - How to analyze Notes/Domino NSDs?

http://www-10.lotus.com/ldd/nd6forum.nsf/7e6c7e584a76331b85256a46006f083e/92d3876c4e4f140a852572810008ea9d?OpenDocument

3

Subject: RE: Server crashes with odd JVM error every day unless we restart server nightly

Thanks.

############################################################

FATAL THREAD 1/2 [nprocmon: 046c: 0ca8]

FP=0x0012fc3c, PC=0x60176b01, SP=0x0012f4c8

stkbase=00130000, total stksize=20480, used stksize=2872

EAX=0x00f30648, EBX=0x77e6ba30, ECX=0x008f0000, EDX=0x008f0000

ESI=0x0012fc54, EDI=0x7c82f4c3, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010206

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x60176b01 nnotes.Panic@4+417 (12fc54,2,7ffdd000,396594)

[ 2] 0x0040110a nprocmon (400000,1,400000,0)

[ 3] 0x0040114f nprocmon (2,400000,0,ece)

[ 4] 0x00401284 nprocmon (0,0,0,2)

[ 5] 0x00401176 nprocmon (2,3f2a28,3f2a60,94)

[ 6] 0x0040150b nprocmon (0,0,7ffdd000,0)

[ 7] 0x77e523e5 KERNEL32.IsProcessorFeaturePresent+158 (40139f,0,78746341,20)

Invalid stack frame detected: Invalid frame pointer (BP): 0

############################################################

PASS 2 : FATAL THREAD with STACK FRAMES 1/2 [nprocmon: 046c: 0ca8]

FP=0012fc3c, PC=60176b01, SP=0012f4c8

stkbase=00130000, total stksize=20480, used stksize=2872

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

Disassembly of c. 10 instructions before and after faulting address 60176b01:



    60176ada 56               push    esi

    60176adb 0506520400       add     eax,0x45206

    60176ae0 68f417ae60       push    0x60ae17f4

    60176ae5 50               push    eax

    60176ae6 e8e5d8e9ff       call    600143d0

    60176aeb 83c40c           add     esp,0xc

    60176aee e8fd3c0000       call    6017a7f0

    60176af3 a1cc0ed260       mov     eax,[60d20ecc]            ds:60d20ecc=00f30648

    60176af8 8388b614000002   or     dword ptr [eax+0x14b6],0x2 ds:00f31afe=0000000a

    60176aff 6a01             push    0x1

FAULT ->60176b01 c605ffffffff00 mov byte ptr [ffffffff],0x0 ds:ffffffff=??

    60176b08 ff154403ae60     call    dword ptr [60ae0344]      ds:60ae0344=77e53051

    60176b0e cc               int     3

    60176b0f cc               int     3

    60176b10 55               push    ebp

    60176b11 8bec             mov     ebp,esp

    60176b13 8b4508           mov     eax,[ebp+0x8]             ss:03c1c812=00000000

    60176b16 8b0d445cdd60     mov     ecx,[60dd5c44]            ds:60dd5c44=008f0000

    60176b1c 6800010000       push    0x100

    60176b21 50               push    eax

    60176b22 81c19d5a0400     add     ecx,0x45a9d

    60176b28 51               push    ecx

---------- Dump of stack from SP to FP ----------

        0012f4c8 01000000 CC030000 0A0A2054 68726561 | .... .... .. T hrea |

        0012f4d8 643D5B30 3436433A 30303032 2D304341 | d=[0 46C: 0002 -0CA |
4

Subject: RE: Server crashes with odd JVM error every day unless we restart server nightly

Hi,

The process is nprocmon . But as I can see, it doesn’t seems to be only related to Notes. Because, you have stacks that don’t have the @ in front of it.

@[ 1] 0x60176b01 nnotes.Panic@4+417 (12fc54,2,7ffdd000,396594)

[ 2] 0x0040110a nprocmon (400000,1,400000,0)

[ 3] 0x0040114f nprocmon (2,400000,0,ece)

[ 4] 0x00401284 nprocmon (0,0,0,2)

[ 5] 0x00401176 nprocmon (2,3f2a28,3f2a60,94)

[ 6] 0x0040150b nprocmon (0,0,7ffdd000,0)

[ 7] 0x77e523e5 KERNEL32.IsProcessorFeaturePresent+158 (40139f,0,78746341,20)

You could try this

http://www-12.lotus.com/ldd/doc/domino_notes/7.0/help7_admin.nsf/f4b82fbb75e942a6852566ac0037f284/5fde2264c59bb7a08525706f00662421?OpenDocument

Monitoring processes running in the Domino server environment
Domino’s “Process Monitor” task applies to Domino on Microsoft Windows platforms only. This task monitors the processes that should be running as part of the Domino server environment. If one of these processes is missing, or if one terminates unexpectedly without completing the usual Domino termination routines, this task causes the server to Panic and indicates the process that has prematurely terminated. The Process Monitor task works with nprocmon.exe, which monitors the nserver.exe process for abnormal termination.

This functionality is implemented in Domino for Unix platforms without using a separate server task.

To disable the Process Monitor task, set the variable PROCESS_MONITOR_DISABLED=1 in the server’s NOTES.INI file.

JYR

5

Subject: RE: Server crashes with odd JVM error every day unless we restart server nightly

Maybe related,

http://www-10.lotus.com/ldd/nd6forum.nsf/DateAllFlatweb/d9e08f3130ead7ff85257380000a6030?OpenDocument

JYR

6

Subject: RE: Server crashes with odd JVM error every day unless we restart server nightly

Thanks, but I’m not using those logging classes.

7

Subject: RE: Server crashes with odd JVM error every day unless we restart server nightly

After we updated to LN 7.03, problem went away!!!