Out of Backend Memory

HCL Domino Domino Forum
1

Hello all

We are having a serious problem with a Domino Server (8.5.2 FP1/Windows 2003 Server 32bits). After several hours of being running, it ends with a crash like this:

LSXBE: ************************************

LSXBE: ****** Out of Backend Memory *******

LSXBE: ************************************

The server is working mainly as a mail and web server (with XPages).

Also I would like to note some INI settings that may be related with this:

HTTPJVMMaxHeapSize=256M

HTTPJVMMaxHeapSizeSet=1

Finally here is some fragments from the NSD file:

NSD Version : 8.5.21.0293 (Release 8.5.2FP1)

OS Version : Windows/2003 R2 5.2 [32-bit] (Build 3790), PlatID=2, Service Pack 2 (4 Processors)

Build time : Tue Nov 30 03:20:27 2010

Latest file mod : Thu Oct 21 20:25:17 2010

Domino Version : Release 8.5.2FP1 (32-bit server)

Keyview Version : 10.8.0.0

<@@ ------ Instances → Instance1 (current) (Time 14:16:01) ------ @@>

            Ini File   : C:\Lotus\Domino\notes.ini

            Data Dir   : C:\Lotus\Domino\Data

            Prog Dir   : C:\Lotus\Domino

            Notes Build: Release 8.5.2FP1 November 29, 2010        

        Proc List:

              C:\Lotus\Domino\nserver.exe (ac4)

              C:\Lotus\Domino\nupdate.EXE (1568)

              C:\Lotus\Domino\nevent.EXE (154c)

              C:\Lotus\Domino\nReplica.EXE (163c)

              C:\Lotus\Domino\nRouter.EXE (1480)

              C:\Lotus\Domino\nupdate.EXE (458)

              C:\Lotus\Domino\nAMgr.EXE (15f0)

              C:\Lotus\Domino\nAdminp.EXE (3e0)

              C:\Lotus\Domino\nSched.EXE (5ec)

              C:\Lotus\Domino\nCalConn.EXE (1628)

              C:\Lotus\Domino\nRnRMgr.EXE (c7c)

              C:\Lotus\Domino\nPOP3.EXE (728)

              C:\Lotus\Domino\nSMTP.EXE (1644)

              C:\Lotus\Domino\nHTTP.EXE (2ac)

              C:\Lotus\Domino\ndomidx.EXE (16ac)

              C:\Lotus\Domino\nprocmon.EXE (1620)

              C:\Lotus\Domino\nAMgr.EXE (cd4)

              C:\Lotus\Domino\nAMgr.EXE (398)

              C:\Lotus\Domino\nAMgr.EXE (f2c)

              C:\Lotus\Domino\nAMgr.EXE (1414)

<@@ ------ System Data → Memory Usage (Time 14:16:01) ------ @@>

Total Physical Memory: 4.0G

Avail Physical Memory: 2.5G

Memory Usage : 36%

Total Paging File : 7.8G

Avail Paging File : 6.2G

Total Virtual Memory: 2.0G

Avail Virtual Memory: 2.0G

Avail Extended Virtual Memory: 0.0K

############################################################

thread 18/62: [ nHTTP: 02ac: 1588]

FP=0x0ab7b514, PC=0x7c82860c, SP=0x0ab7b4a4

stkbase=0x0ab80000, total stksize=262144, used stksize=19292

############################################################

[ 1] 0x7c82860c ntdll.KiFastSystemCallRet+0 (e98,493e0,0,ab7ba9c)

[ 2] 0x77e31c8d kernel32.WaitForSingleObject+18 (e98,493e0,0,ab7bcb8)

@[ 3] 0x6020d5ed nnotes.OSRunExternalScript@8+1309 (12c,1)

@[ 4] 0x6020da6f nnotes.FRTerminateWindowsResources+975 (1,1000,1,0)

@[ 5] 0x6020de98 nnotes.OSFaultCleanupExt@24+984 (f76a68,1000,0,0,0,ab7bfe0)

@[ 6] 0x6020df1a nnotes.OSFaultCleanup@12+26 (0,1000,0)

@[ 7] 0x602197d4 nnotes.OSNTUnhandledExceptionFilter@4+276 (ab7caf0)

[ 8] 0x095838a2 ndomws.dwsSetCallBacks+141330 (ab7caf0,77e31ac1,ab7caf8,0)

[ 9] 0x77e492a3 kernel32.QueryMemoryResourceNotification+4808 (0,0,0,0)

############################################################

FATAL THREAD 18/62 [ nHTTP: 02ac: 1588]

FP=0x0ab7ced0, PC=0x60001716, SP=0x0ab7cec4

stkbase=0ab80000, total stksize=262144, used stksize=12604

EAX=0x00001588, EBX=0x00001588, ECX=0x00000000, EDX=0x09a568a4

ESI=0x0000019a, EDI=0x000002ac, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010206

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x60001716 nnotes.OSLockWriteSem@4+22 (19a)

@[ 2] 0x61cf4a78 nlsxbe.ANNote::ANNAddToCollList+56 (0)

@[ 3] 0x61d26137 nlsxbe.ANDocColl::ANDCNavigate+535 (0)

@[ 4] 0x61d2703f nlsxbe.Java_lotus_domino_local_DocumentCollection_NgetNextDocument@12+223 (7193800,afce860,afce84c)

############################################################

PASS 2 : FATAL THREAD with STACK FRAMES 18/62 [ nHTTP: 02ac: 1588]

FP=0ab7ced0, PC=60001716, SP=0ab7cec4

stkbase=0ab80000, total stksize=262144, used stksize=12604

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

Any ideas I could try to fix this nightmare.

Thanks in advance.

2

Subject: Common issue.

This part of the stack.

@[ 4] 0x61d2703f nlsxbe.Java_lotus_domino_local_DocumentCollection_NgetNextDocument

You are iterating through a document collection. It is very important that you recycle the document objects as you go, or you will run out of memory.

That is the most common reason for this stack line.

3

Subject: Thanks for the info

Yes, we use recycle. But I think that you’re in the right track for this… I’ll do a sweep in the code and see if we are missing some recycle in any loop.

Thanks for the reply

4

Subject: Open a PMR with IBM - I’m actually working a similar issue now.

5

Subject: Out of Backend Memory

Yes, open a pmr. Unfortunately, disabling the jvm is not an option because http would not function without it.

It could be a few different things that could cause that error, so more in-depth troubleshooting is required.

6

Subject: Re: Out of Backend Memory

I had terrible problems with that in the 8.0.2 era. I hate to think it is coming back.

I wonder if there was some regression in the following SPRS?

MSER7GCS7Z (Fixed in 8.5-the same msg you got)

JPAI755QCF (Fixed in 8.5-an out of memory condition)

7

Subject: Re: Out of Backend Memory-2

Also: it seems there may have been some issues running Java on the server. If possible, consider disabling and “runjava” tasks.

Note: the problems I had were on a Solaris server, so YMMV.

8

Subject: Thanks

Thanks all for your quick responses.

I’ve set HTTPJVMMaxHeapSize back to 64M in the NOTES.INI, but same problem rises again so it seems no dependant on the HTTPJVMMaxHeapSize.

I also tried running Domino as a service/running as a program with same results.

Sometimes the crash takes 2-3 hours to appear, sometimes takes more than 6 hours.

I’ve run out of ideas to try here…