Insufficient Memory Errors in Version 6 causing major issues

HCL Domino Domino Forum
1

I’ve search the forum a lot over the past few weeks and have been in contact with IBM on two different occasions but have not found anything that could help me with this issue so I created a login and put this out there for all to see hoping that someone has experienced this and possibly have figured out the cause. I hope someone can help because we’re really struggling with this issue and we’re usually a very self-sufficient crew.

[Problem and a little History]

We are currently running 6.51 Domino server on a single AS/400 server running v5R2. We have about 250 clients most of whom are running 6.02. About 10 remote users who hav not run the upgrade and are still running 5.11…every company has “those guys”

We upgraded to 6.02 in December 2003 and began experiencing problems when Domino would restart after backups ran. It seemed the server would begin to hit “Insufficient Memory” errors. At first the error would appear sporadically in the logs but they would increase in frequency until everything would come to a screeching hault. We contacted Lotus/IBM technical support and quickly upgraded to 6.51 after they recommended we do so. But this past weekend after restarting the Domino it started again. Here are a few lines out of the log file that demonstrate what we’re experiencing. It doesn’t seem to care what the file is so I’ve made all the actual file names generic.

[…it begins…]

04/04/2004 03:03:21 Updated database filename.nsf in catalog

04/04/2004 03:03:21 Unable to update entry in catalog (for filename.nsf): Insufficient memory.

[…then a little later…]

04/04/2004 04:30:43 Document NT00001716 in database /notes/domino01/data/HELP/help5_designer.nsf is damaged: Document has invalid structure

04/04/2004 04:30:43 Document (UNID OF035DAE8D:D990C466-ON8525687E:00587710) in database /notes/domino01/data/HELP/help5_designer.nsf has been deleted

[…still later…]

04/04/2004 04:31:16 NIF: DETECTED STORAGE CORRUPTION ERROR ‘B-tree structure is invalid’

04/04/2004 04:31:17 Error fixing view 4910 in filename.nsf: Insufficient memory.

04/04/2004 04:31:17 Error fixing view 314 in filename.nsf: Insufficient memory.

[…still later…]

04/04/2004 04:40:58 Completed consistency check on Address Book/filename.nsf

04/04/2004 04:40:58 Checking views in database Address Book/filename.nsf

04/04/2004 04:40:58 Error fixing view 486 in Address Book/filename.nsf: Invalid or nonexistent document

04/04/2004 04:40:58 Error fixing view 498 in Address Book/filename.nsf: Invalid or nonexistent document

[…until finally it all come to a screeching hault after about two full log pages of…]

04/04/2004 04:58:52 Insufficient memory.

04/04/2004 04:58:52 Insufficient memory.

04/04/2004 04:58:52 Insufficient memory.

By doing a full IPL on the 400 the problem will be fixed but in it’s wake is left a ton of damaged databases. “Invalid or non-existent documents” and missing design elements such as views, navigators, and forms. Mail files seem to be very vulnerable. This is really killing us and if you have any hints, suggestions, or thoughts, we’d be much appreciated.

Oh, and if you made it this far…thank you for reading this long winded message and I’m sorry it’s so long.

2

Subject: Insufficient Memory Errors in Version 6 causing major issues

We have been having the same kind of problems. We are running 652 on linux es3. All our users are web based and there has been a steady growth of concurrent users on the system.The one problem we have is to determine if we are running out of memory in the JVM or if Domino it self is running out of memory. Figures we get back from the system when we encounter the problem show that domino has more than enough memory, so we are now looking toward our JVM. We have quiet intensive java agents which with every click consume at least 500k of the avaiable 16m memory allocated by the JVM. If we have a lot of users running these agents we could be running into a potential problem, but we are not sure as yet.

We have been running on the 6 version since its release and on 651 and 652 for about 6 mounths. What we have noticed is that our problem has gotten worse with more users, before we never had this problem show up. IBM says its our code and we have supplied them with one .nsd file after another, yet they are not very specific. They do say its our lotusscript agent. Now my understanding is that the ls agents should look after themselves???

Our plan of action is to now monitor our JVM by writing free memory and total avaiable memory to logs and screen, and then to get our intensive agents to force a garbage collection. Hopefully we will stay up longer.

A complete reboot of the server(actual box) does also help, and does make the system stay up longer. Just restarting the Domino server brings it down faster.

Failing all this we may be forced to go to Websphere as Domino is proving to be more and more unstable. Good work IBM ;-p

3

Subject: RE: Insufficient Memory Errors in Version 6 causing major issues

We are seeing the same problem here with insufficient memory. I have tracked our problem down to the Indexer NIF pool.

We have one server that bangs out every time the NIF pool gets to 128Mbytes. Throwing more memory at the box does not seem to be helping that much.

The NIF pool, since R5, has been integrated into the NSF Buffer Pool and you can’t manage it independently.

What you can do is check to see if your NIF pool looks overloaded. Then check the NSF Buffer Pool stats. Gradually increase the NSF Buffer Pool to see if it alleviates the problem.

Other than that, develper suggestions from LS are to go to 6.5.3 CF1 or 6.5.4 when it ships.

Not so helpful but better than nothing.

To be realistic about this, we have circa 200 Windows servers here and only 2-3 of them have this problem.

I would suggest a complete code wipe (leave the data) and re-install as an option. Or even a complete wipe and drop.

We are not seeing this issue on new clean installs, just on older upgrades. We have seen instances of old R5 DLL’s (not configured but referenced from time to time anyway) which have caused havoc with upgraded servers.

Sorry I can’t give a resolution, but I haven’t managed to work it out for myself yet. However there must be a reason and a resolution otherwise we would have 200 problems not 2-3.

4

Subject: RE: Insufficient Memory Errors in Version 6 causing major issues

Has anybody found a fix for this issue yet? We are running 6.5.4FP2 on OS400 V5R3.

The errrors are happing more and more frequently now.

The system IPL’s on Monday morning and the server runs fine all day Monday. Notes goes down for backup Tuesday morning and when it comes back up we start getting “insufficient memory” within an hour.

To fix it we shut the server down and clear the shared memory. That usually keeps the server up for the rest of the week then.

Notes goes down every morning for backup’s we just have the problem on Tuesdays!

We have been on the phone with IBM and they have not yet given us a reason for it or better yet a fix! This has been happening to use since April!

Thanks for any input

Dennis Heinle

5

Subject: Insufficient Memory Errors in Version 6 causing major issues

Jim,

 I've seen memory leakages where the memory utilization shot up 1.5 MB per web page loaded until it took the Domino server down.  It wasn't solvable (I think it's a bug that Lotus needs to fix), but I did manage to find some notes.ini parameters that kept the system in check and stopped our outages.  The parameters are:

o ConstrainedSHMSizeMB

o ConstrainedSHM

o PercentAvailSysResources

In that case, I was consulting for a company that ran 8 Domino servers on one physical Windows server, and I think each Domino instance was thinking it had the whole machine to itself, thus sucking up too much memory. Once we constrained the memory and limited Domino to a percentage of total memory, the situation stabilized.

To read more about these and other notes.ini parameters, go to Darren Crocker’s notes.ini reference website at – http://drcc.dyndns.biz/ref/notesini.nsf/all

Good luck!

Ken

6

Subject: RE: Insufficient Memory Errors in Version 6 causing major issues

I hate those posts that say “I’ve got the same problem” and add nothing new. I sincerely hope that, although I haven’t got the solution, at least you can use the information we’ve gathered.

We have a very important loan approval application running towards insuccess because of this problem. I’ll try to put here our observations and conclusions on this issue. Note that we have this problem reported to IBM and 3 months are gone without any solution presented.

  1. The application

Web application, with great amount of http calls, used by almost 500 users. We use DXLImport and DXLExport functions to transport XML to and from Domino.

  1. The environment

  • 2 GHz processors, 4GB memory

  • Failover cluster with disk array (RAID10)

  • Win2000, .Net, SQL Server, Domino 6.0.3 (also tried 6.5.1 with same results)

  1. The way it works

.Net interface, calls http links in Lotus Domino, passing XML by http parameter. Domino creates and/or gets documents and sends XML back. No big deal. Works like a charm.

  1. The problem

Nhttp memory usage, in Windows Task Manager, increases and increases… and never comes down again (“It’s supposed to work like that” - IBM). Well, until release 5.0.10, it didn’t.

Everyday, when nhttp reaches aprox. 1.2 GB of Memory usage, the domino server crashes with a “PANIC: Insufficient memory”.

  1. Our attempts to solve the problem (specified by IBM and used exclusivelly)

    • ConstrainedSHM=1

    • ConstrainedSHMSizeMB=2048

      Result: “PANIC: Insufficient memory”

    • ConstrainedSHM=0

    • ConstrainedSHMSizeMB=3072

      Result: “PANIC: Insufficient memory”

    • PercentAvailSysResources=80

      Result: “PANIC: Insufficient memory”

    • httpusenotesmemory=0 (don’t bother, it’s a “not publicated feature”)

      Result: “PANIC: Insufficient memory”

  2. Conclusions

Are we doing something wrong here?

By the way… I’ve done some load tests, simulating about 100 users accessing iNotes simultaneously and guess what… nHttp increased and server crashed. So don’t tell me that the problem is .NET like if it was a sin to put .NET and Domino working together.

Oh! I almost forgot. Some people at IBM told us (informally) to create a Domino cluster but others, at the same company, tolds us not to do it. (?!?!)

Thank you and accept my apologies for making you read this long post.

I’ll keep trying to get the solution.

Joaquim

7

Subject: RE: Insufficient Memory Errors in Version 6 causing major issues

I set all the parameters, but it doesnot workI set ConstrainedSHM=1

ConstrainedSHMSizeMB=2048

PercentAvailSysResources=80

any wrong with this?

Also I try to use nservice.exe -jc -c =d:\Lotus\Domino\notes.ini

but it doesnot work as well, nservice.exe and nupdate.exe increas memory usage gradually…

so, any suggestion?

8

Subject: Insufficient Memory Errors in Version 6 causing major issues

Get the same thing on the iSeries - I feel your pain! The problem is with the shared memory not being cleared.

Here is how you clean it up without having to IPL.

1 - End the domino server(s).

2 - Open iSeries Navigator and go to your Domino iSeries

3 - Expand Application Development

4 - Expand Interprocess Communication

5 - Double-Click Shared Memory and delete all entries for Owner Qnotes

6 - Double-Click Semaphoer Sets and delete all entries for Owner Qnotes

7 - Double-Click Kernel Message Queues and delete all entries for Owner Qnotes.

That will clear the Insufficient Memory error messages.

9

Subject: RE: Insufficient Memory Errors in Version 6 causing major issues

Chris,

I’ve followed your solution, and … I works ! ! !

1,000,000 thanks.

Alexandre

10

Subject: RE: Insufficient Memory Errors in Version 6 causing major issues

Thank you both for such a quick response!

It’s nice to know (in a sadistic way) that we’re not a lone in our suffereing.

Ken,

The link to “INI” reference is great! Thanks.

Chris,

Your explanation makes sense. This sounds like a on-demand “KillNotes” for the

server (iSeries) which I love! Is it possible to run these type of commands in

a CL on schedule? Is there possibly an API?

Thanks.

11

Subject: RE: Insufficient Memory Errors in Version 6 causing major issues

Jim, I’m not sure about running this to clear the shared memory. It’s NOT something that you will want to do unless the server is down. If you delete the shared memory while the server is up then it will shut down that notes server in a “panic” mode. Trust me, I’ve done this before!

Also, from what I understand there is a way to access the shared memory/semaphores, etc… through Green Screen but it is nothing less than a headach to do so. That’s why I just stick with the iSeries Navigator.