Hello Everyone,
I’m attempting to get my hands around Domino memory management. Here’s my situation, hopefully I can get some nuggets of insight. We have an app server running Domion 6.5.1 and LEI 6.5.1 on a Windows 2000 Server. A Dual Xeon 2.8ghz with 2gb memory. The server was just put into comission, everything previous was on a dual 1ghz server w/ 1gb of memory.
There are no mail users, however it is an intense app server, lots of LEI scripted activities bringing data down from an AS/400 (possibly 30+ activities kicking off at various schedules, i.e., every 2 hours, 1 hour, etc.) (so therefore using the db2 connectors). There are also agents that run in about a dozen or so databases that wake up and do there work every 15 minutes or so then go back to sleep (the work takes a few minutes or so). There are a couple of monster databases that have a ton of views, and a ton of updates (as a result of bring data down from the as/400 and performing deletes on it). One database has 400 views (yes I know that’s a different issue but out of my hands at the moment).
We have everything some what stable, meaning everything works great for about 6-7 hours and then we get the:
“Cannot Create Product Object Error”
Which to me indicates a Domino out of memory condition. No LEI or LotusScript objects can be created and I verified this by putting in some trace code that dumps to a text file that writes where a particular script stops (and it stop on trying to create an LEISession Object).
We solve by rebooting the machine and it runs fine for another 6-7 hours than again the same error. Clearly there is a memory leak somewhere. I’ve gone through a lot of the scripts, and done cleanup, i.e., explicity setting product objects to nothing when done with them (i.e., set session = nothing), and trying to be a good citizen when creating objects in a while/wend loop (i.e., we had one script create a new NotesDateTime object everytime through the loop, so first I now deallocate the object by setting to nothing before creating a new one).
Sorry I wanted to give you guys/gals context. Specifically how do I know if any of these tweaks, code improvements are helping? Everytime we think we find something, it runs great for 6-7 hours and then stops. We know it’s not something that occurs at a specific time because usually within about 6-7 hours of starting domino it gets the error mentioned above.
I started looking at NSF BUFFER POOL information and I think I understand how it works. It’s used to buffer data i/o from database and view activity (I think). Domino was setting the max NSF BUFFER POOl (Database.Database.BufferPool.Maximum.Megabytes) to 747mb on our 2gb machine. I tried using the SystemPercentAvail (or something like that) notes.ini and tried setting it to 80% of available memory, but it actually seemed to lower the buffer pool max to 650mb? Wierd. I used it b/c I read using the NSF_BUFFER_POOL_MAX_MB (or something like that) was discouraged in R6. Anyway since the previous ini didn’t work, I used NSF_BUFFER_POOL_MAX_MB and set it to 1250, and it seems to have set correctly (i.e., I verified it by doing “show task Database.Database.BufferPool.Maximum.Megabytes”.
Okay phew, here’s my real question, how do I know when memory utilization is getting better? I’ve been watching the:
BufferPool.Maximum.Megabytes
BufferPool.Peak.Megabytes
Trying to see if Peak.Megabytes is any indication that it’s running out of memory. For example when it was BufferPool.Maximum.Megabytes = 747mb (by default) BufferPool.Peak.Megabytes=746mb. So I assumed we needed more bufferpool memory. So I bumped it up to 1250 and now the peak is at 1247mb. Now I don’t think I understand how this works and can’t find any realiable information (I’ve looked in the tech library and google, and there’s some stuff but nothing that says, hey this is how it all works and how to measure your memory and know when things are going bad).
I’m not asking for help on cleaning up scripts (I know we need to do that) or tone down LEI jobs, but I need to know how to measure that things are getting better vs. how I’ve been doing it before by waiting 6-7 hours before it ran out of memory.
Can I get insight on what stats to look at? I did all the perfmon mon stuff and also discovered the console open/close bug (which is frustration in and by itself). Here’s the thread on that topic:
So in closing, here’s what my current stats are and another plea for help from a confused Domino guy who needs real practical insight into how this stuff works!
Thank You!
Database.DAFailoverCount = 0
Database.DARefreshServerInfoCount = 0
Database.DAReloadCount = 0
Database.Database.BufferPool.Maximum.Megabytes = 1250
Database.Database.BufferPool.MM.Reads = 3
Database.Database.BufferPool.MM.Writes = 1
Database.Database.BufferPool.Peak.Megabytes = 1248
Database.Database.BufferPool.PerCentReadsInBuffer = 97.79
Database.DbCache.CurrentEntries = 24
Database.DbCache.HighWaterMark = 304
Database.DbCache.Hits = 70558
Database.DbCache.InitialDbOpens = 73221
Database.DbCache.Lookups = 76967
Database.DbCache.MaxEntries = 3750
Database.DbCache.OvercrowdingRejections = 0
Database.ExtMgrPool.Peak = 65,406
Database.ExtMgrPool.Used = 816
Database.FreeHandleStack.FreeHandleStackHits = 666345
Database.FreeHandleStack.HandleAllocations = 671982
Database.FreeHandleStack.MissRate = 0
Database.LDAP.NAMELookupBindFailures = 0
Database.LDAP.NAMELookupBinds = 0
Database.LDAP.NAMELookupBytesReceived = 0
Database.LDAP.NAMELookupEntries = 0
Database.LDAP.NAMELookupFailures = 0
Database.LDAP.NAMELookupTotal = 0
Database.LDAP.NAMELookupTotalLookupTime = 0
Database.NAMELookupCacheCacheSize = 105,160
Database.NAMELookupCacheHashSize = 1,293,907
Database.NAMELookupCacheHits = 8,714
Database.NAMELookupCacheLookups = 15,471
Database.NAMELookupCacheMaxSize = 16,777,216
Database.NAMELookupCacheMisses = 2,587
Database.NAMELookupCacheNoHitHits = 4,170
Database.NAMELookupCachePool.Peak = 2,097,152
Database.NAMELookupCachePool.Used = 1,485,856
Database.NAMELookupCacheResets = 0
Database.NAMELookupMisses = 1,638
Database.NAMELookupTotal = 14807
Database.NAMELookupTotalLookupTime = 25.474
Database.NIFPool.Peak = 3,145,728
Database.NIFPool.Used = 352,576
Database.NSFPool.Peak = 6,291,456
Database.NSFPool.Used = 772,352
Database.NSF.ClusterHashTable.EntriesWithSameIndex = 0
Database.NSF.ClusterHashTable.FreedEntriesOnCleanup = 0
Database.NSF.ClusterHashTable.HashedEntries = 0
Database.NSF.ClusterHashTable.HashIsFull = 0
Database.NSF.ClusterHashTable.MissedHashHits = 0
Database.NSF.ClusterHashTable.SuccessfullHashHits = 0
Database.NSF.Replicate.NotesMergedBack = 0
Database.NSF.Replicate.NotesReceived = 0
Database.NSF.Replicate.NotesReopened = 9296
Database.NSF.Replicate.NotesSent = 0
Database.NSF.Replicate.UnreadMarks.ChunkLookups = 0
Database.NSF.Replicate.UnreadMarks.ChunksScanned = 0
Database.NSF.Replicate.UnreadMarks.ChunksSkimmed = 0
Database.NSF.Replicate.UnreadMarks.FullReplications = 0
Database.NSF.Replicate.UnreadMarks.LocalMaxChunks = 0
Database.NSF.Replicate.UnreadMarks.LocalUnreadOperations = 0
Database.NSF.Replicate.UnreadMarks.MessagesReceived = 0
Database.NSF.Replicate.UnreadMarks.MessagesSent = 0
Database.NSF.Replicate.UnreadMarks.OperationsReceived = 0
Database.NSF.Replicate.UnreadMarks.OperationsSent = 0
Database.NSF.Replicate.UnreadMarks.UsersActive = 0
Database.NSF.Replicate.UnreadMarks.UsersActiveMax = 0
Database.NSF.SignatureCache.Hits = 9095
Database.NSF.SignatureCache.Tries = 10915
Database.RM.Sys.Logged = Disabled