Invitation to Open Mic: Domino Server Performance

HCL Domino Domino Forum
1

IBM is hosting an Open Mic conference call with Lotus Development and Support Engineering to discuss Domino Server Performance (versions 6, 7 & 8)

The Open Mic call will be held on Wednesday, June 25’nd 2008. This call will take place in one session at 10:00 AM EDT. The call will last 60 minutes. Please dial into the call 5 minutes before the scheduled start. This conference call is designed to be an open question & answer format, so bring your questions.

You may find it helpful to review the following resource prior to the conference call:

             Notes Domino Best Practices: Performance

We also encourage you to share your experiences and perspectives about performance and other Notes/Domino topics in the Notes/Domino Best Practices Wiki.

Please refer to the Open Mic Tech Flash for details about the conference call numbers. Please post any advanced questions within the ND8 forum by creating a response to this document. This Open Mic call will be recorded for future use, and will be made available via the Flash after the call.

2

Subject: Router Memory Usage

Not sure if this is off-topic, but we’ve had some memory-related issues with the router process recently, and while I don’t want to go into those specific problems, it would be helpful to understand the following: 1)an overview of the memory allocation process the router uses to accomplish its tasks. 2) if the router needs to allocate a large amount of memory to process either an enormous email or large email to a big grouplist, will this allocated memory ever be “returned” to the system or will that only happen when the Server is shut down?

3

Subject: Re: Router Memory Usage

Here are some additional technotes that elaborate more on the BLK_OPENED_NOTE memory leaks that I mentioned during the open mic today. These outline various SPRs for leaks in that block as well as INI parameters that can be used to prevent certain leaks.

Title: Excessive Domino server memory usage due to BLK_OPENED_NOTE

Doc #: 1242086

URL: http://www.ibm.com/support/docview.wss?rs=899&uid=swg21242086

Title: Can the number of BLK_OPENED_NOTE blocks be limited?

Doc #: 1198511

URL: http://www.ibm.com/support/docview.wss?rs=899&uid=swg21198511

Title: Crash in any Domino task due to high memory usage in BLK_OPENED_NOTE

Doc #: 1230023

URL: http://www.ibm.com/support/docview.wss?rs=203&uid=swg21230023

Title: Errors about signer rights occur during AdminP process

Doc #: 1233114

URL: http://www.ibm.com/support/docview.wss?rs=203&uid=swg21233114

Title: How to limit the number of threads used for sending large messages

Doc #: 1108351

URL: http://www.ibm.com/support/docview.wss?rs=899&uid=swg21108351

The parameters referenced in this technote also limit messages in different ways:

Title: Mass mailing causes Router to run out of memory and enter a hang state

Doc #: 1227966

URL: http://www.ibm.com/support/docview.wss?rs=899&uid=swg21227966

As I mentioned on the call these aren’t the only reasons for spikes in router memory usage, but these are some very common ones.

4

Subject: Server Performance issues

We have a pretty beefy mail server that was recently upgraded to R8.0.1, and also the version went from 32 to 64 bit. The server did not seem to gain any performance increase, in fact some users complained about delays in their mail files. Is there a Best Practices document specific to R8 or R8.0.1??. If not can you recommend a set of technotes I can use to analyse and determine possible root causes for this performance degradation??

Thanks

5

Subject: upgraded to R8.0.1, and also the version went from 32 to 64 bit

Will need additional information on this one to provide a useful answer* was this specific users? How are they different from users who do not see a change in performance?

  • did this only manifest itself immediately after the upgrade? do the same users still see a problem after first use?

  • OS?

  • specific user actions slower? or overall perception?

6

Subject: Performance Issues.

Thanks for your response, no specific users that we can tell. Even I experience the delays from time to time and I got a pretty beefy machine. This issue is reported from time to time, no specific pattern to it. The problem is also that most most users won’t bother to report the problem, sometime we just get comments as opposed to actual calls to the Helpdesk. What about R8.0.1 specific best practices or technotes docs to help monitor \ diagnose performance behaviour?.

Thanks again.

7

Subject: Performance Issues

Based on your answer, my suspicion is that there is activity that is periodic in nature on the Domino Server causing your issue. I would start by watching the activity pertaining to the Name and Address Book - how many and frequency of Adds/deletions/changes, and the statistics that are listed in “show stat update”

8

Subject: Performance Degradation when upgrading to 64-bit

I’m assuming that you are referring to Windows 64-bit as the OS. In general, moving to 64-bit will help with memory usage issues, but should also result in faster CPU, since it has a bigger straw if you will.

You may want to refer to the 32-bit vs 64-bit discussion earlier in this Open Mic thread. Specifically:

There is a known issue with 64-bit Windows where the System Cache grows large enough that it consumes most of the RAM on the system. This issue is documented and resolved in Domino 8.0.1 and 7.0.4 (which isn’t released yet). If you run Domino on 64-bit Windows and see this issue you can contact support and reference the following technote:

Title: Running Domino in a Windows 64-bit environment greatly impacts performance

Doc #: 1270452

URL: http://www.ibm.com/support/docview.wss?rs=899&uid=swg21270452

As far as recommended Technotes for troubleshooting, I would recommend:

Title: Domino Server Performance Troubleshooting Cookbook

Doc #: 1234550

URL: http://www.ibm.com/support/docview.wss?rs=899&uid=swg21234550

You should also probably open up a ticket with Support if these resources don’t help you nail down the problem. To Andy’s point, we would be looking for a pattern of activity and/or users affected by this slow down. Hope this helps.

9

Subject: Storage Area performance

Our main concern is about Disk Arrays performances.I am currently trying to design a distribution of I/O for a future AIX 5.3 pSeries infrastructure which will run eight Domino 7.0.3FP1 servers. Our Storage team proposed us an EMC Clariion Storage Area with 8 Raid Groups under RAID5 strategy and 2 Raid Groups under RAID 10 strategy. Then we will distribute mail boxes on RAID 5 in several LUNs. And we will host configuration files (like names.nsf, log.nsf…) and binaries and view indexes and translog on RAID 10 in separate LUNs. I read about Translog recommendations and it is said to set the Translog on a separate PHYSICAL disks. Does it mean that i should request for additionnal physical disks to create a new Raid Group in RAID 10 strategy to host the Translog files ? Should i do the same to host the View indexes ? Or should i be lucky and count on the separate LUNs to host the different kind of files and put them all together on the same Raid Group ?Thanks in advance for your help.

10

Subject: Storage

rule of thumbs- separate File Systems for Domino Data/Binaries, - more smaller (within reason ) is better - each FS has its own caches - Domino does lots of smaller sized, Random I/O so ReadAhead Cache is virtually useless, Write Cache is great - we do recommend a RAID version, note RAID 5 has an increased I/O penalty compared to RAID 10, but RAID 10 requires more disks

Tx Log on Raid 10 use the Fastest, most reliable storage you can - sequential writes in 4K blocks

View Rebuild/Page File - own File system Raid 10 or can skip mirroring drives for this

Consider own File system for Mail Journaling, or 3rd party archiving solution

11

Subject: Question

In advance to the telephone conference, I would like to ask a question:

One of the key issues we have with Domino Server performance is the fact that a single killer application can consume all available server resources. So all other applications locate on the server will be affected.

As an administrator, I would like to see a feature, where I can limit the server resources for particular databases.

One option would be a database property which can limit the activity for the indexer task on this database.

Would this be a considerable option for a future release??

Kind Regards

Thomas Hampel

12

Subject: Memory question

We have dedicated Domino 7.0.2FP2 mail servers with circa 700 users each (web access 6 templates but HTTP is not used yet) and a an average mail file size of 900MB (from 150MB to 8GB).Q:How does large mail files impact the memory utilization of the servers?

Our servers defaults to 748MB of Database.BufferPool.

Q: Should we tweak this to 512MB using NSF_BUFFER_POOL_SIZE_MB

Q:To be able to evaluate and monitor our servers performance and setting baselines for future reference, what and how should we monitor the system? SH STAT Perf Mon? If so what counters and stats.

13

Subject: Memory and tasks

Is there a way to limit certain Domino tasks to a specific limit of memory? e.g. a notes.ini setting?

14

Subject: limiting tasks to specific memory

Domino allocates Pools of memory for tasks at startup, we do a much better job of explicitly stating which pool runs out of memory, in many cases we can then change the memory allocation for that Pool. Not sure why you would want to configure memory at this level?

15

Subject: Memory Question re. large NSF Buffer Pool Size

Q:How does large mail files impact the memory utilization of the servers? A- We have drastically changed the way that we do our performance testing to look at this question (old way - 14Mb mail files with 100 docs) New way - 250Mb mail file with 3000 docs in the $inBox and mixed attachment sizes

We had seen that the bigest impact was the # of documents in the inBox, the over all size of the mail file was a lesser impact. Slightly dated (and done on iSeries but applicable to all Domino platforms)but this will give you an understanding “Red Paper #3802”

specific to memory is we will cache Db information , for the most part what we cache should not be Db size dependant

Our servers defaults to 748MB of Database.BufferPool.

Q: Should we tweak this to 512MB using NSF_BUFFER_POOL_SIZE_MB

A- I would suggest starting by watching the Domino statistic Database.Database.PercentReadsinBuffer over ~1 weeks time, IF you do not see this value stay in the 80+% range I would suggest setting the value to 512 and again watching the Database.Database.PercentReadsinBuffer value. Chances are you can set this value to 512 or below. Domino “reserves” this memory at start up, so any and all memory that is not actively needed by Domino can be given back to the OS.

Q:To be able to evaluate and monitor our servers performance and setting baselines for future reference, what and how should we monitor the system? SH STAT Perf Mon? If so what counters and stats.

A- I start with CPU/Memory/Disk/Network at the OS, for Domino I would start with all stats that have the words “Pending, Waiting, Delivered,” and hone the effort from there.

We do offer tools in the Admin Client like Server Health Monitor, Trends, and DDM

16

Subject: Re. single killer application can consume all available server resources

This is not currently a planned effort, if you feel strongly about it , please enter a Feature Request.

a couple ideas for individual applications and update might be

Build-on-first_Use-Collations

and

keeping the Db in cache for a longer period of time by setting NSF_DBCACHE_CLEAN_HOLD_TIME=

where -1 is pre Domino 8 behavior

“N” holds Db in cache for “N” minutes

17

Subject: Memory Question

What are the best 2008 tools/techniques for monitoring memory usage by the Domino Server and the various server processes (especially router and http). Also, how is memory allocated for the JVM, through the Server or as a separate process like the other server tasks?

18

Subject: Memory Question (what tools to use)

Q: What are the best 2008 tools/techniques for monitoring memory usage by the Domino Server and the various server processes (especially router and http). Also, how is memory allocated for the JVM, through the Server or as a separate process like the other server tasks?

A: By 2008, I assume you mean Windows 2008 (the Operating System)? To my knowledge, the techniques we use for troubleshooting memory usage on Windows have not changed significantly for Windows 2008.

The best tools tools to use are:

1). Domino diagnostics such as Domino stats, memory dumps, NSD (this is most helpful to Support, but we have some tooling on the way to assist with analysis in this area)

2). Perfmon stats (resists the temptation to use Task Manager; just don’t do it). With Perfmon, you want to use the Process Object, the Private Bytes and Virtual Bytes counters, and log the instance you are interested in (like http or router).

Private Bytes indicates how much memory has been allocated for private use, whereas Virtual Bytes shows all memory usage (private, reserved, shared). Keep in mind that Domino uses ALOT of shared memory, so Virtual Bytes is the real key here. If you are getting past 1.6-1.7 GB, then you have problems (for Windows 32 bit specifically).

As far as JVM memory usage, the JVM is loaded under the process that uses it (for instance AMGR, HTTP, etc), so it is not a separate process, but must run inside the Domino process that invokes it. This can definitely lead to problems if you set the JVM MAx Heap size too big. Even if the JVM doesn’t use the memory (RAM or pagefile), it will still reserve the space from the address space, which impacts all the other activities that need to happen for a given process, for example HTTP.

Some resources that discuss Perfmon and Task Manager:

Title: Domino Server Memory Usage Drops when Minimizing the Server Console

Doc #: 1109688

URL: http://www.ibm.com/support/docview.wss?rs=899&uid=swg21109688

Title: Using Windows Task Manager to determine Domino memory consumption

Doc #: 1230851

URL: http://www.ibm.com/support/docview.wss?rs=899&uid=swg21230851

Title: Domino Server Performance Troubleshooting Cookbook

Doc #: 1234550

URL: http://www.ibm.com/support/docview.wss?rs=899&uid=swg21234550

19

Subject: Server Sizing Guidelines

Are there any updated guidelines available for determining when a server has reached it’s optimum or maximum capacity so a hardware upgrade or replacement needs to be considered ?

20

Subject: Guidlines to determine when to upgrade hardware

This is where folks who continuously monitor resources and system usage have an advantage. Most times customers run into a “bogged down” system is due to growth over time. adding a piece here, and an application there, a few users from this department, and some application upgrades

Tools we have for Domino include DDM, Statistics, Event Monitors and probes. There are also tools from Tivoli and other 3rd parties