What's good Hardware for Domino6.5 on w2k3 in the >20k range?

HCL Domino Domino Forum
1

I could use some help from anybody who is running 6.5 on a server that costs >20k, anybody with low-mid range dell, hp, ibm rack servers.

Currently they have clustered 2 >2k, pc based servers with mirrored 60gb hard disks, and transaction logging. 1.5gb ddr ram and a single 2ghz amd processor each. They would like to step access times up 10 fold. (Of course they don’t want to spend any money over $500 to upgrade the pc, and expect me to work miracles!)

It takes 5-20 seconds to switch views in notes, and inotes runs incredibly slow for some users. There are only 120 users on the system 20 notes, 100 inotes. This number of users is expected to double every year. Also the server is running blackberry server with attachment handling(cpu hog), CA anti-virus(cpu hog) and is a w2k3 Domain Controller with dns.

If they buy a new (expensive in their eyes)system they also want to consolidate on this new machine from another pc with ISA server, file server, with rras enabled for >5 vpns and veritas backup exec 9.1. They might add sametime to this server as well. They want to keep domain indexes, mail file indexes, SSL and notes tcp encryption enabled as well to slow things down even more.

I was estimating at least a 4gb 2-4 processor 2-3ghz xeon server with 4 scsi disks raid 5 with some raid ram(128?). I’ve run performance monitor, and the current pc based servers use avg 30% cpu, avg 65% disk time and avg 60% ram. What is the best system for them and their money that will give them noticable gains? Basically file transfer times are acceptable, but access times are not and they want to increase both.

I also have to convince this company to purchase a better server machine if they want better performance. I’m still having difficulty explaining that a decent server for them will cost at least 20k. So any links to performance comparisons between Domino on a pc vs real server would be greatly appreciated. I could also use the info on what’s better for access times( more cpu’s? ghz vs cpu’s? more ram? faster raid?)

2

Subject: I would tune the OS and Domino first and do a serious cleanup of the environment…

We figured out that in most cases not the hardware is the limiting factor. Upgrading one of the latest Domino 6 versions alone gives you a lot of performance. Than you should have a view_rebuild_directory. Make sure mail-files are not bigger than 500 MB (okok I know this is almost impossible in large organizations but this really helps to gain speed).

Than analyze which applications are slow and why they are slow. Check the view rebuild logging which can be enabled via LOG_UPDATE =2 LOG_VIEW_EVENTS=1 (see more details in TN #1097302).

Than make sure not too many documents are changing. Some companies have heavy import agents which even generate a lot of deletion stubs…

We also figured out that sometimes not the Domino server is the limiting factor. If they have their data directories on remote network shares this could lead to dramatic performance problems in some cases. We have customers running into trouble in this area and helped them to troubleshoot this with client_clock=1 and also build a extension manager to narrow down network problems. It turned out that the access to desktop.dsk and specially to cache.dsk was the main bottleneck.

Last but not least we have a special catalog collecting a lot of information about all the databases in the system to see what can be optimized. But you can get already a lot of information from the logs and the standard catalog.

New hardware is not always the solution to performance problems. You can make it faster by spending a lot of money. But it could make sense to find the underlying reasons why this is so slow… I just gave you some ideas. There is a lot more in the technote I mentioned and also in other resources in Kbase. I would do some more testing before proposing new hardware.

Daniel (ldd@nashcom.de)

3

Subject: RE: I would tune the OS and Domino first and do a serious cleanup of the environment…

So where do I find information on setting up the view_rebuild_directory? They have the transaction logs on a seperate disk that is underutilized. I could place the view rebuilds there, but I didn’t know that was possible till you both mentioned it.

-Joe Rose

4

Subject: oops sorry this was not the exact spelling for the parameter …

here is an extract from Admin 5 help. That’s just one of the things you might want to tune. There are many other places to look into. A very simple thing is for example to check if the NICs are set to automatic speed detection. It makes sense to set it hard to 100Mbit / full-duplex or whatever works in your environment. We have seen huge performance problems in some cases for example with CISCO switches if you use auto speed in some environment (mostly Unix but also Win32). So those kind of changes can make a real difference. I cannot list all of them but can give you some ideas in this area.

– Daniel

Changing the temporary folder used for view rebuilds

When Domino rebuilds views – for example, when you use updall -R or when a user opens a view whose index has been deleted – it may generate temporary files to sort the data in order to rapidly update the views; Domino deletes these files after rebuilding the views. By default, these temporary files are located in your system’s temporary folder – for example, C:\TEMP. If your system doesn’t have a temporary folder, then Domino puts the files in the Domino data folder.

Depending on the amount of memory available during rebuilding, the space required in the temporary folder for each view being rebuilt is approximately two times the size of the largest view or two times the size of all the data in documents, whichever value is greater. It is recommended that you change the location of the temporary files to a different drive from the Domino data folder. Putting the temporary folder on a different drive distributes disk I/O and ensures that there is enough space to rebuild views. Domino is very conservative when estimating the amount of disk space needed for optimized view rebuilds so that it won’t spend unnecessary time sorting data only to discover that there’s inadequate disk space. Make sure that the temporary folder you specify has plenty of disk space available.

To change the temporary folder used for view rebuilds, add the setting View_Rebuild_Dir to the server’s NOTES.INI file and specify a new location. For example, add:

View_Rebuild_Dir=D:\REBUILD\

Be sure to include the final backslash ().

If Domino estimates that there’s not enough space available in the temporary folder to rebuild a specific view, Domino uses the slower Release 4 method to rebuild the view and logs this message to the Miscellaneous Events view of the log file (LOG.NSF):

Warning: unable to use optimized view rebuild for view due to insufficient disk space at directory. Estimate may need x million bytes for this view. Using standard rebuild instead.

You can add the following setting to the NOTES.INI file to disable optimized view rebuilding. However, do this only as a last resort if you’ve specified a view rebuild folder and you still see the above message for many views. If you see the message for just a few views, don’t disable view rebuilding.

Disable_View_Rebuild_Opt=1

5

Subject: RE: I would tune the OS and Domino first and do a serious cleanup of the environment…

The main problem for this customer is extremly slow response times for several users and normal for others. They all access the same server. The remote users have a T1 or cable connection to the internet.

Even at the same time on the server, some notes users have decent response times while others time out consistently, sometimes those same users don’t time out.

While using web access, over T1 or cable, those users experience 2-3 minute delays switching between pages in inotes or webaccess. Where other users, such as myself, experience 2 seconds on a 28.8 dialup modem! Some users hang on the notes client accessing mail, while a different user logs in on that exact same pc, and has no problem what so ever. They need something done to fix this, but I don’t know how to help, I’ve tested everything hardware/network/OS wise.

I have no idea why some users consistently are so slow that it is unusable. for some users on a T1 connected pc it takes 20+ minutes to login to inotes, but I take 1-5 seconds on dialup, page changes for them take minutes not seconds. And these users have less than 10 messages in their view, so that’s not the issue.

-Joe

6

Subject: some more info: I have a presentation I did last year for German Notes User group about Performance …

if someone is interested drop me a line at ldd@nashcom.de (the last time I said this for the SAI presentation I got hammered with emails and I still get mails once in a while for the SAI presentation).

The session is about Performance best practices and also performance monitoring and new features in Domino 6. We spend a lot of time troubleshooting server availability and performance. Mainly on Unix and Linux but also on Win32 and general Domino settings.

– Daniel

7

Subject: ok here is a request form for the session…

if you click on this link you get a request form where you just have to type in your title/name and email address and the session is send to you ASAP.

http://ldd.nashcom.de/nshlstrq.nsf/dnug2003_performance!openform

By the way this is based on documents created in a request database and a ListServerTask on my Domino Server is running to send out the presentation :wink:

– Daniel

8

Subject: RE: I would tune the OS and Domino first and do a serious cleanup of the environment…

The main problem for this customer is extremly slow response times for several users and normal for others. They all access the same server. The remote users have a T1 or cable connection to the internet.

Even at the same time on the server, some notes users have decent response times while others time out consistently, sometimes those same users don’t time out.

While using web access, over T1 or cable, those users experience 2-3 minute delays switching between pages in inotes or webaccess. Where other users, such as myself, experience 2 seconds on a 28.8 dialup modem! Some users hang on the notes client accessing mail, while a different user logs in on that exact same pc, and has no problem what so ever. They need something done to fix this, but I don’t know how to help, I’ve tested everything hardware/network/OS wise.

I have no idea why some users consistently are so slow that it is unusable. for some users on a T1 connected pc it takes 20+ minutes to login to inotes, but I take 1-5 seconds on dialup, page changes for them take minutes not seconds.

-Joe

9

Subject: do you have reverse DNS queries enabled on the server?

if yes turn it off this could take you into trouble. Also there are tools to put between server and client like a proxy to trace what is going on (one step above a protocol analyzer). This could help to figure out what is going on.

In some cases even the time in domlog.nsf can help to figure out what request exactly is taking long.

It might not be a direct server problem. It could be on the browser, the network, the network services like DNS specially with reverse DNS queries. There are improvements in rDNS lookup you can enable in the server document for caching DNS queries. This should also cache non-successful DNS queries which happen due to network status queries which are done if the remote IP has no DNS entry. If the firewall on the other side blocks the request in the wrong way you could run into time-outs which cause a lot of trouble. Just and additional idea to check… I ran into this and had to figure it out the hard time …

– Daniel

10

Subject: RE: I would tune the OS and Domino first and do a serious cleanup of the environment…

What is the hardware at the far end of the T? iNotes can be that slow if it’s running on outdated hardware at the client end. Also, are these users running personal firewalls that do http redirects? Those can absolutely KILL performance on a DHTML application (in which case, add the server as a trusted server.)

11

Subject: What’s good Hardware for Domino6.5 on w2k3 in the >20k range?

I’m with your customer here : way too expensive !

The hardware they have now should suffice for what they do for now - the domino server probably needs some tweaking here and there. The cpu and memory values are within safe margins. Focus on indexing, check database design, check the configuration - it should run just fine. Also, you could add a harddisk in the machine (put it on a separate controller/channel) to use as transactional logging disk, and for the building of the view indexes.

Now, for the future, 2000 notes users in 5 years can be easily supported on a dual processor system, with 2 Gb RAM, with a raid 5 storagesystem.

If you want to go for high availability or load balancing, use 2 separate machines and build a cluster. It’s not necessary to use a 4 processor machine for 2000 users, and since Domino licenses are per-CPU, you’ll actually get better performance and higher availability putting it on 2 separate machines. It’s probably cheaper too - the second machine doesn’t really need to have 2 processors. Actually, you could start with 2 machines with 1 processor each.

I think your 20K could buy them the whole infrastructure (rack + 2 machines + firewall with vpn + software), and not just 1 (oversized) server. And any midrange server would do.

cheers,

Tom

12

Subject: What’s good Hardware for Domino6.5 on w2k3 in the >20k range?

[Also the server is running blackberry server with attachment handling(cpu
hog), CA anti-virus(cpu hog) and is a w2k3 Domain Controller with dns.]

If they’re running a pair of clustered servers, why are you listing that “the”
server has these extra functions running? Are both cluster members running
Blackberry and domain controller services?

Average CPU and I/O loads aren’t going to help too much here. What we need to
know are peak loads, particular during iNotes usage. That’s the way to
identify bottlenecks. iNotes itself is not a quick piece of software,
especially on weak CLIENT machines. All that DHTML is executing locally at
the client, and older machines can have a lot of trouble keeping up. It’s not
just a web page.

Further, the loads that you list should not result in 20 second view open
times, unless these are just enormous views. Your description smacks of
network bottlenecks to me. Daniel’s suggestions are also good ones.

With 100+ users, the idea of having a dedicated Notes server should not be a
problem here. Put all that domain control/backup/vpn, etc crap on other
machines where immediate response time is not so critical for users. A
dedicated Domino server with good I/O partitioning and fast network access is
not expensive. It would be easy to put one into place for less than $2K (just
priced a server from Dell at $1800). Then you can take Domino off the two
existing boxes, and split up all that other business between them. (If they’re
not willing to spend a couple of grand on hardware, clustering for reliability
is clearly not something they’re serious about.)

By the way, if they want to step up access times 10 fold, that should be easy
and cost next to nothing. Just downgrade to a 10Mbps NIC and hobble the CPU
speed in the BIOS. But I imagine what they want to do is improve access
times 10 fold. :slight_smile: (This is a ridiculous standard to set, by the way, without
considerable investment in identifying performance bottlenecks.)