My idea of how Domino handles HTTP threads is this:When a HTTP request is sent to the server, Domino assigns this request to one of the active (default 40) HTTP threads. Each consecutive request is assigned a the next HTTP thread, so in theory if 80 requests would be sent to a server (with the default settings), each HTTP thread would have two requests assigned to it.
The problem of this method is the following: If a HTTP thread is occupied for a more than average time, because of for example a WebQueryOpen or WebQuerySave agent or even worse, if a HTTP thread hangs, Domino still assigns new requests to this thread, which therefore have a huge delay or won’t be processed at all.
My questions to the community:
-
Do you also think that this is how Domino handles HTTP requests or are my ideas wrong?
-
Have you encountered the problem of hanging HTTP threads?
-
If someone of the Lotus development team reads this: Why doesn’t Domino work with a central queue which assigns a request to each open HTTP thread? Will it be implemented this way in Domino 7?
Subject: HTTP Thread handling by Domino
I’m uncertain as to how Domino’s thread queueing works with regards to your example. I don’t know of any place where it is documented, and it’s obviously impossible to see how queued requests are pre-dispatched to specific threads (if at all).
Domino does not seem to do dynamic thread-spawning as some web servers do. It spawns all of your threads on HTTP startup, and there they sit, waiting for work.
We typically handle things by monitoring the maximum number of threads used. If this number gets near the maximum number of threads (i.e. Domino is close to having to queue requests), we up the maximum number of threads.
There is always a point, however, where upping the maximum number of threads actually hurts - i.e. the server is spending more time context switching the threads than it does actually processing their work. It’s farily simple to test performance to see if that’s the case (typically your HTTP response time under light load gets worse). In that case it’s time to consider refactoring/rearchitecting some of your app, upping your server hardware, or distributing the load to other servers.
To answer another of your questions: I have seen thread “hanging”, at least on server shutdown. Occasionally a “tell http quit” or just “exit” on the entire server will cause everything else to shutdown except HTTP. You’ll see “worker thread processing request” or something similar, over and over on the console. I’ve seen this continue for 30 minutes before forcing the server down and restarting it.
Subject: RE: HTTP Thread handling by Domino
Thanks for your respons. I wonder, how do you measure the maximum number of threads used? It’s not a statistic in Domino.
A hanging http thread would last for 4 hours until it’s killed by Domino, so if you hadn’t forced a restart on your server, you probably could have waited for quite a while still.
Haven’t you had problems with users complaining of long wait times and ‘hanging browsers’ because of hanging HTTP threads?
Subject: RE: HTTP Thread handling by Domino
We have had issues of HTTP just failing to respond for some users (or even the console!) until the server is rebooted - I’m hoping this is fixed in 6.5.4. Do a search for my name here and you’ll find a discussion on the topic.
As for guessing the number of threads used - the closest thing we can get to it is “Domino.Threads.Active.Peak”, which is a fairly accurate representation of the HTTP threads in our case because we only run Amgr,HTTP, and Update on the server, with fairly light Notes clients traffic. If that statistic hits “75”, for example, it’s reasonable for us to assume approximately 70 HTTP threads have been in use at one time.
Of course, if your server is pulling mixed duty then it may not be an accurate representation of just HTTP use.