HTTP server hang

The HTTP server recently starts to just hang. I can’t type “tell http show thread state” on the server console. No web access whatsoever at that point. The only way is fix the problem to restart the entire server.

Is there a way to get HTTP log more verbose on the server console so it’s recorded in the log.nsf?

Is there a way to kill just the HTTP and restart it without bring down the entire server?

Is there any tool to monitor what the HTTP server is doing?

I’d appreciated if someone can direct me where to start troubleshoot the problem.

Thanks!

Subject: RE: HTTP server hang

Hi Raymond,

Just a question what you run in your server? INotes? Web Application? If you run Web Application you use java in your application?



The things below provide better managing and monitoring your HTTP task, so you can do:
  1. Edit the Server document, Basics tab and Fault Recovery section you can check the options:

Automatically Restart Server After Fault/Crash:

(This option try up any service if they down) 
  1. Edit the Configuration Document, Diagnostics Tab, and in the “Mail-in Database for diagnostic reports:” field select the “Lotus Notes/Domino Fault Reports” database.

    You can check also if you have in the %Drive%\Lotus\Domino\Data\IBM_TECHNICAL_SUPPORT path any .log file, this file generaly is a dump of the crashes.

    Let us know more informations about our server for we could help you better.

Regards,

Marce M

Subject: RE: HTTP server hang

Thanks! We don’t use iNotes. All web applications serving up web pages.

The server does not crash, just the HTTP hangs. Can’t get it to “tell http restart”, “tell http q”, etc. But, all the other components are working just fine. I do have fault recovery turned on but the server is not crashing so Fault recovery does not kick-in. I can’t get my finger on what’s going on.

I do have a java agent that get invoked when the homepage is open. It does a InputStreamReader from another web site’s HTML page. But, that was working fine until the recent hang. You think java has something to do with it?

Subject: RE: HTTP server hang

Are you using 6.5.0? I never deployed 6.5.0 because of some issues it had, but I had continuous problems like what you’re describing on 6.5.2FP1 every 2-3 days, and usually around 2AM - 4AM, when the event monitor was doing housekeeping. Here’s what I did:

  1. Upgraded to 6.5.4 – that helped a LOT. 6.5.4FP1 is even better. I still had the occasional hang every 2-3 weeks, though, so I…

  2. Ran an NSD each time the hang took place, and analyzing the dumps showed my events.nsf database and admin.nsf databases were usually open. I thought I might have a bad replica somewhere in there, so I re-created both.

Things have been great since then.

Subject: RE: HTTP server hang

Yes, I’m running 6.5.0. I think I’ll do what you suggest. Should I just go ahead upgrade to 6.5.5? Or 6.5.4 FP2?

How do you run NSD? How does this work? Can you describe more?

Thanks!

Subject: RE: HTTP server hang

I would definitely suggest upgrading – 6.5.0 is a couple of years old at this point – but always test before upgrading your servers.

I would either test 6.5.5 or 6.5.4FP2. 6.5.5 is the “latest and greatest” but just came out, whereas 6.5.4FP2 has been out for awhile and many have had great results with it.

If upgrading fixes your problem don’t worry about the NSD, but if you are still crashing I would open a ticket with Lotus Support.

Subject: RE: HTTP server hang

First go to Server Document / Internet Protocols / HTTP tab and fin Number active threads field, let us know the value is used.

Another thing when your HTTP hangs try this command:

tell http show thread state

With this you can see if threads is full at the moment of hang.

Put the variables below in your notes.ini, with this you can see how to working your java environment.

JavaVerbose=1

JavaVerboseGC=1

And you can put the JavaMaxHeapSize variable in your notes.ini.



This variable specifies the maximum size the Java heap can reach. You can specify the number of bytes directly or use the suffix MB to indicate megabytes. For example, specifying 64 MB is the same as specifying 67108864.

for you set it:

JavaMaxHeapSize=

Subject: RE: HTTP server hang

I have the active threads set at 100 now.

The problem is when the HTTP hangs, I can’t even issue “tell http show thread state”!

I’ll try the java variable suggestion. Thanks!

Subject: I just saw this, too… Re: HTTP server hang

This might also be related to the issue you are having.

SPR # DTEI6AYCCW

Fixed in release:

6.5.5
6.5.4 FP2

Regression in 6.5.2

Product Area: Server Technical Area: Java Platform: Cross Platform

Lotus Customer Support Technote: 1208479

SPR# DTEI6AYCCW - This fix is to ensure that a semaphore is released on termination. After calling a Web query Java agent, any subsequent call to the same or other Web agents hung the HTTP task. When you quit the Lotus Domino server or use the command ‘tell http show thread state,’ you notice the hanging HTTP requests. This regression was introduced in 6.5.2.