Webmail and http database access not responding, http server will not shut down

Last week our Domino sever was powered off due to a power cut in our server room, ever since users cannot access their mail via webmail at the same time users cannot access databases via a web browser either.

Sometimes users may get lucky and are able to get into their mail, but the majority of the time it is saying “waiting for Apollo (our server name)” in the status bar of Internet Browsers. It does not time out, I have left it for hours and the status bar message is the same.

When I try and restart the http server in the Admin Console I get the following messages appear.

01/13/2009 01:08:55 PM HTTP Server: Waiting for session to finish: Session State [Processing Request] Session ID [2522] Thread ID [2680601488 (0x9fc6bb90)] GET /mail/monions.nsf/($Inbox)/363053607FB07B8D80257536004F881E/?OpenDocument&PresetFields=s_View Name;(%24Inbox),h_FolderStorage;(%24Inbox),s_FromMail;1,s_SortBy;4,s_UnreadOnly;0,s_NoMarkRead;1,s_NoAdjWin;1&Form=h_PreviewUI HTTP/1.1

01/13/2009 01:08:55 PM HTTP Server: Waiting for session to finish: Session State [Processing Request] Session ID [2523] Thread ID [2680073104 (0x9fbeab90)] GET /mail/monions.nsf/($Inbox)/363053607FB07B8D80257536004F881E/?OpenDocument&PresetFields=s_View Name;(%24Inbox),h_FolderStorage;(%24Inbox),s_FromMail;1,s_SortBy;4,s_UnreadOnly;0,s_NoMarkRead;1 HTTP/1.1

01/13/2009 01:08:55 PM HTTP Server: Waiting for session to finish: Session State [Processing Request] Session ID [2524] Thread ID [2679544720 (0x9fb69b90)] GET /mail/monions.nsf/($Inbox)/AA675CFF9CCB66FA80257536004E86D5/?OpenDocument&PresetFields=s_View Name;(%24Inbox),h_FolderStorage;(%24Inbox),s_FromMail;1,s_SortBy;4,s_UnreadOnly;0,s_NoMarkRead;1,s_NoAdjWin;1&Form=h_PreviewUI HTTP/1.1

01/13/2009 01:08:55 PM HTTP Server: Waiting for session to finish: Session State [Processing Request] Session ID [2525] Thread ID [2679016336 (0x9fae8b90)] GET /mail/monions.nsf/($Inbox)/AA675CFF9CCB66FA80257536004E86D5/?OpenDocument&PresetFields=s_View Name;(%24Inbox),h_FolderStorage;(%24Inbox),s_FromMail;1,s_SortBy;4,s_UnreadOnly;0,s_NoMarkRead;1 HTTP/1.1

01/13/2009 01:08:55 PM HTTP Server: Waiting for session to finish: Session State [Processing Request] Session ID [2535] Thread ID [2673732496 (0x9f5deb90)] GET /mail/monions.nsf/($Inbox)/363053607FB07B8D80257536004F881E/?OpenDocument&PresetFields=s_View Name;(%24Inbox),h_FolderStorage;(%24Inbox),s_FromMail;1,s_SortBy;4,s_UnreadOnly;0,s_NoMarkRead;1,s_NoAdjWin;1&Form=h_PreviewUI HTTP/1.1

01/13/2009 01:08:55 PM HTTP Server: Waiting for session to finish: Session State [Processing Request] Session ID [2536] Thread ID [2673204112 (0x9f55db90)] GET /mail/monions.nsf/($Inbox)/363053607FB07B8D80257536004F881E/?OpenDocument&PresetFields=s_View Name;(%24Inbox),h_FolderStorage;(%24Inbox),s_FromMail;1,s_SortBy;4,s_UnreadOnly;0,s_NoMarkRead;1 HTTP/1.1

01/13/2009 01:08:55 PM HTTP Server: Waiting for session to finish: Session State [Processing Request] Session ID [2537] Thread ID [2672675728 (0x9f4dcb90)] GET /mail/monions.nsf/($Inbox)/AA675CFF9CCB66FA80257536004E86D5/?OpenDocument&PresetFields=s_View Name;(%24Inbox),h_FolderStorage;(%24Inbox),s_FromMail;1,s_SortBy;4,s_UnreadOnly;0,s_NoMarkRead;1,s_NoAdjWin;1&Form=h_PreviewUI HTTP/1.1

01/13/2009 01:08:55 PM HTTP Server: Waiting for session to finish: Session State [Processing Request] Session ID [2651] Thread ID [2675846032 (0x9f7e2b90)] GET /mail/igore.nsf?Logout&Form=s_LogoutDone HTTP/1.1

01/13/2009 01:08:55 PM HTTP Server: Waiting for session to finish: Session State [Processing Request] Session ID [3246] Thread ID [2678487952 (0x9fa67b90)] GET /mail/jsteed.nsf?Logout&Form=s_LogoutDone HTTP/1.1

01/13/2009 01:08:55 PM HTTP Server: Waiting for session to finish: Session State [Processing Request] Session ID [3293] Thread ID [2674789264 (0x9f6e0b90)] GET /mail/levans.nsf?Logout&Form=s_LogoutDone HTTP/1.1

I have tried restarting Domino ( nsd – kill ) and rebooting the server but either did not rectify the problem

Domino has been running without any problems for about a year, but last week we had a UPS failure that caused the server to be shutdown unexpectedly.

Any help or advice on this problem will be very much appreciated.

Kind regards

Nick

Subject: check hardware, call support

With the power outage and then this happening, and no other changes and rebooting doesn’t help…I would wonder if some hardware was affected. I advise checking the memory and the network card of the server to be sure both are working optimally. Or, if possible, switch them out.

After that, I would say call support. This sounds like a unique issue that would require tech support.

Subject: Not hardware related

Thank you for your response. I do not think the hardware is the issue, Domino is running on the Virtual Machine, and there are 9 other servers that are running fine on the same hardware.

I mentioned the power outage as I thought that domino may have some sort of file lock or temporary files that may need deleting.

On the other hand the power cut could be completely unrelated and it is a coincidence the http server has stopped working .

Subject: file lock, etc.

The file lock is totally possible. We had a server that needed this cleared after a serious problem once, and Lotus support supplied a tool for doing so. I recommend contacting support, as your issue sounds unique to your particular circumstance. Sorry.

Subject: Database Maintenances

Hi there, from the logs it seems like those databases listed there are having some issues due to the abnormal shutdown. Please perform database maintenance (either online or offline) on the list of stated databases (best perform maintenance for all the databases including address book). After which, test on the HTTP portion again.

Any FP installed for the domino version? sometimes recreating new copies of the problematic databases will resolve those issues as well.