After upgrade server 6.5.1 Hang

HCL Domino Domino Forum
1

I upgrade R5.0.11 to R6.5.1 and move to new hardware before upgrade everythink was OK. But after upgrade Domino server hang - server can’t find itself (After upgrade server work all the weekend) but in monday when utilization was “high” (100 mail user) the server hang few times a day.by notesping i can conect to server:

by www i can open html site but nsf i can’t

by notes client i can conect but can’t open server or database

domino server can replicate with another server (only push)

in domino console i receive:

“Error Error connecting to server Server1/ACME: Remote system no longer responding”

HARDWARE:

x335, Intel Xeon 2.6GHz, 512MB, 2 x IBM 36.4GB 10K-rpm Ultra320 SCSI Hot-Swap SL HDD, 2 x IBM Total Storage FAStT FC2-133 Host Bus Adapter

with

IBM FAStT600 Storage Server for Domino Data

conect by

2xIBM TotalStorage SAN Switch F08 - 8-port.

Server TASKS in notes.ini:

Replica,Router,Update,AMgr,Adminp,Sched,CalConn,McAfeeConfig,McAfeeOAScan,McAfeeODScan,McAfeeUpdate,McAfeeReport,maps,HTTP

notes.ini other parameters:

TCPIP_TcpIpAdderss=0,x.x.x.x:1352

Ports=TCPIP

PLEASE HELP ME!!! -

Sorry for my english!

2

Subject: After upgrade server 6.5.1 Hang

  1. During startup of Domino, is not there a warning about registry settings?

  2. Temporarily disable non-critical Domino tasks, like

“Update”: updates view indexes in databases, it then updates all databases that have full-text search indexes set for immediate or hourly updates.

“AMgr”: (Agent Manager) controls scheduled agents.

“CalConn”: Calendar Connector

and only for short period:

“Sched” (Schedule Manager) cares about the scheduled mail routing, replication, agents, programs.

  1. You should check which Domino task consumes the memory resource. Windows Task Manager:

nadminp.exe, namgr.exe, ncalconn.exe, nevent.exe, nreplica.exe, nrouter.exe, nsched.exe, nupdate.exe (you see the Domino task names)

Laszlo

3

Subject: RE: After upgrade server 6.5.1 Hang

does the server show any panic: message?

i had the same and it was releated to the domino directory catalog that was used. i had to deleted the fulltext folder and the database. afterwards i rebuild it and then the problem was gone

cheers

Martin

Omya AG

Information Technology Services (ITS)

Omya Messaging & Collaboration Competence Center (MC3)

CH-4665 Oftringen, Switzerland

4

Subject: RE: After upgrade server 6.5.1 Hang

No one panic message.

I removed all the fulltext folder (like in sg246889.pdf).

5

Subject: RE: After upgrade server 6.5.1 Hang

We recently upgraded to 6.5.1 and run McaFee Groupshield 5.21 version and we are having issues where the servers become unresponsive periodically. We have to 1 of 3 things to get the server back up. 1 restart port tcpip, 2 tell gsdconfig quit, 3 restart the server. server does not crash no NSD file port becomes unresponsive. Have you found any solution yet.

6

Subject: RE: After upgrade server 6.5.1 Hang

As per McAfee, you have to uninstall Group Shield and reinstall. There are agents that look for the older version of Notes, and when they cannot be located, the server hangs or shuts down.

7

Subject: RE: After upgrade server 6.5.1 Hang

Ad.1 I check all is OKAd.2 I removed all the tasks abowe and

increased mem to 1,5 GB - server works 3 hours without hang :))

Ad.3 In OS tasks Adminp takes about 500 MB when i’m checking in domino admin process was working over “Delete in ACL user test test/ACE”.

When i was watching on documents in admin4.nsf server hang again!!!

When server was hanging i was sanding few comand on the server console:

sh port tcpip - 785 open connections (few from the same ip)

sh user - near 100 users (and 45 db open)

Now i’m disable adminp and restart server load tasks abowe and i’m waiting for tomorow.

I still need HELP.

8

Subject: RE: After upgrade server 6.5.1 Hang

I would take all that McAfee junk out of your ServerTasks. Re-boot the server - see what happens. (be sure to backup original notes.ini)Have you looked for any nsd files? BTW - McAfee web site show support for “Other Software: Lotus Domino 5.0 and later (Including 6.0 & 6.5)” - I don’t see 6.5.1. You may want to check that out…Cheers!

9

Subject: Couple of corrections

CalConn is used to connect the calendar functionality between servers, whether it’s one Domino server to another, or other calendar servers. If this is one-server environment, or there are no calendar users on that server, then calconn is not needed.

Sched has nothing to do with agent manager and other scheduled server tasks. Those are handled by those tasks. Instead, Sched checks all the users calendars and updates the freetime information stored on the server in busytime.nsf.

10

Subject: One more information

After a server hang i give few commands on console:1. “sh port tcpip”

  1. “sh u”

  2. “drop all” (few times)

  3. “sh port tcpip”

ad.1: about 350 conections

ad.2: about 100 users and about 50 open db

ad.4: about 150 free “threds” and about 210 conections from many addresses was not released

Sorry for my english again

Now i’m working over configuration in McAfee 5.3 because 6.5.1 is similar to 6.5

11

Subject: After upgrade server 6.5.1 Hang

When server is going down i receive on server console:2004-03-17 14:35:34 Error connecting to server Server1/ACE: Remote system no longer responding

2004-03-17 14:35:40 Error connecting to server Server1/ACE: Remote system no longer responding

2004-03-17 14:36:52 McAfee Configuration Manager: Shutting Down

2004-03-17 14:36:53 McAfee On-Access Scanner: Shutting Down

2004-03-17 14:36:53 McAfee On-Demand Scanner: Shutting Down

2004-03-17 14:36:53 McAfee On-Demand Scanner: Shut Down

2004-03-17 14:36:53 SMTP Server: Waiting for all tasks to complete

2004-03-17 14:36:53 McAfee AutoUpdate: Shutting Down

2004-03-17 14:36:53 McAfee AutoUpdate: Shut Down

2004-03-17 14:36:54 MT Collector: Shutdown

2004-03-17 14:36:56 Database Replicator shutdown

2004-03-17 14:36:56 Starting Server shutdown

2004-03-17 14:36:57 Event Monitor shutdown

2004-03-17 14:36:57 HTTP Server: Shutdown

2004-03-17 14:37:17 McAfee Configuration Manager: Shut Down

2004-03-17 14:37:23 SMTP Server: All tasks have completed

2004-03-17 14:37:23 SMTP Server: Shutdown

2004-03-17 14:38:53 Failed to terminate one or more scan threads

BUT NO TASKS DISAPEAR FROM OPERATING SYSTEM the memory is still reserved by nsmtp, nhttp and other.

12

Subject: Looks like this did not shut down: McAfee On-Access Scanner: Shutting Down

13

Subject: RE: Looks like this did not shut down: McAfee On-Access Scanner: Shutting Down

Are you using Network Compression or Encryption on the Network Ports?

14

Subject: After upgrade server 6.5.1 Hang

I had this same problem after upgrading both my servers to LD 6.5. One was on a w2k server the other on a 2003 server. After running perfectly for a while the console would display Error Error connecting to server Server1/ACME: Remote system no longer responding where Server1 was the local machine. The console would look completly normal other than the erorr, thhe cluster replication and mail routing continued as normal. However user’s where unable to access their mail files. The problem turned out to be that the version of virus scan I was using was causing the crashes. I downloaded the new version which was updated to support ld 6.5.x and I have had the problem since.

My virus scan software is Norton but the same could happen with Mcafee.

15

Subject: RE: After upgrade server 6.5.1 Hang

I’ve seen that this problem occours to a lot of people in this forum. This is happening to me, too.

If you restart port TCPIP (type in console “restart port tcpip”), you can access again to the server.

I think this is not a solution.

In Domino R5, this never happened.

Why does it happen in D6?

Has anybody from lotus found a solution?

16

Subject: After upgrade server 6.5.1 Hang

Hello,

I have this problem too. But after two crazy days (restart server every hour, more or less) I think that I have found a clue. I have this server on a cluster. During some hours, the other server of cluster has been shutdown, and during that time, the other one has not had any problem. It’s not casual, because yesterday EVERY HOUR I had to restart the server, and when I have shutdown the other server, it has worked fine ALL NIGHT (and still it does). I think that in my case it is something related to the cluster replication, but I don’t know on what database.

If I see anything else I will write it.

Cheers.

Marian

17

Subject: RE: After upgrade server 6.5.1 Hang

Are you using Network Compression or Encryption on the Network Ports?

18

Subject: RE: After upgrade server 6.5.1 Hang

No, I have neither encryption nor compression on the TCPIP port, but I have compression on the TCPIPCLUSTER port.

Cheers.

Marian

19

Subject: After upgrade server 6.5.1 Hang

I think that I’ve solved this problem (my server had this problem every hour during this days).

The solution? I don’t know what was the action that has solved the problem, but while waiting by the Lotus response I have done the next tasks:

  • Fixup all databases in mail, mail.box and scan.box

  • Compact all databases in mail

  • Updall -R all databases in mail

  • Replaced dircat.nsf, da.nsf, names.nsf by other copy on other server.

  • Recreated log.nsf, mail.box, scan.box, clubusy.nsf (busytime.nsf)

  • Updated to 6.5.2

After all these tasks, and after one very bad week, I think that the problem is solved. I know that this is not a very good piece of information, but I haven’t seen anymore on this forum. If Lotus tells me any helful information from my debug files, I will tell you.

Hope it helps.

Marian

20

Subject: RE: After upgrade server 6.5.1 Hang

I noticed this today, you may want to test the following Fix Pack.

http://www-1.ibm.com/support/docview.wss?rs=463&q1=1188399&uid=swg21188399&loc=en_US&cs=utf-8&lang=en

Winston.