Show stopper for R5 to R6 upgrades - Intermittent connection

HCL Domino Domino Forum
1

The user complains Notes is slow to open a database (60 seconds) or if you are in a view then it is slow to move anywhere else if you have been inactive for several minutes.However while Notes is hourglassed, on the same client machine you can then open VNC (or PCAnywhere) to the same Domino server and connect immediately, but notes is still trying and will take 60 seconds or more, by then it produces a “max time” message. Try the same again and it immediately works fine. (no issue with network capacity or machine resources).

If you get hourglasses then control break, and then try to open again then Notes works fine at least until you leave it idle for a while.

There are similar posts in this forum:

“Connection problem in 6.5.2 Client?”

"Domino 6 Server “sleepy”

“Possible nasty: Horrible …”

There has never been a resolution to this issue.

From the same workstation I can connect to R5 Domino servers and NEVER have the problem. With R6 (6.5.1 currently)regardless of version I experience the problem. On the Domino newsgroup others also experience the same problem. Some say it is intermittent for all users, other say about 10% of users are affected. That is the same 10% have the intermittent issue and the 90% never complain of it, suggesting a workstation issue? That admin could not find any commonality with networks etc. for the affected users. The affected users became local replicators as a workaround.

On one Domino test server, P4, 1GByte ram etc with only two mail databases with only a few documents and no application databases, no agents running, little resources in use, but still has exactly the same problem.

I have replaced the switch and NICs with 3Com all round. I have played with MTU’s etc re:http://www.aylmer.com.au/notesvt58cj39fa.htm

The workaround is to have the users running local replicas and replicating frequently.

Windows 2000 pro and XP pro are equally affected.

I have used netstat based sniffer tools but can not determine any network issue.

So far I only have a small sample size of R6 users to test, most users will stay on R5 until this issue is resolved as they cannot function with local replicas. I do not have a mix of R5 and R6 users. All organisations are one or the other.

Many individuals users are annoyed by this issue but fail to report it, or it works OK when the Admin arrives to test, or the Admin puts it in the too hard bin.

My assumptions so far:

1/ Only R6 users are affected and all versions.

2/ It is a network related problem, but perhaps not the network per se.

3/ It may be a Lotus issue but not yet recognised as one.

4/ It may be a Notes client problem or related to the way the client installs?

5/ The issue is often misdiagnosed and never rectified partly because it is intermittent and it works when you retry.

6/ Notes is the only product on the user’s workstations that is affected, all other software products on the workstation work happily. The same applies for MTU issues, it only affects Notes and no other products that the user installs.

7/ Organisations affected by this issue may not realise it because retyring fixes it.

8/ In my experience (but not others apparently) I only have the problem when the R6 server and R6 client are on different subnets. I have never had the problem when they are both on the same subnet.

The worst thing is when you apply an update or send mail and it hourglasses, sometimes it does send the mail and others it does not so you have to repeat the process just in case it failed. Invariably the recipient ends up with two messages.

We need to nail this, any thoughts anyone?

John Aylmer

2

Subject: Show stopper for R5 to R6 upgrades - Intermittent connection

A THOUGHT.In my case, a test client on the same subnet as the servers always functions correctly. Problem machines are always the other side of the router.

As I recall when Domino installs it creates two Notes networks TCP and Netbios over TCP. By habit I always delete the Netbios over TCP. But this may be associated with the problem as Netbios can work on the server subnet but cannot get past the router without Netbios over TCP working. Apart from machine name resolution, I don’t know what other functions the Netbios over TCP could solve. DNS should satisfy any queries here I would have thought?

I don’t know if this protocol can be retro fitted or if Domino would have to be reinstalled.

Julie, In the server document, under the “Notes Network Ports” TAB, how many ports are enabled and what protocols are they. I have only one, TCPIP.

3

Subject: RE: Show stopper for R5 to R6 upgrades - Intermittent connection

Hi John

I’ve only got one port enabled, its also TCPIP.

Julie

4

Subject: Link to Julie’s previous post

http://www-10.lotus.com/ldd/nd6forum.nsf/DateAllThreadedweb/beaf911a8ae28c8885256eed005211ce?OpenDocument

Hi Julie,

I don’t normally follow this forum and just noticed your previous post on the same subject, which also references others.

John

5

Subject: RE: Link to Julie’s previous post

Why are there no more posts to this thread? Has anyone seen this?

SPR# JEIN5ZJ6GE - Fixed a performance problem where Notes would sometimes attempt to make network connections to Domino servers for the open window tabs when logging in after idle. This problem happens particularly often when using a screen saver. This regression was introduced in 6.0.

6

Subject: RE: Link to Julie’s previous post

Can someone tell me if this fix resolve client performance problem ?Because of this problem we stopped our migration plan for upgrading users to R6…

7

Subject: WE FIXED IT

Well, at least we now have a workaround. We found the problem was related to the way R6 handles New Mail Polling. (especially noticeable on slow connections/small pipes)

We’ve added this INI setting to force the ND6 client to revert to the R5 way of handling New Mail Polling:

NONEWMAILTHREAD=1

We have found that this fixes pur problem.

Good luck!

8

Subject: RE: WE FIXED IT

Didn’t work for me. After 15 min client still hour glassed or timed out. Also upgraded to 6.5.3 with no help.

9

Subject: RE: WE FIXED IT

We ended up finding that our network (10meg pipe)was causing the issue (mtu issue also). With our router processor pegging we upgraded (Newer)our router and things appear to be fixed (users reporting hour glass no more).

10

Subject: WE FIXED IT - additional ??

Are your users receiving “Delayed Write Failed” while not in Notes - say in Word or browser??

11

Subject: RE: WE FIXED IT

I think this problem should be included here in the “FIXED IT” section too. This seems to be the major fix for our similar problems.

http://www-10.lotus.com/ldd/nd6forum.nsf/55c38d716d632d9b8525689b005ba1c0/f951dc8e76419d9285256f6600620fb6?OpenDocument

http://www-10.lotus.com/ldd/nd6forum.nsf/55c38d716d632d9b8525689b005ba1c0/73926023eac90efb85256fce00563253?OpenDocument

&Gogglehead

12

Subject: RE: WE FIXED IT

We are going to give it a shot as well. I’m keeping my fingers crossed. Still working for you?

13

Subject: Show stopper for R5 to R6 upgrades - Intermittent connection

Turn on client clock debugging. This dumps a HUGE amount of data to a text file, so you want to shut down the client, edit the notes.ini to turn the debug on. Perform ONE ACTION (the db that is giving that client trouble), then shut down notes. I’ve seen clock debug files go to over 2500 actions just for a simple DB open. (that’s why it’s so slow). If you then use the SAME CLIENT, but switch ID’s…and do the same thing, you’ll notice that it takes less than 50 “clocks” to open the same DB.

You can then submit those clocks to IBM/Lotus Support.

IBM/Lotus “best guess” so far is that it is something in the unread-log of the database, that makes the client check the status of every read/unread document in the database (hence the 2500+ line clock debug files).

They don’t have a fix for it yet, but hopefully if enought people send them enough information, they can figure it out.

Also, if you open the same DB with an R5 client on an R6 or R6.5 server, the R5 client never has a problem. It’s something specific to unread marks and ND6 clients and ND6 Servers.

Also, I’ve tried turned off unread marks to no avail. Once the problem cropps up, it’s there and I have yet to find a way to get rid of it, other than creating a database COPY, but it is only a matter of time before that new copy develops the same problem, even with UNREAD marks turned off.

Here is the Debug information.

client_clock=1

debug_outfile=c:\debug.txt (it can be path and filename)

You will need to restart your client. You should test immediately as this is very verbose and will record all user activity, then shut down client and remove debug parameters.

If nothing else, it will confirm that this is/is not your issue.

14

Subject: RE: Show stopper for R5 to R6 upgrades - Intermittent connection

Just curious. Have you signed the databases in question with an ID that is completely trusted by the ECL of the user?

We had one user with a situation very similar to yours. The problem was due to the fact that the ND6.x clients checks signatures on each design element in the target db (R5 didn’t do this). In this case, this user had had a name change, so none of his folders had a signature acceptable to his client. So, each time he opened his mail, his client checked the signature on each note.

Also, we have, enterprise wide, turned off the preference setting “Automatically Refresh Inbox” - which of course would only apply to mail files. I don’t know if your problem is all mail or apps + mail.

15

Subject: RE: Show stopper for R5 to R6 upgrades - Intermittent connection

I’ve done Client Clock debugging, and it doesn’t give me or Lotus any useful information.

(1087-3528 [1087]) OPEN_DB(CN=Mail01/OU=Servers/O=BMC!!mail\badmin.

nsf): 37915 ms. [124+0=124] (Remote system no longer responding)

(1088-3566 [1088]) OPEN_DB(CN=Mail01/OU=Servers/O=BMC!!mail\badmin.

nsf): 25256 ms. [124+0=124] (Remote system no longer responding)

(1089-3592 [1089]) OPEN_DB(CN=Mail01/OU=Servers/O=BMC!!mail\badmin.

nsf): (Connect to Mail01/Servers/ABC: 20 ms) (Exch names: 0 ms)

(Authenticate: 50 ms.)

(OPEN_SESSION: 10 ms)

Also, this problem occurs for me when I do any action that opens a session on the server. For example, it happens when I hit Ctrl-O and try to open a list of databases on the server – or when I try to open the remote console using the Admin client. So I don’t think it’s related to the unread log of any database.

Do you have a PMR number that I can send to my analyst so she can compare the two incidents?

16

Subject: Show stopper for R5 to R6 upgrades - Intermittent connection

Environment: Domino R6.5.2 & Notes R6.5.1 (upgraded from R5.0.8)

After many months of troubleshooting we have finally discovered a cause / solution to one of our most significant "slow response time issues with R6.5.1. This slow response was experienced when a user would open someone’s mail file or calendar (typically an administrative assistant opening his/her executive’s calendar or mail file). It also occurred when the executive opened his/her mail or calendar after the admin had opened the file.

The Cause: If the person opening the file has only Read Public Documents access in the ACL, the Modified Date will be reset in the Colorprofile profile document. The next time someone opens the file (usually the owner will full ACL access - manager or editor), a full rebuild of the Inbox index is triggered because the color profile has been modified (date changed). This can take anywhere from 15 seconds to 5 minutes, depending on the size of the user’s file / Inbox. During our debug process, I focused on one particular user with ~20K records in his inbox and the delay was fairly consistent at ~3 minutes. The Colorprofile’s Modified Date is again changed so when the admin open the file again, a full rebuild of the Inbox index is again triggered, and slow response time is experienced.

The Temporary Solution (reference SPR MALR66ELYY - Lotus recognizes this problem as a bug and intends to fix it in a future release): Creating a “$PublicAccess” field in the Colorprofile profile document and assigning a value of “1”, allows access to the document for users with only read access, prevents the “Modified Date” from being changed, and eliminates the index rebuild. I have included some very basic code that can be used in an agent to adjust a user’s mail file. Realize that running this code will set the Modified Date so an index rebuild will occur the next time the file is opened. I ran an Updall on all of my users’ files after running the agent code to create / set the $PublicAccess field in the colorprofile docs. I would suggest putting in an IF statement to check for the existence of the $PublicAccess field first, and then check to see if the value of the field is set to a “1”. If the conditions are true, bypass the change / save so the Modified Date is not changed and unnecessary index rebuilds are not triggered. After running the initial agent, we actually put the code (with adjustments) in our mail file template so any new users would be setup properly. We will remove the code after Lotus comes up with a permanent fix.

Once we discovered the causse of the problem, I found it very re-creatable. Setting the $PublicAccess field to 0 and then opening the file from clients with full ACL access and alternately clients with Read access consistently repeated the slow response time issue when opening the DB. Setting the $PublicAccess field to 1 completely eliminated slow response time issues on the first open.

Sub Initialize

    Dim session As New NotesSession 

    Dim db As NotesDatabase 

    Dim doc As NotesDocument 

    Set db = session.CurrentDatabase 

    Set doc = db.GetProfileDocument("colorprofile") 

    Dim item As New notesitem(doc, "$PublicAccess", "1")

’ Dim item As notesitem

’ Set item = New notesitem(doc, “$PublicAccess”, “1”)

    Call doc.save(True,False)

End Sub

In addition to the “First Open” problem, we also discovered that R6.5.1 significantly increased I/O and pushed our Storage Area Network’s (SAN) performance over the edge, causing an overall sluggish in our environment. Is was suggested by one Lotus analyst that the increase in I/O when moving from R5 to R6 was ~50%, although I have not been able to verify that statement. I have ask Lotus repeatedly for documentation on the increased I/O activity, but have not received anything yet. From my experience in resolving our response time crisis, I suspect the increased I/O may be somewhat less then 50%, but It was definitely substantial enough to cause us a lot of pain. We had to rebuild all of the arrays on our SAN, splitting data into smaller arrays to regain acceptable performance. If you would like more detail, please email me.

17

Subject: Appreciation! - Show stopper for R5 to R6 upgrades - Intermittent connection

Richard -

Am researching a similar issue with degraded response times, and ran across your post.

Appreciate you taking the time to come back and post your resolution in such detail!

Thank you.

18

Subject: Here is a similar fix for Domino 7.0.3

Here is a similar fix for Domino 7.0.3 in case it’s helpful to anyone.

I had a similar problem to Richard Hill in my Domino 7.0.3 environment. An executive delegated access for his calendar to an administrative assistant. In the ACL, the assistant was listed with No Access, but she had the ability to read public documents. Each time she opened the calendar, it took 30-40 seconds. During that time she saw the error “The view is being updated. It will display as soon as the update completes. You can continue working with other Notes windows while the view is updating.”

IBM sent me Technote 1203397, which had Richard’s solution to the problem. Richard has explained it very well in this discussion forum, so please see his post about it one step up in the discussion thread.

http://www-01.ibm.com/support/docview.wss?rs=899&uid=swg21203397

Richard’s exact fix didn’t work for me. Instead of the “colorprofile” document, I had to fix the “calcolorprofile” document. This is the code that I used.

Sub Initialize

Dim session As New NotesSession

Dim db As NotesDatabase

Dim doc As NotesDocument

Dim item As NotesItem

Set db = session.CurrentDatabase

Set doc = db.GetProfileDocument(“calcolorprofile”)

Set item = doc.ReplaceItemValue(“$PublicAccess”, “1”)

Call doc.save(True,False)

End Sub

I recommend using the ReplaceItemValue method to set the field. In Richard’s original code, he created a new field. This caused problems for me because the profile document already had a field called “$PublicAccess,” which contained a null string. When I used the original code, it created another field with the same name. Two fields with the same name in one profile document are usually bad news.

My thanks go to Richard for sharing the solution. His detailed explanation of the problem led me to go back and do more testing after the steps I got from IBM didn’t work. I have been working on this problem for months and I’m sure that it would have gone on for a lot longer. He probably saved me weeks of effort.

Keywords to help other people find this solution in the forum: calendar, Notes, slow, view, updated, delegation, ACL, calcolorprofile, colorprofile, $PublicAccess.

19

Subject: RE: Show stopper for R5 to R6 upgrades - Intermittent connection

Could some kind soul please let me know exactly how to add this code to an agent? All the specifics of the agent, and ‘where’ to run it?

Thanks!

Michael

20

Subject: RE: Show stopper for R5 to R6 upgrades - Intermittent connection

Michael, did you ever figure out what you needed to do here? We have the same issue and can’t find the Colorprofile profile documents anywhere.