Slowness

I’d just like to get a feedback from people on whether or not they are experiencing any slowness issues with 8.5 that didn’t seem to be there for earlier releases?

I’m starting to pull my hair out with this release…we have over 500 users, 10 clustered servers globally and all I hear is frustration on the end users part. I know bandwidth, hops, configurations and all the other network buzz words apply here, but I’m getting more and more complaints since rolling out 8.5. It was never like this with 7.

Most of the complaints come from trying to hit a server outside your main office…do file, open, lotus notes application and try to hit a server outside my home office. 8 times out of 10 it will sit there and clock, saying connecting to “server name via tcpip”. And it will just clock and clock. I do a ctrl+break (which why doesn’t that work all the time anymore? For some users it doesn’t work at all). Sometimes it will stop and that’s fine, so I’ll hit enter because it has the correct server name in the box and BOOM, it opens the directory right up. So it connects fine…now try opening a mail file, or a large db…it will clock and clock there as well. Sometimes for over 3 minutes if it’s a local db, and by local I mean for me, in the US. The other night I tried to open a mail file on our server in Singapore…It connected to the server right away. Want to know how long it took to actually open the inbox view of the mail file? Well, me, too. After 15 minutes, I finally did a ctrl+brk. Which stopped it, but didn’t stop the client from trying to connect to the server again and again. I had to do a ctrl+brk 4 or 5 times before it finally stopped trying to reconnect to the Singapore server. And eventually, because I couldn’t get it to refresh my client I had to use KillNotes.exe and then re-launch notes. It never did open the mail file I was trying to open on the Singapore server…even after waiting over 15 minutes a second time.

My deal is that I’m a notes administrator…our company has made a conscious decision to use notes. I’m a backer, I’m a supporter, I’m a grass roots guy…I love notes, in other words, because I don’t think people fully respect what it’s capable of, but who’s fault is that? . One of your better blunders IBM, and that’s no joke!!

But when my end users start raising a stink to executive management about how slow it is and how they can’t get their work done and how opening a simple email is causing delays and outlook is much faster at my old company and everything else, I start to wonder where the direction is coming from? We use Notes for EVERYTHING and I do mean EVERYTHING…payroll, hr, ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccwxfinancccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc

Subject: The forever connection syndrome

I’ve seen a similar issue of network spinning with Notes 7 clients hitting Domino 7 servers. Here’s what we typically see:

  1. User has had their email up all day. They’re in their inbox, they press F9 to refresh the view.

  2. Lightning bolt lights up steady, indicating waiting on something with the network.

  3. User waits for 2-3 minutes, calls me over.

  4. I walk over, remote console into mail server. It’s just sitting there – CPU is low, disk is low, nothing going on.

  5. I hit control-break on the user’s client. IMMEDIATELY the “new mail” icon appears in their systray and a new, unread email appears in their inbox. Subsequent F9s are lightning quick.

The sad part is these clients and servers are on local 1 gigabit ethernet, very vanilla setups.

I think there’s some suspicious semaphore deadlock going on there, and it wasn’t around (or wasn’t exposed) in R5 or ND6. I’ve encountered this myself on my own workstation, and it’s definitely frustrating. I wonder if it’s worse in 8.x?

I’ve seen it connecting via Notes 7 clients to these forums dbs in Notes also.

Subject: I think it’s definitely worse in 8…

I always noticed it in 7 as well, but the frequency seems to have increased with 8. I know the frustration level has…

Subject: Please take a few nsds on Domino Server when you see this slowness

Did you run updall on all you databases after upgraded to Domino 8.5 and before restarting Domino Server? There will be a 1 time rebuild of all view indexes that needs to occur… If this is not done before starting the Domino Server than user’s will end up waiting for the views to be rebuilt as they access them… .This is a one time hit…

Please Open a PMR with Lotus Support about the performance issue and send them the nsds.

Send them to me directly as well and I’ll see if anything jumps out at me from the call stacks.

jpaganet@us.ibm.com

Subject: I already have a PMR open with IBM…

Which I’ve subsequently closed because all the console logging and debugging actually slowed the server down. Turns out that might have been a firewall issue and it might have been coincidental that everyone complained about it being even more slow than normal on that day.

Anyway, it’s really not the servers that are slow…you can be sitting in the office where the corp server is at and it’s fast, but it has that occasional “clocking” as mentioned above that drives everyone nuts. That being said if you try to hit the servers in our data center or the ones that are located overseas, it can be very slow and almost painful trying to open a mail file.

And to answer your question, no, updall wasn’t run on all the databases after the upgrade and before starting the server…we upgraded the servers first, then upgraded the clients by office. So am I supposed to shut down the server every time and run updall?

Subject: Debugging on the client.

You can use the following debug on the clients notes.ini

DEBUG_THREADID=1

CLIENT_CLOCK=1

CONSOLE_LOG_ENABLED=1

This will give further information on what your client is up to when trying to make the connection.

Subject: I see that you had opened a PMR with IBM…

earlier this week. However, the level of detail you have provided here is more than what was supplied in the PMR.

Please work with Support, and based on your post here, focus on looking at establishing initial connectivity with a server. It there is a delay in connecting the server, clearly many other actions are affected.

It sounds like the initial connection is failing under some circumstances, and going through a number of timeouts before failing. The Ctrl-Break does cause a drop in all connections, looking at the connection via Preferences\Port\Trace may provide additional insight in the bad and good cases.

Doug Conmy
Lotus Notes Lead Architect