We upgraded 5 notes servers in branch offices with out an issue except the sending of emails with large attachments from Dell Optiplex 240s. Once the upgrade was performed these machines went from sending emails with 5mb attachments in a few seconds to over 60 seconds. This is consistant in each branch office that was upgraded.
In the Notes client debug log the average ms during the object to a server in the same room.
(205-1171 [344]) WRITE_OBJECT(REP85256F1B:0007E887-RRV005DADFE,0x8000 at 0x78000): 250 ms. [32834+12=32846]
(206-1171 [345]) WRITE_OBJECT(REP85256F1B:0007E887-RRV005DADFE,0x8000 at 0x80000): 235 ms. [32834+12=32846]
(207-1172 [346]) WRITE_OBJECT(REP85256F1B:0007E887-RRV005DADFE,0x8000 at 0x88000): 359 ms. [32834+12=32846]
The same information is appearing in a packet capture.
When the same attachment is copied using windows copy between an affected machine and the new upgraded server the time to copy is 2 to three seconds total. No other system experiencing any performance related issues.
All non 240s work fine.
Below is a list of things we have tried to do to identify the issue with no luck.
Problem
After replacing the server hardware and upgrading from 6.5.3 6.5.5 fp 2 in the Duluth office (also affects Seattle and Austin) that had the same upgrade, many users experiencing extreme slowness when hitting the send button with emails containing attachments of 1+MB or greater. The problem happens when forwarding, replying or creating the mail from scratch. Time span ranged from 40 - 90 seconds depending on the size of the attachment.
After much troubleshooting, problem has been narrowed down to 1 specific model machine the Dell GX 240s which are the dominant machine in the these offices.
PMR34515,L6Q has been (and still is) open with Lotus. carolyn has record of all correspondence from IBM.
Troubleshooting Recap
Current Status
IBM ticket is still open, however they are not providing much in the way of support.
Old DU server being shipped back to Delaware, it will be reimaged and setup to be TestServerB temporarily for testing. Will create a new domino server instance and let Joe test this upgraded machine in Cherry Hill against some GX240 machines he has there. Cherry HIll will not be upgraded until this issue is resolved.
Test PC that was on site at Duluth is being shipped back to Cherry Hill. The hard drive went bad during testing, so Joe will get it reimaged, try an Intel Network Card for testing, and then he will send it to Delaware for testing against our TESTSERVERC upgraded server.
Affected Office Cherry Hill IT Staff along with Carolyn have performed the following testing.
Recap from Carolyn
Client testing started with 6.5.3 and is now at 6.5.5 and no change.
Carolyn reviewed client configuration (connection, location, ini settings). Found one instance where the location document was not set to the home mail server as fully qualified name but rather Affected server , made the change, tested, but same outcome.
Joe and I tested turning off the Trend PC Firewall - this is the ONLY change that had an positive impact on the problem although still slow
Compared the old server .ini and the new server .ini - no major differences that would impact the issue.
Turned on server debugging with IBM Support and logs showed nothing wrong fromt he server side.
Tech placed a different (3COM) network card in Kristina Bailey’s machine and the performance actually got worse.
Recap from Jeff
Machines in DU sent the same 5Mb mail attachment message to test transit from almost every machine in DU office.
All systems of the machine model Dell Optiplex GX240 exhibited the slowness in transmitting the message.
On specific machines the attachments were sent from the location C:\Temp\Test.
Sending message internally in the office or externally to the office exhibited the same speed in attachment transmit.
We were provided the instructions to add debug logging options to the NOTES.INI.
CLIENT_CLOCK=1
DEBUG_OUTFILE=C:\OUTFILE.TXT
With this options in place we created debug logs from machines at two separate locations with separate notes servers and network switch generated the same slowness on Dell GX240 systems.
We have performed these tests as multiple users from same machine and same user from multiple machines including Dell GX240, Dell GX260 and Dell GX270.
Same results Dell GX240 displays slowness sending message with attachment(s).
In regard to the Dell GX240 systems we have tried later generations of the 3Com 3C90X network driver, forcing speed and duplex on NIC and switch port as well as moving to alternate switch ports.
We have upgraded to later editions of Dell GX240 BIOS.
We brought a system image used in the Good Office office loaded unto an Affected Office Dell GX240 installed mandated corporate software (WholeSecurity and Trend Micro AV) and performed test with debug options against server in CH successfully.
Shipped Dell GX240 with Good Office image to DU and performed test with debug options against server in DU and received the slowness transmitting message.
Honestly we do not know where to go from here other then reproduce the results we are seeing in our home office. Replacing 180 computers is not and option but is the only think we know will work at this time after significant effort and time spent