Hi everyone, we’re using Domino 6.53 under an IBM Regatta running AIX 2.5. The machine’s partition has 2 processors and 7Gb of RAM. There are 2000+ registered users and a peak of 800 concurrent users online.
For the past two weeks we’ve been experiencing serveral server hangs during the day. The service does not freezes completely but it stops responding for a few seconds… Analyzing the server’s console log we’ve found many occurrences of semaphore timeouts.
Well, even after reading a lot of documents on this matter we could not reach to a substantial conclusion for solving this issue. Here’s a sample of how this statistic grows in just one morning:
Sem.Timeouts = 0244:105 0931:22 11DA:4 0294:4 0939:4 030B:2 0A1B:1 03AB:1 4245:1 040D:1 0A0B:1
Sem.Timeouts = 0244:117 0931:23 030B:16 11DA:4 0294:4 0939:4 0A1B:1 03AB:1 4245:1 040D:1 0A0B:1 043D:1
Sem.Timeouts = 0244:132 0931:30 030B:20 11DA:4 0294:4 0939:4 3A08:2 0A1B:1 03AB:1 4245:1 040D:1 0A0B:1 043D:1 410F:1 3A10:1
Sem.Timeouts = 0244:132 0931:30 030B:27 11DA:4 0294:4 0939:4 3A08:2 0A1B:1 03AB:1 4245:1 040D:1 0A0B:1 043D:1 410F:1 3A10:1
We also noticed that these timeouts happens randomly through out the day. It does not appear to have any pattern at all and it is not associated to a single task either. It may happen to the Http task as well as the Agent Manager or the Cluster Replicator tasks. Here’s the output it produces on the console log:
ti=“00455920-03257037” sq=“001F2368” THREAD [610412:00011-03599] WAITING FOR RWSEM 0x3A05 Agent semaphore (@3009575C) (R=0,W=1,WRITER=610412:10538,1STREADER=610412:10538) FOR 30000 ms
00000020: 019A 0000 0000 0000 5F00 3A05 0000ti=“00455920-03257037” sq=“001F236A” THREAD [610412:00043-11823] WAITING FOR RWSEM 0x3A05 Agent semaphore (@3009575C) (R=0,W=1,WRITER=610412:10538,1STREADER=610412:10538) FOR 30000 ms
ti=“00455D08-03257037” sq=“001F2387” THREAD [610412:00012-03856] WAITING FOR RWSEM 0x3A05 Agent semaphore (@3009575C) (R=0,W=1,WRITER=610412:10538,1STREADER=610412:10538) FOR 30000 ms
00000020: 019A 0000 0000 0000 5F00 3A05 0000ti=“00455D08-03257037” sq=“001F2389” THREAD [610412:00013-04113] WAITING FOR RWSEM 0x3A05 Agent semaphore (@3009575C) (R=0,W=1,WRITER=610412:10538,1STREADER=610412:10538) FOR 30000 ms
Does anyone ever had this problem? Any sugestions are welcome…
Thanks in advance.
Best regards,
Andre