Semaphore Heaven

hcl-bot · September 9, 2009, 7:48pm

I recently applied Fix Pack 1 to an 8.5 Windows 2003 Server and the following day the server was hung with Semaphore locks (doesnt make me look good). After bouncing the server its now going through consistency checking all the data and its been doing this for 2 hours! DOH, why dont we use transaction logging???

Anyway Im looking for any advice on troubleshooting or assisting this issue. I know the server is pretty heavily fragmented but we ran a defrag a few weeks ago and the disk was still heavily fragmented afterwards. Im not sure another defrag will save us.

ti=“0003F223-CA25762D” sq=“0000320D” THREAD [0A24:008D-0F68] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010DB5A3) (R=0,W=1,WRITER=0A24:0574,OWNER=0A24:0574) FOR 30000 ms

ti=“0003F224-CA25762D” sq=“0000320E” THREAD [0A24:0087-0E5C] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010DB5A3) (R=0,W=1,WRITER=0A24:0574,OWNER=0A24:0574) FOR 30000 ms

ti=“0003F224-CA25762D” sq=“0000320F” THREAD [0A24:008C-0A64] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010DB5A3) (R=0,W=1,WRITER=0A24:0574,OWNER=0A24:0574) FOR 30000 ms

ti=“0003F226-CA25762D” sq=“00003210” THREAD [0A24:008A-0DAC] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010DB5A3) (R=0,W=1,WRITER=0A24:0574,OWNER=0A24:0574) FOR 30000 ms

ti=“0003F228-CA25762D” sq=“00003211” THREAD [0A24:04E3-0864] WAITING FOR WRITE LOCK ON RWSEM 0x410F (@00F1277A) (R=0,W=1,WRITER=099C:0B0C,OWNER=099C:0B0C) FOR 30000 ms

ti=“0003F248-CA25762D” sq=“00003212” THREAD [0A24:006A-0248] WAITING FOR WRITE LOCK ON RWSEM 0x410F (@00F1277A) (R=0,W=1,WRITER=099C:0B0C,OWNER=099C:0B0C) FOR 30000 ms

ti=“0003F425-CA25762D” sq=“00003213” THREAD [0A24:07D0-081C] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010DB5A3) (R=0,W=1,WRITER=0A24:0574,OWNER=0A24:0574) FOR 30000 ms

ti=“0003F5A2-CA25762D” sq=“00003214” THREAD [0A24:00B5-0E6C] WAITING FOR WRITE LOCK ON RWSEM 0x410F (@00F1277A) (R=0,W=1,WRITER=099C:0B0C,OWNER=099C:0B0C) FOR 30000 ms

ti=“0003F5D1-CA25762D” sq=“00003215” THREAD [0A24:05B6-0870] WAITING FOR WRITE LOCK ON RWSEM 0x410F (@00F1277A) (R=0,W=1,WRITER=099C:0B0C,OWNER=099C:0B0C) FOR 30000 ms

ti=“0003F617-CA25762D” sq=“00003216” THREAD [0A24:008E-0F64] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@3AD0AB4F) (R=0,W=1,WRITER=0A24:0EA0,OWNER=0A24:0EA0) FOR 30000 ms

ti=“0003F62C-CA25762D” sq=“00003217” THREAD [0A24:0089-0A5C] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@3AD0AB4F) (R=0,W=1,WRITER=0A24:0EA0,OWNER=0A24:0EA0) FOR 30000 ms

ti=“0003F6F4-CA25762D” sq=“00003218” THREAD [0A24:0589-0414] WAITING FOR WRITE LOCK ON RWSEM 0x410F (@00F1277A) (R=0,W=1,WRITER=099C:0B0C,OWNER=099C:0B0C) FOR 30000 ms

ti=“0003F7BA-CA25762D” sq=“00003219” THREAD [0A24:0532-0508] WAITING FOR WRITE LOCK ON RWSEM 0x410F (@00F1277A) (R=0,W=1,WRITER=099C:0B0C,OWNER=099C:0B0C) FOR 30000 ms

ti=“0003FADD-CA25762D” sq=“0000321A” THREAD [0A24:008B-0A60] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010595A3) (R=0,W=1,WRITER=0E4C:0E50,OWNER=0E4C:0E50) FOR 30000 ms

ti=“0003FAE0-CA25762D” sq=“0000321B” THREAD [0A24:0540-03A0] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010595A3) (R=0,W=1,WRITER=0E4C:0E50,OWNER=0E4C:0E50) FOR 30000 ms

ti=“0003FB0E-CA25762D” sq=“0000321C” THREAD [0A24:06FC-07D8] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010595A3) (R=0,W=1,WRITER=0E4C:0E50,OWNER=0E4C:0E50) FOR 30000 ms

ti=“0003FB0E-CA25762D” sq=“0000321D” THREAD [0A24:012B-0824] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010595A3) (R=0,W=1,WRITER=0E4C:0E50,OWNER=0E4C:0E50) FOR 30000 ms

ti=“0003FB17-CA25762D” sq=“0000321E” THREAD [0A24:03FC-0834] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010595A3) (R=0,W=1,WRITER=0E4C:0E50,OWNER=0E4C:0E50) FOR 30000 ms

ti=“0003FB27-CA25762D” sq=“0000321F” THREAD [0A24:0565-0844] WAITING FOR WRITE LOCK ON RWSEM 0x410F (@00F1277A) (R=0,W=1,WRITER=099C:0B0C,OWNER=099C:0B0C) FOR 30000 ms

ti=“0003FB35-CA25762D” sq=“00003220” THREAD [0F7C:0002-019C] WAITING FOR SEM 0x010D (@010707AC) (OWNER=0928:092C) FOR 30000 ms

ti=“0003FB38-CA25762D” sq=“00003221” THREAD [0A24:0203-041C] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010595A3) (R=0,W=1,WRITER=0E4C:0E50,OWNER=0E4C:0E50) FOR 30000 ms

ti=“0003FB46-CA25762D” sq=“00003222” THREAD [0A24:02C4-0260] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010595A3) (R=0,W=1,WRITER=0E4C:0E50,OWNER=0E4C:0E50) FOR 30000 ms

ti=“0003FB62-CA25762D” sq=“00003223” THREAD [0A24:0250-0E9C] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010595A3) (R=0,W=1,WRITER=0E4C:0E50,OWNER=0E4C:0E50) FOR 30000 ms

ti=“0003FB62-CA25762D” sq=“00003224” THREAD [0A24:0182-0358] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010595A3) (R=0,W=1,WRITER=0E4C:0E50,OWNER=0E4C:0E50) FOR 30000 ms

ti=“0003FB68-CA25762D” sq=“00003225” THREAD [0A24:025A-0808] WAITING FOR WRITE LOCK ON RWSEM 0x02BE Per-database file descriptor group semaphore (@010595A3) (R=0,W=1,WRITER=0E4C:0E50,OWNER=0E4C:0E50) FOR 30000 ms

ti=“0003FC70-CA25762D” sq=“00003226” THREAD [0A24:0224-067C] WAITING FOR WRITE LOCK ON RWSEM 0x410F (@00F1277A) (R=0,W=1,WRITER=099C:0B0C,OWNER=099C:0B0C) FOR 30000 ms

ti=“0003FC82-CA25762D” sq=“00003227” THREAD [0A24:000C-0ADC] WAITING FOR WRITE LOCK ON FRWSEM 0x0256 NSF buffer pool container (@2E5720FA) (R=8,W=0,WRITER=0000:0000,1STREADER=0A24:0344) FOR 30000 ms

ti=“0003FD8E-CA25762D” sq=“00003228” THREAD [0A24:0385-07AC] WAITING FOR WRITE LOCK ON RWSEM 0x410F (@00F1277A) (R=0,W=1,WRITER=099C:0B0C,OWNER=099C:0B0C) FOR 30000 ms

ti=“0003FDEC-CA25762D” sq=“00003229” THREAD [0A24:0065-0888] WAITING FOR SEM 0x0944 Task table entry semaphore (@2611554F) (OWNER=0A24:0824) FOR 30000 ms

ti=“0003FFA9-CA25762D” sq=“0000322A” THREAD [08E4:0017-0C64] WAITING FOR SEM 0x017D (@01199DC4) (OWNER=0A24:0AEC) FOR 30000 ms

hcl-bot · September 9, 2009, 8:48pm

Subject: Looks similar

Looks similar

Error: …‘WAITING FOR READ LOCK ON FRWSEM’…using Server Statistic Collection document

http://www-01.ibm.com/support/docview.wss?uid=swg21315806

JYR

hcl-bot · September 10, 2009, 6:58pm

Subject: Thanks

Thanks for that. We took the server down and removed Fix Pack 1, added that line to the ini and renamed the log and brought the server back up and it appears to be ok now. Logged a PMR but haven’t received anything valid from IBM yet.

hcl-bot · September 10, 2009, 7:58pm

Subject: Semaphore Heaven…

If you have anything from IBM. please post the result to help others…

JYR

hcl-bot · September 23, 2009, 8:44pm

Subject: Response

We did not have transaction logging enabled on this server and a consistency check was running when a number of processes was trying to access the server. We fixed the issue by blocking off access to the server “server_restricted=1”, removing tasks and starting the server up. Once the consistency check had finished we manually started tasks and opened up access to the server again.

Heres part of the response from IBM…

The semdebug log shows DATABASE SEMAPHORES for many databases indicating

that these databases were locked by some process for a long time.

Extracts from semdebug:

=====================

09-09-2009 11:26:38 PM ZE10 sq=“000008FE” THREAD [060C:0002-092C] WAITING

FOR READ LOCK ON FRWSEM 0x0244 database semaphore (@0E52A07F) (D:\Lotus

\Domino\Data\mail*.nsf) (R=0,W=1,WRITER=0DE4:0F2C,1STREADER=0000:0000)

FOR 30000 ms

09-09-2009 11:27:07 PM ZE10 sq=“00000902” THREAD [0DE4:0071-0E7C] WAITING

FOR WRITE LOCK ON FRWSEM 0x0244 database semaphore (@2885E2E3) (D:\Lotus

\Domino\Data\mail*.nsf)

(R=2,W=0,WRITER=0000:0000,1STREADER=0938:093C) FOR 30000 ms

10-09-2009 08:25:32 AM ZE10 sq=“000058FC” THREAD [0DE4:000C-049C] WAITING

FOR READ LOCK ON FRWSEM 0x0244 database semaphore (@0105AEA7) (D:\Lotus

\Domino\Data\names.nsf) (R=1,W=0,WRITER=0000:0000,1STREADER=0DE4:054C) FOR

30000 ms

10-09-2009 08:25:35 AM ZE10 sq=“000058FD” THREAD [0DE4:00F4-094C] WAITING

FOR READ LOCK ON FRWSEM 0x0244 database semaphore (@01058EA7) (D:\Lotus

\Domino\Data\log.nsf) (R=1,W=0,WRITER=0000:0000,1STREADER=0DE4:0678) FOR

30000 ms

10-09-2009 08:20:31 AM ZE10 sq=“0000581D” THREAD [0184:0008-0918] WAITING

FOR READ LOCK ON FRWSEM 0x0244 database semaphore (@213762E3) (D:\Lotus

\Domino\Data\mail*.nsf)

(R=0,W=1,WRITER=1134:1208,1STREADER=0000:0000) FOR 30000 ms

The Log.nsf indicates that consistency check was being performed on these

databases at that time.

These databases could be large sized dbs. Also, some other processes such

as adminp, agent, etc. were trying to access these databases.

You may run the below offline maintenance tasks on all the databases.

       fixup -F                (-J, instead of -F if Transaction

logging is enabled.)

    updall -R -X

          compact -c -i

Examples from Log.nsf:

====================

09/09/2009 11:01:07 PM Updating took over 30 minutes for mail*.nsf

10/09/2009 01:23:11 AM Updating took over 30 minutes for mail*.nsf

10/09/2009 03:20:37 AM Updating took over 30 minutes for mail*.nsf

10/09/2009 06:39:58 AM Admin Process: Path: mail*.nsf: This

database cannot be opened because a consistency check of it is in progress.

Monitor.Last.Server.Warning(High)Text = Unable to write to database -

database D:\Lotus\Domino\Data\mail*.nsf would exceed its disk quota of

204800 KB by 1 GB.

Admin Process Performing batched ACL modification on [mail

*.nsf] for 1 users