6.5.1 server crashes every 3-4 hours

hcl-bot · April 26, 2004, 6:05am

6.5 was crashing every 3-4 hours… like this:

-rw-r–r-- 1 notes notes 8512557 Apr 26 06:40 nsd_all_SUNSPA_sweb01_04_26@06_32.log

-rw-r–r-- 1 notes notes 8540096 Apr 26 03:33 nsd_all_SUNSPA_sweb01_04_26@03_24.log

-rw-r–r-- 1 notes notes 9187989 Apr 24 08:11 nsd_all_SUNSPA_sweb01_04_24@08_00.log

…

-So I reinstalled fresh 6.5.1.

-Deleted all system dbs so server would recreate; ie. log.nsf, events4.nsf, etc

Did a 20+ hours fixup -j, compact -c, updall -r session. All DBs are ODS43.
Copied the names.nsf and notes.ini from a working cluster member

Still crashing every 3-4 hours. This is what I see:

Stack base = 0xf7b2230c, Stack size = 6808 bytes

Fatal Error signal = 0x0000000b PID/TID = 2022/63

Mon Apr 26 06:32:14 Fault recovery is in progress

Mon Apr 26 06:32:14 Running NSD

NSD is in progress …

Please attach the following files to your bug report along with the server log:

Log file : /extdata01/nsd_all_SUNSPA_sweb01_04_26@06_32.log

Mon Apr 26 06:40:59 Terminating tasks

Here is that 2022/63 thread:

###################################

[1] ff2978d0 nanosleep (f7b1fe18, f7b1fe20)

[2] ff0baedc _ti__nanosleep (1, 7fffffff, 40, f7b202b0, ff0d5980, ff0ce000) + 34

[3] fda098c0 OSRunExternalScript (f7b203dc, fed30eec, ff089e50, 0, 800, 125c00) + 1dc

[4] fda08080 OSFaultCleanup (0, 0, fc400000, fede3f24, fede3f24, 5) + 460

[5] fd9e48cc fatal_error (b, f7b20ed8, f7b20c20, ff0ce000, f7b21e10, f7b21e00) + 1a4

[6] ff0b90b0 __libthread_segvhdlr (b, f7b20ed8, f7b20c20, ff0ce000, b, 0) + e4

[7] ff0bbc20 __sighndlr (b, f7b20ed8, f7b20c20, ff0b8fcc, f7b21e10, f7b21e00) + c

[8] ff0b8424 sigacthandler (b, f7b21d78, f7b20c20, ff0ce000, f7b21d78, f7b20ed8) + 6e0

[9] — called from signal handler with signal 11 (SIGSEGV) —

[10] fda14ebc OSPathResolve (103, 0, 0, 0, 0, 0) + 174

###################################

This doesn’t seem to tell me too much other than the fact that the server crashed.

I noticed that the rest of the SERVER threads (mostly) looked the same… something about waitonnativesemaphore… maybe this is a clue???

I have all the logs if anyone can help.

We are getting desperate and willing to pay ($$) if a consultant/freelancer wants to help solve this. We’ll be opening an IBM support call otherwise… but…

Damian

800-661-4269x88

damian@svothi.com

For reference, all the server threads:

##############################################################

prog: /opt/dominor6/lotus/notes/latest/sunspa/server pid(s) : 2022

======== Attaching to /opt/dominor6/lotus/notes/latest/sunspa/server 2022

###################################

thread 1/80 :: server, pid=2022, lwp=1, tid=1

###################################

[1] ff2990cc poll (0, 0, 1388)

[2] ff0bb278 poll (0, 25d78, 975e0, 98968, 0, 0) + 34

[3] 0002a3a0 ServerPoller (0, ca000, 0, ca000, c8c00, ca000) + 34

[4] 00029ad0 FirstProcessMain (1, 111e04, dd, ffbdf5d3, 0, 8d) + 2c

[5] 00029a6c ServerMain (1, 0, 0, ffbefad4, ffff, 0) + 338

[6] 000293e4 main (1, ffbefa7c, 1, c7c00, 0, 0) + 98

[7] 00029334 _start (0, 0, 0, 0, 0, 0) + 108

###################################

thread 2/80 :: server, pid=2022, lwp=2, tid=2

###################################

[1] ff29a890 signotifywait ()

[2] ff0ae920 _dynamiclwps (ff0ce000, 5c, 0, 4, 0, 0) + 1c

[3] ff0b1c78 thr_yield (0, 0, 0, 0, 0, 0) + 8c

###################################

thread 3/80 :: server, pid=2022, lwp=3, tid=3

###################################

[1] ff29aec0 lwp_sema_wait (fd30de30)

[2] ff0a97e4 _park (fd30de30, ff0ce000, 0, fd30dd78, 2502c, 0) + 114

[3] ff0a94c0 _swtch (fd30dd78, fd30dd78, ff0ce000, 5, ff0ce000, 1) + 400

[4] ff0ad8d8 _reap_wait (ff0d2a30, ff0ce000, 0, a, 0, a) + 38

[5] ff0ad630 _reaper (ff0cee58, ff0d4798, ff0d2a30, ff0cee30, 1, fe400000) + 38

[6] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 4/80 :: server, pid=2022, lwp=4, tid=N/A

###################################

[1] ff298454 door (0, 0, 0, 0, fd455d18, 4)

[2] ff0aa440 _lwp_start (fd455d78, 0, 6000, fd695b7c, 0, 0) + 18

[3] ff0b1c78 thr_yield (0, 0, 0, 0, 0, 0) + 8c

###################################

thread 5/80 :: server, pid=2022, lwp=5, tid=4

###################################

[1] ff29ae74 lwp_cond_wait (fc46a5d8, fc46a5c0, fcba1a40)

[2] ff292b30 _lwp_cond_timedwait (37724660, 408ce560, fcba1ab0, fc46a5d8, fc46a5c0, 1) + 98

[3] ff0a7ac8 cond_timedwait (fcba1d78, 408ce560, 6909f, ff0ce000, 0, 37724660) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc46a5d8, fc46a5c0, fcba1b88, 0, 0, 3e80) + c

[5] fd9eccf8 WaitForThreadSem (f424, 1f4, 1b805c4, e31bc, 1c63780, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (1f4, 0, 1f4, 0, 2f000, ff089e50) + fc

[7] fd9e9304 OSWaitEvent (fd53b190, 1f4, fc349770, 0, 0, 0) + 18

[8] fda14184 AutoFDGCleanupThreadProc (0, 1, 1, 28850, 0, 13df488) + 50

[9] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[10] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 6/80 :: server, pid=2022, lwp=6, tid=5

###################################

[1] ff29ae74 lwp_cond_wait (fc46a358, fc46a340, fca71a20)

[2] ff292b30 _lwp_cond_timedwait (2350b9c8, 40ce6e65, fca71a90, fc46a358, fc46a340, 1) + 98

[3] ff0a7ac8 cond_timedwait (fca71d78, 408ce52e, e9167, ff0ce000, 0, 2350b9c8) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc46a358, fc46a340, fca71b68, 0, 0, 0) + c

[5] fd9eccf8 WaitForThreadSem (408ce52e, ffffffff, 408ce52e, e9165, 48a15, a0750) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, 0, 0, 0, 2f000, ff089e50) + fc

[7] fd9e935c OSWaitEvent (fa5bda5c, 0, 1, 1, 0, 5) + 70

[8] fe1f72a8 NAMELookupThread (fa5bda2c, 5, fa5bda5c, feebc8c0, ff05eaf0, feebc9f8) + 138

[9] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[10] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 7/80 :: server, pid=2022, lwp=7, tid=6

###################################

[1] ff29ae74 lwp_cond_wait (fc46ab58, fc46ab40, fca219f0)

[2] ff292b30 _lwp_cond_timedwait (2bd340e8, 40ce453c, fca21a60, fc46ab58, fc46ab40, 1) + 98

[3] ff0a7ac8 cond_timedwait (fca21d78, 408ce52e, e9245, ff0ce000, 0, 2bd340e8) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc46ab58, fc46ab40, fca21b38, 52520, 4fbf7, 13efdc) + c

[5] fd9eccf8 WaitForThreadSem (408cbc05, ffffffff, 408ce52e, e9243, 6b7c9, 7da7a) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, 0, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9304 OSWaitEvent (fc3181d8, 0, 4, fc317dd4, 80, 8) + 18

[8] 00029dbc Scheduler (0, fc317dd4, c8c00, 17894c, 10000, c7fe8) + b8

[9] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[10] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 8/80 :: server, pid=2022, lwp=8, tid=7

###################################

[1] ff2990cc poll (0, 0, 12c)

[2] ff0bb278 poll (0, 2454, 9150, 927c, 0, 0) + 34

[3] fda1ba00 OSConsoleRead (fc9d1233, fc9d1236, 1, 0, 1214, 0) + 220

[4] 0003e170 ReadNext (fa0630cc, fc9d17c3, fc9d17c6, 0, 0, 0) + 1c

[5] 0003e264 ReadNextChar (fa0630cc, 2, c8400, c9a54, 4, 99da) + 3c

[6] 0003e0e4 ConsoleRead (fc9d18ac, fc9d18a8, fa0630cc, 1, 0, 32) + f8

[7] 0003df34 ConsoleProgram (fc9d1c70, fc9d1c6c, 0, 80, 80, 80) + 3b8

[8] 0002a0e0 Scheduler (0, fa05e0a8, c8c00, 17894c, 99da, c7fe8) + 3dc

[9] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[10] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 9/80 :: server, pid=2022, lwp=9, tid=8

###################################

[1] ff29ae74 lwp_cond_wait (fc46ab58, fc46ab40, fc9819f0)

[2] ff292b30 _lwp_cond_timedwait (2c6da070, 40ce453c, fc981a60, fc46ab58, fc46ab40, 1) + 98

[3] ff0a7ac8 cond_timedwait (fc981d78, 408ce52e, e8c12, ff0ce000, 0, 2c6da070) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc46ab58, fc46ab40, fc981b38, 52520, 4fbf7, 13efdc) + c

[5] fd9eccf8 WaitForThreadSem (408cbc05, ffffffff, 408ce52e, e8c0d, 6df4e, 7acbf) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, 0, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9304 OSWaitEvent (fc3181d8, 0, 8, fc317dd4, 80, 8) + 18

[8] 00029dbc Scheduler (0, fc317dd4, c8c00, 17894c, 10000, c7fe8) + b8

[9] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[10] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 10/80 :: server, pid=2022, lwp=10, tid=9

###################################

[1] ff29ae74 lwp_cond_wait (fc46ab58, fc46ab40, fc9319f0)

[2] ff292b30 _lwp_cond_timedwait (32ec50e0, 40ce453b, fc931a60, fc46ab58, fc46ab40, 1) + 98

[3] ff0a7ac8 cond_timedwait (fc931d78, 408ce52e, e940d, ff0ce000, 0, 32ec50e0) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc46ab58, fc46ab40, fc931b38, 52540, 4fc16, 13f058) + c

[5] fd9eccf8 WaitForThreadSem (408cbc04, ffffffff, 408ce52e, e940a, 888f4, 60b16) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, 0, 0, 0, 2f000, ff089e50) + fc

[7] fd9e935c OSWaitEvent (fc3181d8, 0, c, fc317dd4, 80, 8) + 70

[8] 00029dbc Scheduler (0, fc317dd4, c8c00, 17894c, 10000, c7fe8) + b8

[9] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[10] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 11/80 :: server, pid=2022, lwp=12, tid=11

###################################

[1] ff29ae74 lwp_cond_wait (fc46b018, fc46b000, fc891a20)

[2] ff292b30 _lwp_cond_timedwait (2ca8ec20, 40ce6047, fc891a90, fc46b018, fc46b000, 1) + 98

[3] ff0a7ac8 cond_timedwait (fc891d78, 408ce52e, e9371, ff0ce000, 0, 2ca8ec20) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc46b018, fc46b000, fc891b68, 1c3c0, 1b5a2, 6d688) + c

[5] fd9eccf8 WaitForThreadSem (408cd710, ffffffff, 408ce52e, e936f, 6ee7c, 7a4f3) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, 0, 0, 0, 2f000, ff089e50) + fc

[7] fd9e935c OSWaitEvent (fa5d765c, 0, 1, 1, 0, 1) + 70

[8] fe1f72a8 NAMELookupThread (fa5d762c, 0, fa5d765c, feebc8c0, ff05eaf0, feebc9f8) + 138

[9] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[10] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 12/80 :: server, pid=2022, lwp=13, tid=12

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, fbaf1ba4, fbaf1ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (fbaf1c74, 1, fbaf1cb4, 98968, 0, fa05e1f8) + 40

[7] 0002a074 Scheduler (0, fa05e1f8, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 13/80 :: server, pid=2022, lwp=14, tid=13

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, fbaa1ba4, fbaa1ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (fbaa1c74, 1, fbaa1cb4, 98968, 1194, fa05e328) + 40

[7] 0002a074 Scheduler (0, fa05e328, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 14/80 :: server, pid=2022, lwp=15, tid=14

###################################

[1] ff29ae74 lwp_cond_wait (fc46bd18, fc46bd00, fba51840)

[2] ff292b30 _lwp_cond_timedwait (18d2dfa8, 40ce4517, fba518b0, fc46bd18, fc46bd00, 1) + 98

[3] ff0a7ac8 cond_timedwait (fba51d78, 408ce52e, e9329, ff0ce000, 0, 18d2dfa8) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc46bd18, fc46bd00, fba51988, 529c0, 50072, 1401c8) + c

[5] fd9eccf8 WaitForThreadSem (408cbbe0, ffffffff, 408ce52e, e9327, 1da81, cb8a6) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, 0, 0, 0, 2f000, ff089e50) + fc

[7] fd9e935c OSWaitEvent (fc318994, 0, 0, 3fff, 2, 1) + 70

[8] 000379e4 ClAdminWorkThread (10, 0, 1, fc317dd4, 10, 0) + 1b0

[9] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[10] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 15/80 :: server, pid=2022, lwp=16, tid=15

###################################

[1] ff2990cc poll (0, 0, 1388)

[2] ff0bb278 poll (0, 25d78, 975e0, 98968, 28400, 285bc) + 34

[3] fd9f0280 OSStaticMem (75e, 8132, 1d78, 1000, fede3f24, fee56d94) + 270

[4] fd9ef064 OSStaticMemBeginInit (0, 8132, 690, fba01a5c, 13f4f48, fede3f24) + 8c

[5] fd9d71fc NotesSDKData (17c9e4, a8000, 2a, 270c748b, 400, 518) + 14

[6] fd9d7884 AddInShouldTerminate (2710, fba01b90, e, fa05e888, 0, 1) + 4

[7] 0006523c OrgNameCacheRefreshTask (0, ca000, 0, 0, ca000, 0) + 1bc

[8] 0002a0e0 Scheduler (0, fa05e7d4, c8c00, 17894c, 99da, c7fe8) + 3dc

[9] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[10] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 16/80 :: server, pid=2022, lwp=17, tid=16

###################################

[1] ff2990cc poll (0, 0, 1388)

[2] ff0bb278 poll (0, 25d78, 975e0, 98968, 28400, 285bc) + 34

[3] fd9f03f4 OSStaticMem (170, fc34feb8, 1000, 5c0, fc36ceb0, 10) + 3e4

[4] fd9ffa34 OSCheckYieldTimer (0, fb9b1870, 70646c00, 25400, c000, 2547c) + c

[5] fe4d0fe0 cmd_poll (3e8, fb9b192c, fb9b1928, fb9b1870, 0, 0) + 5ec

[6] fe41c030 nti_poll (1194, fb9b192c, fb9b1928, fb9b1870, 1, fe4d09f4) + 248

[7] fe433be0 PortDriver_AcceptWait (fb9b19c0, 1194, fede3f24, 2e, c000, 0) + d4

[8] fe43258c PortDriver_WaitHelper (fb9b1a6c, fb9b1a68, 0, fbc58540, 49c4, 0) + 3d4

[9] fe431f6c PortDriver_WaitExtended (fb9b1b00, fb9b1afc, 1, 49c4, 1, 1a4) + 14c

[10] fe431e10 PortDriver_Wait (fb9b1b80, fb9b1b7c, 1, 1000, fede3f24, fee56d94) + 38

[11] fe3a1898 NetWaitGuts (fb9b1c00, a4272c, 49c4, 1, fede3f24, 49c4) + b0

[12] fe3a12cc NetWait (fb9b1c68, fee1d184, 5000, 5040, fc3486f4, 0) + 2c

[13] 0002a194 Scheduler (0, fa05e458, c8c00, 17894c, 99da, c7fe8) + 490

[14] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[15] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 17/80 :: server, pid=2022, lwp=18, tid=17

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, fb961ba4, fb961ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (fb961c74, 1, fb961cb4, 98968, 9, 99da) + 40

[7] 0002a074 Scheduler (0, fa05e6ac, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 18/80 :: server, pid=2022, lwp=19, tid=18

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, fb3f1ba4, fb3f1ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (fb3f1c74, 1, fb3f1cb4, 98968, fc00, ffff) + 40

[7] 0002a074 Scheduler (0, fa05e584, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 19/80 :: server, pid=2022, lwp=20, tid=19

###################################

[1] ff29a350 read (d, fb1f1ca0, 10)

[2] ff0baa44 read (1, f4ee0bcc, fbc64d8c, fede3f24, 0, feec4a88) + 34

[3] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[4] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 20/80 :: server, pid=2022, lwp=21, tid=20

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, ffffffff, 28400) + 34

[3] fe42bfc0 iocp_ShutdownThread (10dac, 1, 1, 28850, 0, 13df488) + 5c

[4] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[5] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 21/80 :: server, pid=2022, lwp=22, tid=21

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, fb151828)

[2] ff292b30 _lwp_cond_timedwait (172260e8, 40ce6e66, fb151898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (fb151d78, 408ce52f, 16bcb, ff0ce000, 0, 172260e8) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, fb151970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b797ff, 5ec21, bd8420, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 1e, 1e, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, fb151b9c, 0, 0, fb151ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 15, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa060824, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 22/80 :: server, pid=2022, lwp=23, tid=22

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f87f1828)

[2] ff292b30 _lwp_cond_timedwait (17094b08, 40ce6e66, f87f1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f87f1d78, 408ce52f, 1655f, ff0ce000, 0, 17094b08) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f87f1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6d0eb, 5e5b5, bcb6a0, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, a, a, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f87f1b9c, 0, 0, f87f1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 16, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa060954, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 23/80 :: server, pid=2022, lwp=24, tid=23

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f87a1828)

[2] ff292b30 _lwp_cond_timedwait (171b96a0, 40ce6e66, f87a1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f87a1d78, 408ce52f, 16a0e, ff0ce000, 0, 171b96a0) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f87a1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b7621c, 5ea64, bd4c80, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 15, 15, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f87a1b9c, 0, 0, f87a1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 17, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05f8b4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 24/80 :: server, pid=2022, lwp=25, tid=24

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8751828)

[2] ff292b30 _lwp_cond_timedwait (172118c8, 40ce6e66, f8751898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8751d78, 408ce52f, 16b77, ff0ce000, 0, 172118c8) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8751970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b78dd3, 5ebcd, bd79a0, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 1c, 1c, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8751b9c, 0, 0, f8751ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 18, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05e944, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 25/80 :: server, pid=2022, lwp=26, tid=25

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8701828)

[2] ff292b30 _lwp_cond_timedwait (171c90a0, 40ce6e66, f8701898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8701d78, 408ce52f, 16a4d, ff0ce000, 0, 171c90a0) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8701970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b769dc, 5eaa4, bd5480, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 16, 16, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8701b9c, 0, 0, f8701ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 19, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa060364, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 26/80 :: server, pid=2022, lwp=27, tid=26

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f86b1828)

[2] ff292b30 _lwp_cond_timedwait (178a65d0, 40ce6e66, f86b1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f86b1d78, 408ce52f, 1866d, ff0ce000, 0, 178a65d0) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f86b1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, bad17e, 606c2, c0d840, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 26, 26, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f86b1b9c, 0, 0, f86b1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 1a, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05f194, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 27/80 :: server, pid=2022, lwp=28, tid=27

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8661828)

[2] ff292b30 _lwp_cond_timedwait (171f3c38, 40ce6e66, f8661898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8661d78, 408ce52f, 16afd, ff0ce000, 0, 171f3c38) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8661970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b77f0d, 5eb53, bd6a60, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 1a, 1a, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8661b9c, 0, 0, f8661ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 1b, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa060f44, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 28/80 :: server, pid=2022, lwp=29, tid=28

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8611828)

[2] ff292b30 _lwp_cond_timedwait (17253b60, 40ce6e66, f8611898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8611d78, 408ce52f, 16c86, ff0ce000, 0, 17253b60) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8611970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b7aea4, 5ecdc, bd9b80, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 22, 22, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8611b9c, 0, 0, f8611ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 1c, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05ffd4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 29/80 :: server, pid=2022, lwp=30, tid=29

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f85c1828)

[2] ff292b30 _lwp_cond_timedwait (172070d0, 40ce6e66, f85c1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f85c1d78, 408ce52f, 16b4d, ff0ce000, 0, 172070d0) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f85c1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b7889e, 5eba2, bd7440, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 1b, 1b, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f85c1b9c, 0, 0, f85c1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 1d, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa060a84, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 30/80 :: server, pid=2022, lwp=31, tid=30

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (1f, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f8571a3c, f8571a38, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (12, 1, 4, ff05f1cc, 13ee100, 28400) + 20

[6] 0002c94c ExecGetPoolTaskShutdownPending (12, f8571b9c, 0, 0, f8571ba0, 0) + 24

[7] 00063430 WorkThreadTask (f1d64, 1e, a07, c8c00, 51ce, 12) + 2bc

[8] 0002a0e0 Scheduler (0, fa05f784, c8c00, 17894c, 99da, c7fe8) + 3dc

[9] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[10] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 31/80 :: server, pid=2022, lwp=32, tid=31

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8521828)

[2] ff292b30 _lwp_cond_timedwait (17239d50, 40ce6e66, f8521898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8521d78, 408ce52f, 16c1c, ff0ce000, 0, 17239d50) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8521970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b7a1ce, 5ec72, bd8e40, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 20, 20, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8521b9c, 0, 0, f8521ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 1f, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa061534, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 32/80 :: server, pid=2022, lwp=33, tid=32

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f84d1828)

[2] ff292b30 _lwp_cond_timedwait (170b5a60, 40ce6e66, f84d1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f84d1d78, 408ce52f, 165e6, ff0ce000, 0, 170b5a60) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f84d1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6e144, 5e63c, bcc780, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, e, e, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f84d1b9c, 0, 0, f84d1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 20, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05eba4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 33/80 :: server, pid=2022, lwp=34, tid=33

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8481828)

[2] ff292b30 _lwp_cond_timedwait (170566f0, 40ce6e66, f8481898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8481d78, 408ce52f, 16470, ff0ce000, 0, 170566f0) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8481970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6b20a, 5e4b6, bc96c0, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 1, 1, 1, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8481b9c, 0, 0, f8481ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 21, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05fb14, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 34/80 :: server, pid=2022, lwp=35, tid=34

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8431828)

[2] ff292b30 _lwp_cond_timedwait (1721c4a8, 40ce6e66, f8431898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8431d78, 408ce52f, 16ba3, ff0ce000, 0, 1721c4a8) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8431970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b79327, 5ebf9, bd7f20, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 1d, 1d, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8431b9c, 0, 0, f8431ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 22, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05ee04, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 35/80 :: server, pid=2022, lwp=36, tid=35

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f83e1828)

[2] ff292b30 _lwp_cond_timedwait (171d4450, 40ce6e66, f83e1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f83e1d78, 408ce52f, 16a7c, ff0ce000, 0, 171d4450) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f83e1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b76f6e, 5ead2, bd5a40, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 17, 17, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f83e1b9c, 0, 0, f83e1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 23, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05fc44, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 36/80 :: server, pid=2022, lwp=37, tid=36

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8391868)

[2] ff292b30 _lwp_cond_timedwait (1eb538a8, 40ce6e66, f83918d8, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8391d78, 408ce52f, 35c24, ff0ce000, 0, 1eb538a8) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f83919b0, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, f3b2a7, 7dc79, fb8f20, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 13de7ac, 28, 28, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, fc3486f4, fc349770, 0, 0, 0) + 1c

[9] 0006428c WorkThreadUtilityTask (fc317dd4, d4438980, 0, 3a98, f99b7604, ffff) + 204

[10] 0002a0e0 Scheduler (0, fa0618c4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 37/80 :: server, pid=2022, lwp=38, tid=37

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8341828)

[2] ff292b30 _lwp_cond_timedwait (1708a310, 40ce6e66, f8341898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8341d78, 408ce52f, 16535, ff0ce000, 0, 1708a310) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8341970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6cbb6, 5e58a, bcb140, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 8, 8, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8341b9c, 0, 0, f8341ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 25, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05ea74, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 38/80 :: server, pid=2022, lwp=39, tid=38

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f82f1828)

[2] ff292b30 _lwp_cond_timedwait (1722fd28, 40ce6e66, f82f1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f82f1d78, 408ce52f, 16bf3, ff0ce000, 0, 1722fd28) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f82f1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b79cd7, 5ec49, bd8920, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 1f, 1f, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f82f1b9c, 0, 0, f82f1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 26, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa060bb4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 39/80 :: server, pid=2022, lwp=40, tid=39

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f82a1828)

[2] ff292b30 _lwp_cond_timedwait (17064d68, 40ce6e66, f82a1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f82a1d78, 408ce52f, 1649b, ff0ce000, 0, 17064d68) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f82a1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6b92f, 5e4f1, bc9e20, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 2, 2, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f82a1b9c, 0, 0, f82a1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 27, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa060234, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 40/80 :: server, pid=2022, lwp=41, tid=40

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8251828)

[2] ff292b30 _lwp_cond_timedwait (171de860, 40ce6e66, f8251898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8251d78, 408ce52f, 16aa6, ff0ce000, 0, 171de860) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8251970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b77484, 5eafc, bd5f80, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 18, 18, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8251b9c, 0, 0, f8251ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 28, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05f2c4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 41/80 :: server, pid=2022, lwp=42, tid=41

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8201828)

[2] ff292b30 _lwp_cond_timedwait (1725e358, 40ce6e66, f8201898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8201d78, 408ce52f, 16cb1, ff0ce000, 0, 1725e358) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8201970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b7b3d9, 5ed07, bda0e0, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 23, 23, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8201b9c, 0, 0, f8201ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 29, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05f064, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 42/80 :: server, pid=2022, lwp=43, tid=42

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f81b1828)

[2] ff292b30 _lwp_cond_timedwait (17325ac0, 40ce6e66, f81b1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f81b1d78, 408ce52f, 16fe3, ff0ce000, 0, 17325ac0) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f81b1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b816c8, 5f038, be0700, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 25, 25, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f81b1b9c, 0, 0, f81b1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 2a, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa0612d4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 43/80 :: server, pid=2022, lwp=44, tid=43

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8161828)

[2] ff292b30 _lwp_cond_timedwait (170cf0a0, 40ce6e66, f8161898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8161d78, 408ce52f, 1664f, ff0ce000, 0, 170cf0a0) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8161970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6eddc, 5e6a4, bcd480, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 11, 11, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8161b9c, 0, 0, f8161ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 2b, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05fea4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 44/80 :: server, pid=2022, lwp=45, tid=44

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8111828)

[2] ff292b30 _lwp_cond_timedwait (170a8b58, 40ce6e66, f8111898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8111d78, 408ce52f, 165b1, ff0ce000, 0, 170a8b58) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8111970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6dad9, 5e607, bcc0e0, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, d, d, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8111b9c, 0, 0, f8111ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 2c, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa061664, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 45/80 :: server, pid=2022, lwp=46, tid=45

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f80c1828)

[2] ff292b30 _lwp_cond_timedwait (171b0230, 40ce6e66, f80c1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f80c1d78, 408ce52f, 169e8, ff0ce000, 0, 171b0230) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f80c1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b75d82, 5ea3e, bd47c0, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 14, 14, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f80c1b9c, 0, 0, f80c1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 2d, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05f654, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 46/80 :: server, pid=2022, lwp=47, tid=46

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8071828)

[2] ff292b30 _lwp_cond_timedwait (1709e748, 40ce6e66, f8071898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8071d78, 408ce52f, 16588, ff0ce000, 0, 1709e748) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8071970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6d5c3, 5e5dd, bcbba0, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, b, b, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8071b9c, 0, 0, f8071ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 2e, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05f524, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 47/80 :: server, pid=2022, lwp=48, tid=47

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f8021828)

[2] ff292b30 _lwp_cond_timedwait (170b8170, 40ce6e66, f8021898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f8021d78, 408ce52f, 165f0, ff0ce000, 0, 170b8170) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f8021970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6e27a, 5e646, bcc8c0, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, f, f, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f8021b9c, 0, 0, f8021ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 2f, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05f9e4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 48/80 :: server, pid=2022, lwp=49, tid=48

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7fd1828)

[2] ff292b30 _lwp_cond_timedwait (170c8728, 40ce6e66, f7fd1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7fd1d78, 408ce52f, 16633, ff0ce000, 0, 170c8728) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7fd1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6ea97, 5e689, bcd120, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 10, 10, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7fd1b9c, 0, 0, f7fd1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 30, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05fd74, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 49/80 :: server, pid=2022, lwp=50, tid=49

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7f81828)

[2] ff292b30 _lwp_cond_timedwait (17268f38, 40ce6e66, f7f81898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7f81d78, 408ce52f, 16d0f, ff0ce000, 0, 17268f38) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7f81970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b7b92d, 5ed33, bda660, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 24, 24, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7f81b9c, 0, 0, f7f81ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 31, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa060104, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 50/80 :: server, pid=2022, lwp=51, tid=50

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7f31828)

[2] ff292b30 _lwp_cond_timedwait (17055f20, 40ce6e66, f7f31898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7f31d78, 408ce52f, 1645f, ff0ce000, 0, 17055f20) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7f31970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6b1cc, 5e4b4, bc9680, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 0, 0, 0, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7f31b9c, 0, 0, f7f31ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 32, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa0606f4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 51/80 :: server, pid=2022, lwp=52, tid=51

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7ee1828)

[2] ff292b30 _lwp_cond_timedwait (1708e578, 40ce6e66, f7ee1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7ee1d78, 408ce52f, 16548, ff0ce000, 0, 1708e578) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7ee1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6cdc5, 5e59b, bcb360, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 9, 9, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7ee1b9c, 0, 0, f7ee1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 33, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05ef34, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 52/80 :: server, pid=2022, lwp=53, tid=52

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7e91828)

[2] ff292b30 _lwp_cond_timedwait (171a5e20, 40ce6e66, f7e91898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7e91d78, 408ce52f, 169bf, ff0ce000, 0, 171a5e20) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7e91970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b7586c, 5ea14, bd4280, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 13, 13, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7e91b9c, 0, 0, f7e91ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 34, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa060494, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 53/80 :: server, pid=2022, lwp=54, tid=53

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7e41828)

[2] ff292b30 _lwp_cond_timedwait (1707f730, 40ce6e66, f7e41898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7e41d78, 408ce52f, 16508, ff0ce000, 0, 1707f730) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7e41970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6c662, 5e55e, bcabc0, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 6, 6, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7e41b9c, 0, 0, f7e41ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 35, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05ecd4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 54/80 :: server, pid=2022, lwp=55, tid=54

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7df1828)

[2] ff292b30 _lwp_cond_timedwait (1718ac88, 40ce6e66, f7df1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7df1d78, 408ce52f, 16972, ff0ce000, 0, 1718ac88) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7df1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b74afb, 5e9a5, bd34a0, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 12, 12, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7df1b9c, 0, 0, f7df1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 36, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa0605c4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 55/80 :: server, pid=2022, lwp=56, tid=55

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7da1828)

[2] ff292b30 _lwp_cond_timedwait (170860a8, 40ce6e66, f7da1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7da1d78, 408ce52f, 16524, ff0ce000, 0, 170860a8) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7da1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6c9a7, 5e579, bcaf20, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 7, 7, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7da1b9c, 0, 0, f7da1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 37, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa060e14, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 56/80 :: server, pid=2022, lwp=57, tid=56

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7d51828)

[2] ff292b30 _lwp_cond_timedwait (170a4120, 40ce6e66, f7d51898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7d51d78, 408ce52f, 1659f, ff0ce000, 0, 170a4120) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7d51970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6d88c, 5e5f4, bcbe80, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, c, c, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7d51b9c, 0, 0, f7d51ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 38, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa060ce4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 57/80 :: server, pid=2022, lwp=58, tid=57

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7d01828)

[2] ff292b30 _lwp_cond_timedwait (1706b6e0, 40ce6e66, f7d01898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7d01d78, 408ce52f, 164b7, ff0ce000, 0, 1706b6e0) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7d01970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6bc74, 5e50c, bca180, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 3, 3, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7d01b9c, 0, 0, f7d01ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 39, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa061074, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 58/80 :: server, pid=2022, lwp=59, tid=58

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7cb1828)

[2] ff292b30 _lwp_cond_timedwait (1707a140, 40ce6e66, f7cb1898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7cb1d78, 408ce52f, 164f2, ff0ce000, 0, 1707a140) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7cb1970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6c3b8, 5e548, bca900, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 5, 5, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7cb1b9c, 0, 0, f7cb1ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 3a, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa0611a4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 59/80 :: server, pid=2022, lwp=60, tid=59

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7c61828)

[2] ff292b30 _lwp_cond_timedwait (171e8c70, 40ce6e66, f7c61898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7c61d78, 408ce52f, 16ad0, ff0ce000, 0, 171e8c70) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7c61970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b7799a, 5eb26, bd64c0, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 19, 19, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7c61b9c, 0, 0, f7c61ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 3b, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa061404, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 60/80 :: server, pid=2022, lwp=61, tid=60

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7c11828)

[2] ff292b30 _lwp_cond_timedwait (17072828, 40ce6e66, f7c11898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7c11d78, 408ce52f, 164d4, ff0ce000, 0, 17072828) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7c11970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b6bff7, 5e529, bca520, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 4, 4, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7c11b9c, 0, 0, f7c11ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 3c, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa061794, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 61/80 :: server, pid=2022, lwp=62, tid=61

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7bc1858)

[2] ff292b30 _lwp_cond_timedwait (1b22aba8, 40ce6e66, f7bc18c8, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7bc1d78, 408ce52f, 27205, ff0ce000, 0, 1b22aba8) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7bc19a0, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, d758c7, 6f259, de4b20, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 13de7ac, 27, 27, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, fc3486f4, fc349770, 0, 0, 0) + 1c

[9] 000649c4 RemoteLiveConsoleTask (f7bc1c70, fc317dd4, 1400, 3d, ffff, 10d8) + 17c

[10] 0002a0e0 Scheduler (0, fa0619f4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 62/80 :: server, pid=2022, lwp=63, tid=62

###################################

[1] ff29ae74 lwp_cond_wait (fc471818, fc471800, f7b71828)

[2] ff292b30 _lwp_cond_timedwait (17248f80, 40ce6e66, f7b71898, fc471818, fc471800, 1) + 98

[3] ff0a7ac8 cond_timedwait (f7b71d78, 408ce52f, 16c5a, ff0ce000, 0, 17248f80) + c8

[4] ff0a79d0 pthread_cond_timedwait (fc471818, fc471800, f7b71970, 1fffffdb, fffffed8, 24e0) + c

[5] fd9eccf8 WaitForThreadSem (900b, ffffffff, b7a950, 5ecb0, bd9600, 3b9aca00) + 10c

[6] fd9ea218 WaitOnNativeSemaphore (ffffffff, fc31cf6c, 0, 0, 2f000, ff089e50) + fc

[7] fd9e9be4 OSLockSemInt (fc31cf6c, 4000, 4, 21, 21, 28400) + 14c

[8] 0002c944 ExecGetPoolTaskShutdownPending (12, f7b71b9c, 0, 0, f7b71ba0, 0) + 1c

[9] 00063430 WorkThreadTask (f1d64, 3e, a07, c8c00, 51ce, 12) + 2bc

[10] 0002a0e0 Scheduler (0, fa05f3f4, c8c00, 17894c, 99da, c7fe8) + 3dc

[11] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[12] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 63/80 :: server, pid=2022, lwp=64, tid=63

###################################

[1] ff2978d0 nanosleep (f7b1fe18, f7b1fe20)

[2] ff0baedc _ti__nanosleep (1, 7fffffff, 40, f7b202b0, ff0d5980, ff0ce000) + 34

[3] fda098c0 OSRunExternalScript (f7b203dc, fed30eec, ff089e50, 0, 800, 125c00) + 1dc

[4] fda08080 OSFaultCleanup (0, 0, fc400000, fede3f24, fede3f24, 5) + 460

[5] fd9e48cc fatal_error (b, f7b20ed8, f7b20c20, ff0ce000, f7b21e10, f7b21e00) + 1a4

[6] ff0b90b0 __libthread_segvhdlr (b, f7b20ed8, f7b20c20, ff0ce000, b, 0) + e4

[7] ff0bbc20 __sighndlr (b, f7b20ed8, f7b20c20, ff0b8fcc, f7b21e10, f7b21e00) + c

[8] ff0b8424 sigacthandler (b, f7b21d78, f7b20c20, ff0ce000, f7b21d78, f7b20ed8) + 6e0

[9] — called from signal handler with signal 11 (SIGSEGV) —

[10] fda14ebc OSPathResolve (103, 0, 0, 0, 0, 0) + 174

###################################

thread 64/80 :: server, pid=2022, lwp=65, tid=64

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f7ad1ba4, f7ad1ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f7ad1c74, 1, f7ad1cb4, 98968, fa061fd0, fc00) + 40

[7] 0002a074 Scheduler (0, fa061c4c, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 65/80 :: server, pid=2022, lwp=66, tid=65

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f7a81ba4, f7a81ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f7a81c74, 1, f7a81cb4, 3d090, fa062d24, fc00) + 40

[7] 0002a074 Scheduler (0, fa062b90, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 66/80 :: server, pid=2022, lwp=67, tid=66

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f7a31ba4, f7a31ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f7a31c74, 1, f7a31cb4, f424, 57a0, fa061fdc) + 40

[7] 0002a074 Scheduler (0, fa061fdc, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 67/80 :: server, pid=2022, lwp=68, tid=67

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f79e1ba4, f79e1ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f79e1c74, 1, f79e1cb4, f424, 566c, fa061ea8) + 40

[7] 0002a074 Scheduler (0, fa061ea8, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 68/80 :: server, pid=2022, lwp=69, tid=68

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f7991ba4, f7991ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f7991c74, 1, f7991cb4, f424, 5a44, fa0627f4) + 40

[7] 0002a074 Scheduler (0, fa0627f4, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 69/80 :: server, pid=2022, lwp=70, tid=69

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (26800, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f7941ba4, f7941ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f7941c74, 1, f7941cb4, f424, 5a14, fa062354) + 40

[7] 0002a074 Scheduler (0, fa062354, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 70/80 :: server, pid=2022, lwp=71, tid=70

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f78f1ba4, f78f1ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f78f1c74, 1, f78f1cb4, f424, 5a2c, fa0625a4) + 40

[7] 0002a074 Scheduler (0, fa0625a4, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 71/80 :: server, pid=2022, lwp=72, tid=71

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f78a1ba4, f78a1ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f78a1c74, 1, f78a1cb4, f424, 5a20, fa06247c) + 40

[7] 0002a074 Scheduler (0, fa06247c, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 72/80 :: server, pid=2022, lwp=73, tid=72

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f7851ba4, f7851ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f7851c74, 1, f7851cb4, f424, 5a50, fa062934) + 40

[7] 0002a074 Scheduler (0, fa062934, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 73/80 :: server, pid=2022, lwp=74, tid=73

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f7801ba4, f7801ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f7801c74, 1, f7801cb4, f424, 5a08, fa06222c) + 40

[7] 0002a074 Scheduler (0, fa06222c, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 74/80 :: server, pid=2022, lwp=75, tid=74

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f77b1a74, f77b1a70, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1aad, 1, 0, 8c1, 13ee100, f800) + 20

[6] fdf2b384 DbCacheOpenFiles (28400, 0, fc314f8c, 2858c, fc314f8c, 1800) + 94

[7] fdf2a468 NSFProcessDbCache (0, fc314f8c, 1a74, 14f8, 1, fc314f8c) + dc

[8] 000357d8 DbCacheTask2 (fa06291c, f77b1c6c, fc00, ffff, 1, 37) + 180

[9] 0002a0e0 Scheduler (0, fa061d74, c8c00, 17894c, 99da, c7fe8) + 3dc

[10] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[11] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 75/80 :: server, pid=2022, lwp=76, tid=75

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f7761ba4, f7761ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f7761c74, 1, f7761cb4, f424, 5a38, fa0626cc) + 40

[7] 0002a074 Scheduler (0, fa0626cc, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 76/80 :: server, pid=2022, lwp=77, tid=76

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f7711ba4, f7711ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f7711c74, 1, f7711cb4, f424, 5a5c, fa062a68) + 40

[7] 0002a074 Scheduler (0, fa062a68, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 77/80 :: server, pid=2022, lwp=78, tid=77

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f76c1ba4, f76c1ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f76c1c74, 1, f76c1cb4, f424, 59fc, fa062104) + 40

[7] 0002a074 Scheduler (0, fa062104, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 78/80 :: server, pid=2022, lwp=79, tid=78

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f7531ba4, f7531ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 1, 13fa56c, 0, 13ee100, 28c44) + 20

[6] 0002dce4 UnlockTaskSem (fa062e58, f7531c6c, 1, 19, 5df8, fa062e58) + 20

[7] 0002a11c Scheduler (0, fa062e58, c8c00, 17894c, 99da, c7fe8) + 418

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 79/80 :: server, pid=2022, lwp=80, tid=79

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f74e1ba4, f74e1ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 0, fc317dd4, fc317dd4, 13ee100, 2710) + 20

[6] 0002db10 GetTaskPtr (f74e1c74, 1, f74e1cb4, 1c9c38, 5e04, fa062f80) + 40

[7] 0002a074 Scheduler (0, fa062f80, c8c00, 17894c, 99da, c7fe8) + 370

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

###################################

thread 80/80 :: server, pid=2022, lwp=81, tid=80

###################################

[1] ff2990cc poll (0, 0, 3e8)

[2] ff0bb278 poll (0, 7918, 1e460, 1e848, c00, 285bc) + 34

[3] fd9ecfec StaticHangEnable (0, 0, 400, 5e0, fee56d84, 0) + a0

[4] fd9f6424 LockHandle (126200, f7491ba4, f7491ba0, ff089e50, fc400000, 285bc) + 44

[5] fd9f5e40 OSLockObject (1a49, 1, 13fa56c, 0, 13ee100, 28c44) + 20

[6] 0002dce4 UnlockTaskSem (fa062d30, f7491c6c, 1, 18, 5dec, fa062d30) + 20

[7] 0002a11c Scheduler (0, fa062d30, c8c00, 17894c, 99da, c7fe8) + 418

[8] fda048f4 ThreadWrapper (5a10, fc3486f4, 5800, 5a14, fc3486f4, 5a18) + 124

[9] ff0bbb24 _thread_start (0, 0, 0, 0, 0, 0) + 40

hcl-bot · April 26, 2004, 6:28am

Subject: 6.5.1 server crashes every 3-4 hours

Could this be OS patch related? We havent updated the patches on this box because be / is nearly full and the root drives need to be upgraded…

the 6.5.1 install mentioned some patches were missing. the list was mostly obsolete patches, and when mapped to new patches on sun.com looking like this:

109951-01

109657-09

108993-33

108869-23

108528-29

108725-16

108652-79

hcl-bot · April 26, 2004, 10:46am

Subject: RE: 6.5.1 server crashes every 3-4 hours

Just returning from vacation, otherwise I’d have responded earlier. I’m presuming that this post is the same problem as your earlier post (6.5 crashing / 6.0.3 stable).

There is only one thing which has the capability to delete NSD logs, that’s the File Retention feature of Automatic Diagnostic Collection (ADC). You can find the settings in the Configuration document for your server under the Diagnostics tab. The retntion is specified in days if automatic removal is enabled.

But it does sound like your patch level may be suspect. In addidtion to the odd NSD removal issue, the stack trace shows that OSPathResolve was passed a NULL pointer (curious) but even stranger is that this is at the top of the stack. It would be more useful to see who was calling this function.

The patch requirements aren’t given to us by vendors, they fix specific problems that are encountered during the development and testing phases. Some resolve issues that were found by other customers, but we have excellent relationships with the vendors and often get fixes from them that we uncover ourselves. It is very prudent to install the patches and in any case will be required if you open a support call.

hcl-bot · April 26, 2004, 1:49pm

Subject: RE: 6.5.1 server crashes every 3-4 hours

Jim,

Thanks for your response.

The NSD seems to be working now after the 6.5.1 reinstall. However, since the process causing the fault appears to be SERVER, its not providing much info. I read somewhere that setting nsd setuid root can give more info, but i have no idea if that’s true.

The NSD log that I sent you is exactly the same every time the box reboots… it obviously has a lot more info in it, but nothing that I can understand.

Is there anything else I can do or give you to help troubleshoot?? I can email you the whole file?

I will try to install the patches if you really think that might help… (although we are having a problem doing so due to limited disk space on / .)

Thanks!!

Damian

damian@svothi.com

hcl-bot · April 27, 2004, 9:59am

Subject: One More thing

Since you are using Fault Recovery on Solaris, be sure to add the line

msgsys:msginfo_msgtql=1024

to the /etc/system file and reboot. The number of available message queues on Solaris must be increased from the default (40) or Fault Recovery won’t work properly

hcl-bot · April 30, 2004, 12:28am

Subject: RE: One More thing

Thanks Jim. We’re actually using these (per latest Redbook):

Settings for Lotus Domino R6

set rlim_fd_max=65536

set msgsys:msginfo_msgtql=1024

set segmap_percent=40

set autoup=600

set tune_t_fsflushr=1

set maxpgio=16384

set sq_max_size=50

Machine is still crashing 4 times per day. It does restart in a just minutes tho… and we have it in a cluster. The other machine never crashes. Same OS. Same Domino.

In any event, we are moving 3.4TB of data off of this box as we speak so that we’ll be able to rebuild it once it can be taken out of production. I am going to do a fresh OS and fresh Domino install on it. If that doesn’t work, I will have very serious issues with Lotus.

We are also going to pull other services running off the box (realserver and NFS). PLUS, we are going to switch the Domino data directories to RAID 1+0 as opposed to a legacy RAID 5 install…

D

hcl-bot · May 5, 2004, 10:52pm

Subject: RE: One More thing

Running Recommended_8 patches tomorrow…

Will keep y’all posted if that stops the crashmonster…

We have been able to stay in business because the servers are clustered and failover works great with SSO enabled.

hcl-bot · May 11, 2004, 10:23am

Subject: RE: One More thing

We installed Recommended_8 and ran PatchPro to get all patches up to date. (This was more of a production that it should have been… Sun LDAP patch bug, required increasing OS drive size and setting up new RAIDs, etc.)

The “so far so good” news is that Domino has been stable for over 48 hours since applying these patches. (It was crashing every few hours).

In about 2 weeks we’ll know if it’s really stable.

Just for reference, the cluster sister to this box never crashes. It’s rock solid. So we hope this box will follow in those footsteps…

Will keep y’all posted.

D

hcl-bot · May 26, 2004, 10:33am

Subject: Domino is now rock stable after Patches

Ok, guys, I just wanted to share this good news.

After installing all the latest patches on our Solaris 8 box, R6 is now rock solid. Before the patches, it was crashing several times per day.

So the morale of this story is… if you run Solaris, install your patches!!!

hcl-bot · November 24, 2004, 9:18pm

Subject: RE: Domino is now rock stable after Patches

Hi Damian!

 I'm having the same problem with patches, but I already installed the latest patches and Domino is still telling me that some patches are missing, the thing is that these patches are obsolete and I do have the patches (newer) that replace these ones, so, what to do? Any ideas? Why Domino is asking for obsolete patches if I have the latest ones?

Thanks for your answer!

Rafael

hcl-bot · April 26, 2004, 2:39pm

Subject: RE: 6.5.1 server crashes every 3-4 hours

The fact that the process that’s crashing is SERVER should have no bearing at all on how much info is provided. NSD gathers information for all processes without prejudice. Setting setuid root privileges should have no ipact at all on NSD’s ability to function. That might be somehow related to R5 days when partitions were run with different Unix user id’s.

I strongly suggest that you install the patches. It will be difficult to get help until you do.

hcl-bot · April 26, 2004, 2:39pm

Subject: RE: 6.5.1 server crashes every 3-4 hours

Another thing I might mention is…

the server is running two other apps…

ipf (firewall)

realserver

no ports are shared in common, however, the IP address is.

however, R5 worked in this environment.

also important to note… when i FIRST installed 6.5 … it ran 4 days without problem, until it started crashing 6 times per day… now it won’t stop crashing… the only thing i changed was converting all the DBs to ODS 43, fixup, compact, etc.