HTTP threads waiting for semaphore

hcl-bot · August 11, 2008, 5:33pm

I have a Domino 8.0.1 server on 32-bit W2K3 SP2 that’s experiencing some kind of memory leak. This could be due to an add-in, such as the Symantec BackupExec 12 agent for Domino, or it could be due to scheduled agents that run in a large application on the server (although these ran fine on a previous Domino 7 on iSeries box), or it could be something else entirely.

So far the server hasn’t crashed, but it did become virtually unresponsive over the weekend, with users just receiving the message “Insufficient Memory” when attempting to open any database. I ran NSD against the server while it was still up, and then stopped the Domino service and rebooted the machine. It’s hard to get a clear understanding of the issue from the NSD log, as nothing actually ever crashed. But it does show 60 active threads for the NHTTP task. This surprised me, as web use on this server is very light. However, DDM Probes (including HTTP) are turned on. Looking more closely, I noticed that almost all threads contained the kernel32.WaitForSingleObject and nnotes.WaitOnNativeSemaphore functions.

############################################################

thread 56/60: [ nHTTP: 07c8: 0580]

FP=3345ff00, PC=7c8285ec, SP=3345fe90

stkbase=33460000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (948,3e8,0,3345ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (948,3e8,2a2a24f7,2a2a24db)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (229,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2a2a24f7,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2a2a24db,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2a2a24db,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (60105380,0,0,fffe0129)

In fact, there were also many router, agent manager and McAfee Groupshield threads with similar stack traces.

I have now run NSD against the server again, for comparison. The server is not having any “Insufficient Memory” errors, but the new NSD still shows many threads with semaphore waits. However, the SEMDEBUG.TXT file contains only one line, which seems to be a pointer to a non-existent log file:

SEMDEBUG_myserver_2008_08_11@08_24_45.TXT

Any thoughts as to what might be causing this would be very welcome.

Rupert Clayton

Chicago

hcl-bot · August 13, 2008, 10:17am

Subject: You may want to open a support ticket about this.

It’s possible that your add-in task is the culprit (does it support 8.0.1?)

But if it isn’t, you should consider opening a support ticket. I had this exact problem happening on servers running later versions of 6.5.x.

Our servers were only running HTTP, Amgr, and Update. No add-ins or anything else. CPU was idle, disk was idle, there was nothing running at all, yet there were HTTP and Amgr threads waiting on semaphores.

The problem didn’t seem to occur on 6.5.2, but after going to 6.5.4 it was very predictable behavior and would manifest every 48-72 hours. We had to start rebooting servers every 2 days in the middle of the night to avoid the problem. Upgrading to 6.5.5 and 6.5.6 didn’t help.

According to Lotus Support, though, ND7 uses an entirely different semaphore structure than 6.x. So we upgraded to 7.0.2 and the problem disappeared. AFAIK Lotus Support never did identify the exact problem. It’s possible that it has re-surfaced in ND8.

hcl-bot · August 19, 2008, 9:20am

Subject: Looks like I’ll need to keep an eye on the semaphores

Thanks Erik,

Following the resolution steps here fixed the “belnapi.exe has terminated abnormally” errors we were seeing. http://seer.entsupport.symantec.com/docs/301510.htm

We also have seen no recurrence of the insufficient memory errors we were seeing.

However, the same tasks are still showing large numbers of active threads in NSD.

I’ll keep an eye on this and log a support call if there seem to be any negative consequences from it.

Rupert Clayton

Chicago

hcl-bot · August 12, 2008, 3:16pm

Subject: re: HTTP threads waiting for semaphore

That callstack looks like an idle http thread simply waiting for work.

hcl-bot · August 12, 2008, 5:00pm

Subject: Does 59 idle HTTP threads sound normal?

Thanks for your thoughts on the stack trace snippet. I’m surprised however by the large number of apparently idle threads for NHTTP, NSERVER, NAMGR, NEVENT and NROUTER. The full stack for NHTTP threads is pasted below.

Maybe it’s just that I haven’t really noticed these when analysing NSD logs before. Those logs have generally been from crashed servers and clients, and so I have focused on the threads identified as FATAL.

Thanks,

Rupert

<@@ ------ Notes Process Info → Call Stack for Process :: [ nhttp: 0558] (Time 17:08:02) ------ @@>

############################################################

thread 1/59: [ nhttp: 0558: 0750]

FP=0006f998, PC=7c8285ec, SP=0006f930

stkbase=00070000, total stksize=45056, used stksize=1744

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (7d0,0,6f9b8,6009668a)

[ 2] 0x77e424ed kernel32.Sleep+15 (7d0,1f95fb4,6f9d0,601df890)

@[ 3] 0x6009668a nnotes.OSDelayThread@4+42 (7d0)

@[ 4] 0x601df890 nnotes.AddInIdleDelay@4+80 (7d0)

@[ 5] 0x10018baa nhttpstack.HTAddInMain@12+746 (400000,1,17f517c)

@[ 6] 0x0040103f nhttp.NotesMain@8+47 (1,400000)

@[ 7] 0x00401174 nhttp.notes_main+212 (0,0,0,1)

@[ 8] 0x00401066 nhttp.main+22 (1,1861640,3d2ab0,403000)

@[ 9] 0x0040134d nhttp.mainCRTStartup+323 (0,0,7ffd6000,0)

[10] 0x77e6f23b kernel32.ProcessIdToSessionId+521 (40120a,0,78746341,20)

############################################################

thread 2/59: [ nhttp: 0558: 1734]

FP=2d61fe88, PC=7c8285ec, SP=2d61fe18

stkbase=2d620000, total stksize=262144, used stksize=488

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (3f8,ffffffff,0,7c81a3ab)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (3f8,ffffffff,a39a8,2d901568)

[ 3] 0x7f012270 J9THR23.j9thread_monitor_try_enter_using_threadId+1280 (1,7f012c82,3da090,0)

[ 4] 0x7f01261f J9THR23.j9thread_monitor_try_enter_using_threadId+2223 (3da090,0,0,0)

[ 5] 0x7f012c82 J9THR23.j9thread_monitor_wait+18 (3da090,7f0f4f40,a39a8,a9210)

[ 6] 0x7f320f79 j9jit23.J9VMDllMain+2235481 (0,0,0,0)

############################################################

thread 3/59: [ nhttp: 0558: 1280]

FP=2d65feec, PC=7c8285ec, SP=2d65fe7c

stkbase=2d660000, total stksize=32768, used stksize=388

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (400,3e8,0,7c81a3ab)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (400,3e8,1039c0,3e8)

[ 3] 0x7f0119aa J9THR23.j9thread_sleep_interruptable+250 (1df47bd,0,0,3d9380)

[ 4] 0x7f0f6e09 j9jit23.j9aot_relocate_jxe+19721 (0,7f01338f,1039c0,0)

[ 5] 0x7f013370 J9THR23.j9thread_exit+256 (1039c0,0,3d7e48,7c34940f)

[ 6] 0x7f01338f J9THR23.j9thread_exit+287 (0,0,0,0)

############################################################

thread 4/59: [ nhttp: 0558: 171c]

FP=2d9cfe98, PC=7c8285ec, SP=2d9cfe28

stkbase=2d9d0000, total stksize=32768, used stksize=472

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (454,ffffffff,0,7c81a3ab)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (454,ffffffff,0,101348)

[ 3] 0x7f012270 J9THR23.j9thread_monitor_try_enter_using_threadId+1280 (3dc844,1,7f012c82,3d9c18)

[ 4] 0x7f011e12 J9THR23.j9thread_monitor_try_enter_using_threadId+162 (0,0,0,0)

############################################################

thread 5/59: [ nhttp: 0558: 1624]

FP=2da0fed0, PC=7c8285ec, SP=2da0fe60

stkbase=2da10000, total stksize=32768, used stksize=416

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (460,ffffffff,0,7c81a3ab)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (460,ffffffff,2d745ed8,a9210)

[ 3] 0x7f012270 J9THR23.j9thread_monitor_try_enter_using_threadId+1280 (0,0,3dca98,18cd8e8)

[ 4] 0x7f012bba J9THR23.j9thread_monitor_enter+58 (0,0,0,0)

############################################################

thread 6/59: [ nhttp: 0558: 0808]

FP=2da4fe84, PC=7c8285ec, SP=2da4fe14

stkbase=2da50000, total stksize=32768, used stksize=492

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (468,ffffffff,0,7c81a3ab)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (468,ffffffff,1,a88f0)

[ 3] 0x7f012270 J9THR23.j9thread_monitor_try_enter_using_threadId+1280 (3dccec,1,7f012c82,3d91c8)

[ 4] 0x7f012c23 J9THR23.j9thread_monitor_exit+35 (3d91c8,0,0,0)

[ 5] 0x7f012c82 J9THR23.j9thread_monitor_wait+18 (3d91c8,a9210,6ee50,1)

[ 6] 0x7f8ae18f j9gc23.J9VMDllMain+116863 (0,0,0,0)

############################################################

thread 7/59: [ nhttp: 0558: 097c]

FP=2e09fe78, PC=7c8285ec, SP=2e09fe08

stkbase=2e0a0000, total stksize=32768, used stksize=504

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (554,ffffffff,0,7c81a3ab)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (554,ffffffff,2d745ed8,18cd8e8)

[ 3] 0x7f012270 J9THR23.j9thread_monitor_try_enter_using_threadId+1280 (18cd928,2d745ed8,1,7f012c82)

[ 4] 0x7f01261f J9THR23.j9thread_monitor_try_enter_using_threadId+2223 (18cd8e8,0,0,0)

[ 5] 0x7f012c82 J9THR23.j9thread_monitor_wait+18 (18cd8e8,2e09ff7c,2e09ff50,190d3d0)

[ 6] 0x7f89677d j9gc23.J9VMDllMain+20077 (0,0,0,0)

############################################################

thread 8/59: [ nhttp: 0558: 14d8]

FP=2e9efd40, PC=7c8285ec, SP=2e9efcd0

stkbase=2e9f0000, total stksize=32768, used stksize=816

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (560,ffffffff,0,7c81a3ab)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (560,ffffffff,18ce498,2d73a500)

[ 3] 0x7f012270 J9THR23.j9thread_monitor_try_enter_using_threadId+1280 (18ce440,0,2d73a500,1)

[ 4] 0x7f01202d J9THR23.j9thread_monitor_try_enter_using_threadId+701 (0,0,0,0)

############################################################

thread 9/59: [ nhttp: 0558: 16cc]

FP=2fafff20, PC=7c8285ec, SP=2faffeb0

stkbase=2fb00000, total stksize=8192, used stksize=336

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (578,7530,0,2fafff4c)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (578,7530,3,2809d914)

@[ 3] 0x004fe2cc ninotes.Semaphore::Lock+28 (7530,0,2809d914,190d3d0)

@[ 4] 0x0041cc80 ninotes.CacheCleanerThread::Run+80 (194be50,2fafffb8,7c34940f,2809d914)

@[ 5] 0x004fe475 ninotes.WorkerThread::ThreadProc+21 (2809d914,0,0,190d3d0)

[ 6] 0x7c34940f MSVCR71.endthread+170 (190d3d0,0,0,190d3d0)

[ 7] 0x77e64829 kernel32.GetModuleHandleA+223 (7c3493a3,190d3d0,0,905a4d)

############################################################

thread 10/59: [ nhttp: 0558: 139c]

FP=2fd1ff00, PC=7c8285ec, SP=2fd1fe90

stkbase=2fd20000, total stksize=4096, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (580,1388,0,2fd1ff2c)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (580,1388,280be244,280bc014)

@[ 3] 0x004fe2cc ninotes.Semaphore::Lock+28 (1388,0,190d3d0,280bc014)

@[ 4] 0x0050f6ee ninotes.StatThread::Run+46 (194bee8,2fd1ffb8,7c34940f,280bc014)

@[ 5] 0x004fe475 ninotes.WorkerThread::ThreadProc+21 (280bc014,0,0,190d3d0)

[ 6] 0x7c34940f MSVCR71.endthread+170 (190d3d0,0,0,190d3d0)

[ 7] 0x77e64829 kernel32.GetModuleHandleA+223 (7c3493a3,190d3d0,0,905a4d)

############################################################

thread 11/59: [ nhttp: 0558: 0920]

FP=2ffdff08, PC=7c8285ec, SP=2ffdfe98

stkbase=2ffe0000, total stksize=262144, used stksize=360

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (628,ffffffff,0,2ffdff4c)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (628,ffffffff,280c622c,280c6242)

@[ 3] 0x6009cd26 nnotes.WaitOnNativeSemaphore@16+150 (1f5,ffffffff,0,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (280c622c,280c622c)

@[ 5] 0x2fe64685 probes.RunProbe+469 (280c6014,0,0,280c6275)

@[ 6] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 7] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 12/59: [ nhttp: 0558: 02c8]

FP=300dff08, PC=7c8285ec, SP=300dfe98

stkbase=300e0000, total stksize=262144, used stksize=360

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (634,ffffffff,0,300dff4c)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (634,ffffffff,280c6a2c,280c6a42)

@[ 3] 0x6009cd26 nnotes.WaitOnNativeSemaphore@16+150 (1ec,ffffffff,0,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (280c6a2c,280c6a2c)

@[ 5] 0x2fe64685 probes.RunProbe+469 (280c6814,0,0,280c6a75)

@[ 6] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 7] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 13/59: [ nhttp: 0558: 1720]

FP=301dff08, PC=7c8285ec, SP=301dfe98

stkbase=301e0000, total stksize=262144, used stksize=360

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (63c,ffffffff,0,301dff4c)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (63c,ffffffff,280c722c,280c7242)

@[ 3] 0x6009cd26 nnotes.WaitOnNativeSemaphore@16+150 (1f4,ffffffff,0,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (280c722c,280c722c)

@[ 5] 0x2fe64685 probes.RunProbe+469 (280c7014,0,0,280c7275)

@[ 6] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 7] 0x77e64829 kernel32.GetModuleHandleA+223 (60105380,0,0,905a4d)

############################################################

thread 14/59: [ nhttp: 0558: 1554]

FP=3066ff04, PC=7c8285ec, SP=3066fe94

stkbase=30670000, total stksize=262144, used stksize=364

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (644,c8,0,3066ff48)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (644,c8,100476c2,100476a6)

@[ 3] 0x6009cd26 nnotes.WaitOnNativeSemaphore@16+150 (1e8,c8,c8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (100476c2,c8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (c8,0,100476a6,1002a30a)

@[ 6] 0x1002a47c nhttpstack.HTTimeThread::ThreadMain+44 (100476a6,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 15/59: [ nhttp: 0558: 1084]

FP=3076fed4, PC=7c8285ec, SP=3076fe64

stkbase=30770000, total stksize=262144, used stksize=412

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (658,3e8,0,3076ff18)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (658,3e8,100475a8,100475be)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (1ed,3e8,5265c00,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (100475a8,100475a8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (5265c00,0,1004758c,3)

@[ 6] 0x100017ee nhttpstack.HTConfigurationThread::ThreadMain+174 (2082668,3076ffb8,6010542d,1004758c)

@[ 7] 0x1002a30a nhttpstack.HTThreadBeginProc@4+42 (1004758c)

@[ 8] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 9] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 16/59: [ nhttp: 0558: 07dc]

FP=3086fe1c, PC=7c8285ec, SP=3086fdac

stkbase=30870000, total stksize=262144, used stksize=596

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (668,3e8,0,3086fe60)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (668,3e8,100477d4,100477b8)

@[ 3] 0x6009cd26 nnotes.WaitOnNativeSemaphore@16+150 (1f6,3e8,3e8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (100477d4,3e8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (3e8,0,100477b8,3)

@[ 6] 0x1001858f nhttpstack.HTLoggerThread::ThreadMain+95 (2082668,3086ffb8,6010542d,100477b8)

@[ 7] 0x1002a30a nhttpstack.HTThreadBeginProc@4+42 (100477b8)

@[ 8] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 9] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 17/59: [ nhttp: 0558: 09b8]

FP=3096ff00, PC=7c8285ec, SP=3096fe90

stkbase=30970000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (678,3e8,0,3096ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (678,3e8,2ea49b84,2ea49b68)

@[ 3] 0x6009cd26 nnotes.WaitOnNativeSemaphore@16+150 (21b,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea49b84,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea49b68,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea49b68,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 18/59: [ nhttp: 0558: 0bd8]

FP=30a6ff00, PC=7c8285ec, SP=30a6fe90

stkbase=30a70000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (688,3e8,0,30a6ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (688,3e8,2ea4a65d,2ea4a641)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (205,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea4a65d,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea4a641,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea4a641,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 19/59: [ nhttp: 0558: 17d4]

FP=30b6ff00, PC=7c8285ec, SP=30b6fe90

stkbase=30b70000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (698,3e8,0,30b6ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (698,3e8,2ea4b136,2ea4b11a)

@[ 3] 0x6009cd26 nnotes.WaitOnNativeSemaphore@16+150 (21a,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea4b136,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea4b11a,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea4b11a,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 20/59: [ nhttp: 0558: 1528]

FP=30c6ff00, PC=7c8285ec, SP=30c6fe90

stkbase=30c70000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (6a8,3e8,0,30c6ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (6a8,3e8,2ea4bc0f,2ea4bbf3)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (217,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea4bc0f,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea4bbf3,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea4bbf3,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 21/59: [ nhttp: 0558: 1354]

FP=30d6ff00, PC=7c8285ec, SP=30d6fe90

stkbase=30d70000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (6b8,3e8,0,30d6ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (6b8,3e8,2ea4c6e8,2ea4c6cc)

@[ 3] 0x6009cd26 nnotes.WaitOnNativeSemaphore@16+150 (216,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea4c6e8,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea4c6cc,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea4c6cc,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (60105380,0,0,fffe0129)

############################################################

thread 22/59: [ nhttp: 0558: 170c]

FP=30efff00, PC=7c8285ec, SP=30effe90

stkbase=30f00000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (6c8,3e8,0,30efff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (6c8,3e8,2ea4d1c1,2ea4d1a5)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (20d,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea4d1c1,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea4d1a5,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea4d1a5,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 23/59: [ nhttp: 0558: 1604]

FP=30ffff00, PC=7c8285ec, SP=30fffe90

stkbase=31000000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (6d8,3e8,0,30ffff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (6d8,3e8,2ea4dc9a,2ea4dc7e)

@[ 3] 0x6009cd26 nnotes.WaitOnNativeSemaphore@16+150 (209,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea4dc9a,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea4dc7e,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea4dc7e,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 24/59: [ nhttp: 0558: 160c]

FP=310fff00, PC=7c8285ec, SP=310ffe90

stkbase=31100000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (6e8,3e8,0,310fff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (6e8,3e8,2ea4e773,2ea4e757)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (218,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea4e773,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea4e757,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea4e757,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 25/59: [ nhttp: 0558: 01b8]

FP=311fff00, PC=7c8285ec, SP=311ffe90

stkbase=31200000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (6f8,3e8,0,311fff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (6f8,3e8,2ea4f24c,2ea4f230)

@[ 3] 0x6009cd26 nnotes.WaitOnNativeSemaphore@16+150 (207,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea4f24c,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea4f230,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea4f230,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 26/59: [ nhttp: 0558: 028c]

FP=312fff00, PC=7c8285ec, SP=312ffe90

stkbase=31300000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (708,3e8,0,312fff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (708,3e8,2ea4fd25,2ea4fd09)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (1f8,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea4fd25,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea4fd09,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea4fd09,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (60105380,0,0,fffe0129)

############################################################

thread 27/59: [ nhttp: 0558: 15e4]

FP=3148ff00, PC=7c8285ec, SP=3148fe90

stkbase=31490000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (718,3e8,0,3148ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (718,3e8,2ea507fe,2ea507e2)

@[ 3] 0x6009cd26 nnotes.WaitOnNativeSemaphore@16+150 (212,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea507fe,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea507e2,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea507e2,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 28/59: [ nhttp: 0558: 0234]

FP=3158ff00, PC=7c8285ec, SP=3158fe90

stkbase=31590000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (728,3e8,0,3158ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (728,3e8,2ea512d7,2ea512bb)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (206,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea512d7,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea512bb,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea512bb,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 29/59: [ nhttp: 0558: 0904]

FP=3168ff00, PC=7c8285ec, SP=3168fe90

stkbase=31690000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (738,3e8,0,3168ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (738,3e8,2ea51db0,2ea51d94)

@[ 3] 0x6009cd26 nnotes.WaitOnNativeSemaphore@16+150 (20e,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea51db0,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea51d94,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea51d94,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 30/59: [ nhttp: 0558: 03a8]

FP=3178ff00, PC=7c8285ec, SP=3178fe90

stkbase=31790000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (748,3e8,0,3178ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (748,3e8,2ea52889,2ea5286d)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (21c,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea52889,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea5286d,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea5286d,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 31/59: [ nhttp: 0558: 1770]

FP=3188ff00, PC=7c8285ec, SP=3188fe90

stkbase=31890000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (758,3e8,0,3188ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (758,3e8,2ea53362,2ea53346)

@[ 3] 0x6009cd26 nnotes.WaitOnNativeSemaphore@16+150 (20b,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea53362,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea53346,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea53346,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (60105380,0,0,fffe0129)

############################################################

thread 32/59: [ nhttp: 0558: 17cc]

FP=31a1ff00, PC=7c8285ec, SP=31a1fe90

stkbase=31a20000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (768,3e8,0,31a1ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (768,3e8,2ea53e3b,2ea53e1f)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (219,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea53e3b,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea53e1f,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea53e1f,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 33/59: [ nhttp: 0558: 1790]

FP=31b1ff00, PC=7c8285ec, SP=31b1fe90

stkbase=31b20000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (778,3e8,0,31b1ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (778,3e8,2ea54914,2ea548f8)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (214,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea54914,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea548f8,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea548f8,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 34/59: [ nhttp: 0558: 0978]

FP=31c1ff00, PC=7c8285ec, SP=31c1fe90

stkbase=31c20000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (788,3e8,0,31c1ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (788,3e8,2ea553ed,2ea553d1)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (203,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea553ed,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea553d1,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea553d1,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 35/59: [ nhttp: 0558: 0c98]

FP=31d1ff00, PC=7c8285ec, SP=31d1fe90

stkbase=31d20000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (798,3e8,0,31d1ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (798,3e8,2ea55ec6,2ea55eaa)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (20a,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea55ec6,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea55eaa,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea55eaa,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 36/59: [ nhttp: 0558: 1634]

FP=31e1ff00, PC=7c8285ec, SP=31e1fe90

stkbase=31e20000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (7a8,3e8,0,31e1ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (7a8,3e8,2ea5699f,2ea56983)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (21d,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea5699f,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea56983,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea56983,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (60105380,0,0,fffe0129)

############################################################

thread 37/59: [ nhttp: 0558: 1328]

FP=31faff00, PC=7c8285ec, SP=31fafe90

stkbase=31fb0000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (7b8,3e8,0,31faff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (7b8,3e8,2ea57478,2ea5745c)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (210,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea57478,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea5745c,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea5745c,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 38/59: [ nhttp: 0558: 15dc]

FP=320aff00, PC=7c8285ec, SP=320afe90

stkbase=320b0000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (7c8,3e8,0,320aff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (7c8,3e8,2ea57f51,2ea57f35)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (215,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea57f51,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea57f35,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea57f35,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 39/59: [ nhttp: 0558: 07fc]

FP=321aff00, PC=7c8285ec, SP=321afe90

stkbase=321b0000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (7d8,3e8,0,321aff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (7d8,3e8,2ea58a2a,2ea58a0e)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (213,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea58a2a,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea58a0e,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea58a0e,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 40/59: [ nhttp: 0558: 04c8]

FP=322aff00, PC=7c8285ec, SP=322afe90

stkbase=322b0000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (7e8,3e8,0,322aff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (7e8,3e8,2ea59503,2ea594e7)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (211,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea59503,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea594e7,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea594e7,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 41/59: [ nhttp: 0558: 0620]

FP=323aff00, PC=7c8285ec, SP=323afe90

stkbase=323b0000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (7f8,3e8,0,323aff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (7f8,3e8,2ea59fdc,2ea59fc0)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (20f,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea59fdc,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea59fc0,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea59fc0,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (60105380,0,0,fffe0129)

############################################################

thread 42/59: [ nhttp: 0558: 0158]

FP=3253ff00, PC=7c8285ec, SP=3253fe90

stkbase=32540000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (80c,3e8,0,3253ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (80c,3e8,2ea5aab5,2ea5aa99)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (201,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea5aab5,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea5aa99,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea5aa99,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 43/59: [ nhttp: 0558: 0c24]

FP=3263ff00, PC=7c8285ec, SP=3263fe90

stkbase=32640000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (81c,3e8,0,3263ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (81c,3e8,2ea5b58e,2ea5b572)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (1ff,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea5b58e,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea5b572,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea5b572,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 44/59: [ nhttp: 0558: 0640]

FP=3273ff00, PC=7c8285ec, SP=3273fe90

stkbase=32740000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (82c,3e8,0,3273ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (82c,3e8,2ea5c067,2ea5c04b)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (1fd,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea5c067,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea5c04b,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea5c04b,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 45/59: [ nhttp: 0558: 0ca0]

FP=3283ff00, PC=7c8285ec, SP=3283fe90

stkbase=32840000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (83c,3e8,0,3283ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (83c,3e8,2ea5cb40,2ea5cb24)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (1f7,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea5cb40,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea5cb24,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea5cb24,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 46/59: [ nhttp: 0558: 12e0]

FP=3293ff00, PC=7c8285ec, SP=3293fe90

stkbase=32940000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (84c,3e8,0,3293ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (84c,3e8,2ea5d619,2ea5d5fd)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (1fb,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea5d619,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea5d5fd,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea5d5fd,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (60105380,0,0,fffe0129)

############################################################

thread 47/59: [ nhttp: 0558: 0cb0]

FP=32acff00, PC=7c8285ec, SP=32acfe90

stkbase=32ad0000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (85c,3e8,0,32acff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (85c,3e8,2ea5e0f2,2ea5e0d6)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (20c,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea5e0f2,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea5e0d6,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea5e0d6,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 48/59: [ nhttp: 0558: 14a0]

FP=32bcff00, PC=7c8285ec, SP=32bcfe90

stkbase=32bd0000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (86c,3e8,0,32bcff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (86c,3e8,2ea5ebcb,2ea5ebaf)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (208,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea5ebcb,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea5ebaf,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea5ebaf,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 49/59: [ nhttp: 0558: 0c10]

FP=32ccff00, PC=7c8285ec, SP=32ccfe90

stkbase=32cd0000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (87c,3e8,0,32ccff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (87c,3e8,2ea5f6a4,2ea5f688)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (204,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea5f6a4,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea5f688,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea5f688,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 50/59: [ nhttp: 0558: 124c]

FP=32dcff00, PC=7c8285ec, SP=32dcfe90

stkbase=32dd0000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (88c,3e8,0,32dcff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (88c,3e8,2ea6017d,2ea60161)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (202,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea6017d,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea60161,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea60161,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 51/59: [ nhttp: 0558: 1648]

FP=32ecff00, PC=7c8285ec, SP=32ecfe90

stkbase=32ed0000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (89c,3e8,0,32ecff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (89c,3e8,2ea60c56,2ea60c3a)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (200,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea60c56,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea60c3a,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea60c3a,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (60105380,0,0,fffe0129)

############################################################

thread 52/59: [ nhttp: 0558: 03c8]

FP=3305ff00, PC=7c8285ec, SP=3305fe90

stkbase=33060000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (8ac,3e8,0,3305ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (8ac,3e8,2ea6172f,2ea61713)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (1f9,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea6172f,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea61713,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea61713,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 53/59: [ nhttp: 0558: 158c]

FP=3315ff00, PC=7c8285ec, SP=3315fe90

stkbase=33160000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (8bc,3e8,0,3315ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (8bc,3e8,2ea62208,2ea621ec)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (1fc,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea62208,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea621ec,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea621ec,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 54/59: [ nhttp: 0558: 1534]

FP=3325ff00, PC=7c8285ec, SP=3325fe90

stkbase=33260000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (8cc,3e8,0,3325ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (8cc,3e8,2ea62ce1,2ea62cc5)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (1fa,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea62ce1,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea62cc5,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea62cc5,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 55/59: [ nhttp: 0558: 0c30]

FP=3335ff00, PC=7c8285ec, SP=3335fe90

stkbase=33360000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (8dc,3e8,0,3335ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (8dc,3e8,2ea637ba,2ea6379e)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (1fe,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea637ba,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea6379e,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea6379e,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 56/59: [ nhttp: 0558: 172c]

FP=3345ff00, PC=7c8285ec, SP=3345fe90

stkbase=33460000, total stksize=262144, used stksize=368

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (8ec,3e8,0,3345ff44)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (8ec,3e8,2ea64293,2ea64277)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (1eb,3e8,bb8,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (2ea64293,bb8)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (bb8,0,2ea64277,3)

@[ 6] 0x1002dc9a nhttpstack.HTWorkerThread::ThreadMain+106 (2ea64277,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (60105380,0,0,fffe0129)

############################################################

thread 57/59: [ nhttp: 0558: 081c]

FP=335eff04, PC=7c8285ec, SP=335efe94

stkbase=335f0000, total stksize=262144, used stksize=364

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (8fc,3e8,0,335eff48)

[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (8fc,3e8,1004774b,1004772f)

@[ 3] 0x6009cf23 nnotes.WaitOnNativeSemaphore@16+659 (21e,3e8,7530,0)

@[ 4] 0x600c97b9 nnotes.OSWaitEvent@8+105 (1004774b,7530)

@[ 5] 0x1000f706 nhttpstack.HTEvent::Wait+22 (7530,0,1004772f,1002a30a)

@[ 6] 0x10032dbc nhttpstack.HTStatsThread::ThreadMain+44 (1004772f,0,0,0)

@[ 7] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[ 8] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 58/59: [ nhttp: 0558: 1344]

FP=336eeda0, PC=7c8285ec, SP=336eed64

stkbase=336f0000, total stksize=262144, used stksize=4764

############################################################

[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (904,3b4,1,4)

[ 2] 0x71b21a03 MSWSOCK+6659 (2,336ef00c,336eef08,336ef110)

[ 3] 0x71c0283c WS2_32.select+161 (2,336ef00c,336eef08,336ef110)

@[ 4] 0x62c550ca nNTCP.TCPEndp_CheckSocketReady@12+1178 (2,336ef370,c8)

@[ 5] 0x62c5579f nNTCP.cmd_poll@16+463 (0,336ef370,336eff7c,336ef320)

@[ 6] 0x600b5387 nnotes.nti_poll@12+359 (c8,2,336eff7c)

@[ 7] 0x1000159f nhttpstack.HTAcceptThread::ThreadMain+367 (2082668,336effb8,6010542d,1004761d)

@[ 8] 0x1002a30a nhttpstack.HTThreadBeginProc@4+42 (1004761d)

@[ 9] 0x6010542d nnotes.ThreadWrapper@4+173 (0)

[10] 0x77e64829 kernel32.GetModuleHandleA+223 (0,0,0,0)

############################################################

thread 59/59: [ nhttp: 0558: 05b0]

FP=017bfff4, PC=7c81a3e2, SP=017bffcc

stkbase=017c0000, total stksize=16384, used stksize=52

############################################################

[ 1] 0x7c81a3e2 ntdll.DbgBreakPoint+1 (0,0,0,0)