Our SMTP server has all of a sudden started locking up…only thing that has changed is the upgrade to 8.5.2. It will send emails out, but after half a day or so, it will stop routing internet emails internally. The server just becomes very slow and sluggish and the only thing I can do to correct it is to reboot the box.
Anyone else seeing issues like this?
Subject: Us too…
This has to be a fairly severe problem. We don’t move that much mail across the server that we are experiencing this with but someone must send us something at around 11:30 every day that triggers it.
Rob
Subject: domino 7.04 same SMTP issue.
we are using domino 7.04, our SMTP servcie CPU usuage always from 0 to 11% when have new msg come in.
So, is my 7.0.4 hacve same issue? by the way IBM said 7.04 no any support now, so is it mean I only can do is upgrade to 8.5.x? thanks
Because I have same SMTP slow preferoamnce issue in our 7.0.4 , but still no idea.
thanks
Subject: New Technote #1445280
Subject: Thanks!
Thanks for the info, Mark…I appreciate it.
Subject: SMTP CPU Spike 50-100%
Hi Steve and Mark,
Yes this issue has been reported to IBM and we have reproduced this issue in house on Windows and AIX servers. We are currently working on a fix for this issue and we have documented this Under Technote:
http://www-01.ibm.com/support/docview.wss?uid=swg21445280
and it is being tracked under the following new SPR#LMIL88Q62X
When You run an NSD it may take a long time however you will see:
On Windows:
############################################################
############################################################
@[ 1] 0x600013e6 nnotes.OSUnlockSem@4+54 (15bb72b4)
@[ 2] 0x601df473 nnotes.UnlockMemHandle@4+19 (15bb728c)
@[ 3] 0x60002c31 nnotes.OSUnlockObject@4+129 (611e9090)
@[ 4] 0x6001b68c nnotes.NSFAddToNameLookupTable@12+268 (15c22208,1ed,611e7718)
@[ 5] 0x6001dd2a nnotes.ItemAppendByBLOCKID@32+570 (150dd8b4,4,60eebc54,4,0,150dd870,f10f10,ffffffff)
@[ 6] 0x60048ecd nnotes.NSFItemAppendByBLOCKID@32+93 (59,3,60eebc54,15c22110,0,150dd8a0,f10f10,ffffffff)
@[ 7] 0x600522e0 nnotes.NSFItemAppend@28+160 (59,3,60eebc54,4,0,150dd8dc,f10f10)
@[ 8] 0x60358756 nnotes.NSFMimePartAppendTG@36+182 (59,60eebc54,4,2,0,150dd928,f10f10,ffffffff,0)
@[ 9] 0x60351e29 nnotes.MIMEItemizeMessageBodyItemAppend+2393 (150deb30,59,60eebc54,4)
@[10] 0x603527da nnotes.MIMEItemizeMessageBody+1162 (150deb30,59,60eebc54,4)
@[11] 0x603522d4 nnotes.MIMEItemizeMessage+804 (150deb30,59,60eebc54,4)
@[12] 0x603525b7 nnotes.MIMEItemizeMessageBody+615 (150deb30,59,60eebc54,4)
@[13] 0x603522d4 nnotes.MIMEItemizeMessage+804 (150deb30,59,60eebc54,4)
@[14] 0x603525b7 nnotes.MIMEItemizeMessageBody+615 (150deb30,59,60eebc54,4)
@[15] 0x603522d4 nnotes.MIMEItemizeMessage+804 (150deb30,59,60eebc54,4)
@[16] 0x60352a74 nnotes.MIMEItemizeUsingCallbackExt@32+372 (59,60eebc54,4,7,0,150deb40,f10f10,ffffffff)
@[17] 0x60352cad nnotes.MIMEItemizeStreamExt@28+93 (59,0,4,7,0,150dee78,f10f10)
@[18] 0x603531a2 nnotes.MIMEItemizeStream@24+34 (59,0,0,7,0,150dee9c)
@[19] 0x602b24ed nnotes.CIMsgImport::Import+221 (14f513b8,397030,59,3)
@[20] 0x004136d3 nSMTP.CSMTPProtocol::SubmitMessage+467 (14f513b8,150dfc6a,2,14648018)
@[21] 0x0041192f nSMTP.CSMTPProtocol::CommandDATA+1407 (8018,2,14648018,0)
@[22] 0x00412bb6 nSMTP.CSMTPProtocol::StateConnected+1158 (14648018,390000,397010,397008)
@[23] 0x00412ecd nSMTP.CSMTPProtocol::Run+429 (14648018,128de6c,43bf68,128de6c)
@[24] 0x0042183e nSMTP.CBaseTask::StateMachine+398 (397008,128de6c,43bf68,128de88)
@[25] 0x00402fe3 nSMTP.CSMTPSrv::OnConnect+211 (43bf68,128de88,13d80001,128de6c)
@[26] 0x004173a6 nSMTP.CIServ::ServerTaskProtocolMachine+262 (43bf68,128de6c,1232ee8,12357f0)
@[27] 0x0041bd9a nSMTP.CIServ::ServerTaskIOCP+922 (43bf68,0,6015584f,0)
@[28] 0x0041c71d nSMTP.ServerThread@4+29 (0)
[29] 0x77e6482f kernel32.GetModuleHandleA+223 (601557a0,0,0,dddd04d2)
also we saw this stack as well:
############################################################
############################################################
@[ 1] 0x600013e6 nnotes.OSUnlockSem@4+54 (18ff3808)
@[ 2] 0x601df473 nnotes.UnlockMemHandle@4+19 (18ff37e0)
@[ 3] 0x601df5cf nnotes.OSLockObjectExt@8+143 (611e9090,611e9142)
@[ 4] 0x60002b7e nnotes.OSLockObject@4+14 (94)
@[ 5] 0x6001b694 nnotes.NSFAddToNameLookupTable@12+276 (a40a08,1ef,611ed988)
@[ 6] 0x6001dd2a nnotes.ItemAppendByBLOCKID@32+570 (1978c138,4,60eebc54,4,0,1978c0f4,f10f10,ffffffff)
@[ 7] 0x60048ecd nnotes.NSFItemAppendByBLOCKID@32+93 (83,3,60eebc54,a40910,0,1978c124,f10f10,ffffffff)
@[ 8] 0x600522e0 nnotes.NSFItemAppend@28+160 (83,3,60eebc54,4,0,1978c160,f10f10)
@[ 9] 0x60358756 nnotes.NSFMimePartAppendTG@36+182 (83,60eebc54,4,2,0,1978c1ac,f10f10,ffffffff,0)
@[10] 0x60351e29 nnotes.MIMEItemizeMessageBodyItemAppend+2393 (1978d3b4,83,60eebc54,4)
@[11] 0x603527da nnotes.MIMEItemizeMessageBody+1162 (1978d3b4,83,60eebc54,4)
@[12] 0x603522d4 nnotes.MIMEItemizeMessage+804 (1978d3b4,83,60eebc54,4)
@[13] 0x603525b7 nnotes.MIMEItemizeMessageBody+615 (1978d3b4,83,60eebc54,4)
@[14] 0x603522d4 nnotes.MIMEItemizeMessage+804 (1978d3b4,83,60eebc54,4)
@[15] 0x603525b7 nnotes.MIMEItemizeMessageBody+615 (1978d3b4,83,60eebc54,4)
@[16] 0x603522d4 nnotes.MIMEItemizeMessage+804 (1978d3b4,83,60eebc54,4)
@[17] 0x60352a74 nnotes.MIMEItemizeUsingCallbackExt@32+372 (83,60eebc54,4,13,0,1978d3c4,f10f10,ffffffff)
@[18] 0x60352cad nnotes.MIMEItemizeStreamExt@28+93 (83,60eebc54,4,13,0,1978d6fc,f10f10)
@[19] 0x603531a2 nnotes.MIMEItemizeStream@24+34 (83,60eebc54,4,13,0,1978d720)
@[20] 0x602afd79 nnotes.CIMsgImport::ImportConvertDeliveryStatus+505 (19891498,83,1,1978f000)
@[21] 0x602b0ba8 nnotes.CIMsgImport::ImportConvertHeaders+216 (19891498,c736d0,83,1978f000)
@[22] 0x602b2587 nnotes.CIMsgImport::Import+375 (19891498,c736d0,83,3)
@[23] 0x004136d3 nSMTP.CSMTPProtocol::SubmitMessage+467 (19891498,1978fc6a,2,19cba018)
@[24] 0x0041192f nSMTP.CSMTPProtocol::CommandDATA+1407 (a018,2,19cba018,656e6968)
@[25] 0x00412bb6 nSMTP.CSMTPProtocol::StateConnected+1158 (19cba018,c70000,c736b0,c736a8)
@[26] 0x00412ecd nSMTP.CSMTPProtocol::Run+429 (19cba018,12ddd64,43bf68,12ddd64)
@[27] 0x0042183e nSMTP.CBaseTask::StateMachine+398 (c736a8,12ddd64,43bf68,12ddd80)
@[28] 0x00402fe3 nSMTP.CSMTPSrv::OnConnect+211 (43bf68,12ddd80,12240001,12ddd64)
@[29] 0x004173a6 nSMTP.CIServ::ServerTaskProtocolMachine+262 (43bf68,12ddd64,1282ee8,12857f0)
@[30] 0x0041bd9a nSMTP.CIServ::ServerTaskIOCP+922 (43bf68,0,6015584f,0)
@[31] 0x0041c71d nSMTP.ServerThread@4+29 (0)
[32] 0x77e6482f kernel32.GetModuleHandleA+223 (0,0,0,0)
On AIX we see:
Thread consuming CPU
###################################
###################################
[1] 0x0000000000003438 _check_lock() + 0x18
[2] 0x0000000110ee5c8c ???(??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??) + ??
[3] 0x09000000006ede70 UnlockHandle(0x1100ac798) + 0x1c
[4] 0x09000000006ef054 OSLockObjectExt(0x1d0000001d, 0x0) + 0xd4
[5] 0x09000000006ef1dc OSLockObject(0x1d0000001d) + 0x1c
[6] 0x0900000000b4fd6c NSFAddToNameLookupTable(0x11019df08, 0x2deb38000a) + 0x218
[7] 0x0900000000b665e8 ItemAppendByBLOCKID(0x110ee6070, 0x3000000000003, 0x9000000028cddd8, 0x4000000000004, 0x2deaf0003a, 0x3c0000003c, 0x0) + 0x254
[8] 0x0900000000b67430 NSFItemAppendByBLOCKID(0xa0000000a, 0x3000000000003, 0x9000000028cddd8, 0x4000000000004, 0x2deaf0003a, 0x3c0000003c, 0x0) + 0xec
[9] 0x0900000000b67d3c NSFItemAppend(0xa0000000a, 0x3000000000003, 0x9000000028cddd8, 0x4000000000004, 0x19000000000019, 0x11003cce0, 0x3a0000003a) + 0x134
[10] 0x0900000000ea9b68 NSFMimePartAppendTG(0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x2000000000002, 0x100000001, 0x110ee6408, 0x26000000000026, 0x26000000000026) + 0x170
[11] 0x090000000102f47c MIMEItemizeMessageBodyItemAppend(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,tagMIMEPartType,int,int,tagMimeWellKnownSymbolId,tagMimeWellKnownSymbolId,tagMimeWellKnownSymbolId,tagMimeWellKnownSymbolId,CMemStr&,CMemStr&,CMemStr&,CStream*,CMemStr&)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x200000002, 0x100000001, 0x100000001) + 0xe48
[12] 0x090000000102fa7c MIMEItemizeMessageBody(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,CMIMEReadBuffer&,CMimeParse&)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x110ee84d0, 0x110ee7200) + 0x364
[13] 0x090000000102d108 MIMEItemizeMessage(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,CMIMEReadBuffer&,CMimeParse&,unsigned int*)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x110ee84d0, 0x110ee7200, 0x0) + 0x4c8
[14] 0x090000000102f988 MIMEItemizeMessageBody(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,CMIMEReadBuffer&,CMimeParse&)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x110ee84d0, 0x110ee7b90) + 0x270
[15] 0x090000000102d108 MIMEItemizeMessage(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,CMIMEReadBuffer&,CMimeParse&,unsigned int*)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x110ee84d0, 0x110ee7b90, 0x0) + 0x4c8
[16] 0x090000000102f988 MIMEItemizeMessageBody(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,CMIMEReadBuffer&,CMimeParse&)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1200000013, 0x110ee84d0, 0x110ee8510) + 0x270
[17] 0x090000000102d108 MIMEItemizeMessage(tagAttachContext*,unsigned int,char*,unsigned short,unsigned int,CMIMEReadBuffer&,CMimeParse&,unsigned int*)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1300000013, 0x110ee84d0, 0x110ee8510, 0x0) + 0x4c8
[18] 0x090000000102c2e8 MIMEItemizeUsingCallbackExt(0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1300000013, 0x9001000a0781240, 0x110ee88b0, 0x0, 0x0) + 0x178
[19] 0x090000000102bb04 MIMEItemizeStreamExt(0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1300000013, 0x110ee9760, 0x0, 0x0) + 0xc4
[20] 0x090000000102ba04 MIMEItemizeStream(0xa0000000a, 0x9000000028cddd8, 0x4000000000004, 0x1300000013, 0x110ee9760, 0x0) + 0x44
[21] 0x0900000001038a7c CIMsgImport::ImportConvertDeliveryStatus(unsigned int,int,unsigned int*)(0x1100d2cf8, 0xa0000000a, 0x100000001, 0x110eeac44) + 0x364
[22] 0x0900000001035880 CIMsgImport::ImportConvertHeaders(CStream&,unsigned int,unsigned int*,CONVERSION_CONTROLS*,int)(0x1100d2cf8, 0x1113161e8, 0xa0000000a, 0x110eeac44, 0x700000004ee64a0, 0x100000001) + 0x11f0
[23] 0x09000000010343c4 CIMsgImport::Import(CStream&,unsigned int,unsigned short,unsigned int,CONVERSION_CONTROLS*)(0x1100d2cf8, 0x1113161e8, 0xa0000000a, 0x3000000000003, 0x2700000027, 0x700000004ee64a0) + 0x204
[24] 0x000000010002f7b8 CSMTPProtocol::SubmitMessage(unsigned short*)(0x1100eb638, 0x110eebcc0) + 0x308
[25] 0x000000010002a418 CSMTPProtocol::CommandDATA()(0x1100eb638) + 0x138
[26] 0x00000001000267c0 CSMTPProtocol::StateConnected()(0x1100eb638) + 0x25c
[27] 0x0000000100024630 CSMTPProtocol::Run()(0x1100eb638) + 0x29c
[28] 0x000000010001bd78 CBaseTask::StateMachine()(0x1113161b0) + 0x25b0
[29] 0x00000001000049f0 CSMTPSrv::OnConnect(SESSIONID&,unsigned int,INETTASK*)(0x1100005c8, 0x1101e9068, 0x110d0000110d0, 0x1101e9038) + 0x98
[30] 0x0000000100013f10 CIServ::ServerTaskProtocolMachine(INETTASK*)(0x1100005c8, 0x1101e9038) + 0x5c8
[31] 0x00000001000122c4 CIServ::ServerTaskIOCP(int)(0x1100005c8, 0x100000001) + 0x7ac
[32] 0x0000000100011a8c ServerThread(0x0) + 0x30
[33] 0x09000000006b6900 ThreadWrapper(0x0) + 0x118
[34] 0x09000000004a34f4 _pthread_body(??) + 0xdc
Subject: nSMTP CPU Utilization Issue
Just wanted to add some feedback.
We upgraded our servers within days of the web release of 8.5.2, and didn’t have this problem.
However over the last few days I have upgraded to ODS51 on all servers, and now we’re experiencing this issue.
Now I can’t see how the two could be related, but to not have the problem for nearly two months on 8.5.2, and then the issues arises on two servers, within days of the ODS upgrade?
I can confirm that the fix resolves the problem, in fact I have one server with the fix applied, and the second without. The server without the fix is running SMTP at 25%-50% continously, has to be rebooted periodically, and Domino fails to shutdown cleanly.
Other factors that are more likely to contribute are that we are running Qmail (with SpamAssassin) SMTP relays on the front-end.
Thanks for the fix.
Steve
Subject: Same here
The SMTP task is running near 100% and eventually stops routing. We upgraded a little over 24 hours ago from 8.5.1 FP4 to 8.5.2. It was fine for a while and suddenly mail started piling up on other servers. Only a reboot will fix (for a short time.) On quiting, the server always hangs for a long time (> 5 minutes) and eventually crashes generating an NSD.
We opened a PMR and got told “…issues with 8.5.2, we are working patch…”
We’re downgrading back to 8.5.1 FP4 for now.
So much for the QA process.
Subject: Fix available for download on Fix Central
Go to IBM Support: Fix Central
Subject: More on the issue
We discovered that certain messages that were malformed, would lead us down a code path that got us caught in an infinite loop. A bug fix in 8.5.2, takes us down the code path where the exposure is which 8.5.1 did not. We are finding that certain anti-spam products that modify the messages, contribute to putting the message in the malformed state, i.e. Starting a section but not closing it properly. It is good practice to handle this case in our code. We handle other similar cases defensively all throughout MIME but this case was not handled. Preliminary testing of the fix is going well. We are undergoing production deployment testing internally and externally and as mentioned in the technote, if everything goes smoothly, will have a fix posted on fix central by Sep 8th. Thank you for your patience and we are taking this test escape back into our processes to determine where our coverage was insufficient so we can improve.