My server crashed 6 times today after being up and running without incident for 4 or 5 months. I’ve got a PMR open. Lotus Support says they are getting a lot of calls today about crashes. They believe an email is corrupting mail.box and bringing down the server.
Are others having the same problem?
They said to stop the SMTP task for now. Or enable SMTP debugging and send the logs.
Thanks,
Greg
Subject: Are there a lot of server crashes today?
Greg,
It appears to be SMTP but IBM hasn’t come up with a solution yet. We are experiencing the same thing here. Hopefully this will be fixed soon.
Subject: RE: Are there a lot of server crashes today?
I’m seeing this too. SMTP service related Domino crashes. It’s related to images I think in emails which are probably attempted exploits. MIME conversion causes the crash.
In my case, initially shutting down SMTP solved the problem but that is unacceptable in my architecture and so I’ve subsequently re-enabled SMTP but disabled POP3 which is the only client that uses MIME conversion except the Webmail on my server. Thats given me some breathing space while still allowing me to receive and forward email.
Subject: RE: Are there a lot of server crashes today?
Grant,How is that working for you? It is necessary to stop http and POP3?
Thanks,
Greg
Subject: RE: Are there a lot of server crashes today?
We had the same problems yesterday until with random crashes, on SMTP conversion. We’re running R6.5.5 FP1, HF3 on 2003 Server.
After we applied the fix suggested in this technote and our server has been stable.
http://www-1.ibm.com/support/docview.wss?fdoc=wplcdomi&rs=463&uid=swg21247835
Duane.
Subject: RE: Are there a lot of server crashes today?
Hi,
Can you post your FATAL THREAD from your NSD?
JYR
Subject: RE: Fatal Thread
############################################################### FATAL THREAD 22/74 [ nserver: 1728: 2440]
Exception code: c0000094 (INT_DIVIDE_BY_ZERO)
############################################################
Subject: RE: Fatal Thread
Hi Greg,
can you post the 5-20 lines just under this fatal thread?
thank you,
JYR
Subject: RE: Fatal Thread
Thanks for your response. You are correct, the 1st fatal threat points to nserver. The 2nd through 6th NSD point to the nsmtp in the fatal thread. Here’s from the 1st and 2nd NSD:
1ST FATAL THREAD.
############################################################
Exception code: c0000094 (INT_DIVIDE_BY_ZERO)
############################################################
@[ 1] 0x60004ccf nnotes._OSAddressInVARRAY@8+63 (43cd23b0,1)
@[ 2] 0x60007953 nnotes._STMFillPinAccessDescNSF@24+67 (d09cce8,1,0,0,6bdf090,ad857c)
@[ 3] 0x60006ad6 nnotes._BUFPin@24+166 (6bdf5f8,1ccc05,60c6d020,1,6bdf090,ad857c)
@[ 4] 0x60006d7a nnotes._iNSFSlotPin@24+122 (6bdf5f8,1ccc05,60c6d020,1,6bdf090,ad857c)
@[ 5] 0x60006ccb nnotes._NSFSlotPinCtx@24+235 (6bdf5f8,1ccc05,60c6d020,1,6bdf090,ad857c)
@[ 6] 0x60016cc1 nnotes._NSFSlotPin@24+49 (264,1ccc05,60c6d020,1,6bdf090,ad857c)
@[ 7] 0x6002993f nnotes._NIFLockAtomTable@8+95 (4248af24,4248b0c4)
@[ 8] 0x60029aa5 nnotes._InitCtx@8+149 (42480001,1)
@[ 9] 0x600639a5 nnotes._ReadEntries@68+725 (2b7e0fc,4208ea4a,8010,0,6bdf090,ad857c,3ac,1,6bdf668,0,a,600639a5,a,6bdfc98,1003f,0,0)
@[10] 0x608f0c99 nnotes._NIFReadEntriesExt@76+361 (0,4208ea4a,8010,0,6bdf090,ad857c,3ac,1,6bdfc98,0,a,608f0c99,b,6bdfd0c,1003f,0,0,0,0)
@[11] 0x10035d0a nserverl._ServerReadEntries@8+1498 (4208e8f4,65940008)
@[12] 0x1001a737 nserverl._DbServer@8+2327 (974d0044,84ec0003)
@[13] 0x1002de4d nserverl._WorkThreadTask@8+1597 (492183c,0)
@[14] 0x100016cb nserverl._Scheduler@4+763 (0)
@[15] 0x60116d74 nnotes._ThreadWrapper@4+212 (0)
@[16] 0x7c57b396 KERNEL32.lstrcmpiW+197
############################################################
Exception code: c0000094 (INT_DIVIDE_BY_ZERO)
############################################################
# 06bdf43c 0d09cce8 00000001 00000000 6000110d |...............`|
# 06bdf44c 6000114a 600011cc 446f6303 446f62de |J..`...`.coD.boD|
# 06bdf45c 06bdf46c 600010be 446f6303 60001216 |l......`.coD...`|
# 06bdf46c 6000127b 446f6303 446f62de 06bdf484 |{..`.coD.boD....|
# 06bdf47c 60001833 446f6303 06bdf4a0 6000296e |3..`.coD....n).`|
# 06bdf48c 446f62de 000006c0 00000001 0d09cce8 |.boD............|
# 06bdf49c 00ba0014 06bdf4b4 60001216 6000127b |...........`{..`|
# 06bdf4ac 0d09cebc 0d09cce8 06bdf4cc 60007432 |............2t.`|
# 06bdf4bc 0d09cebc 0d09cce8 00000001 06bdf4e8 |................|
@[ 1] 0x60004ccf nnotes._OSAddressInVARRAY@8+63 (43cd23b0,1)
# 06bdf4c8 06bdf4e8 60007953 43cd23b0 00000001 |....Sy.`.#.C....|
# 06bdf4d8 00000001 0d09cce8 4248b186 00000000 |..........HB....|
@[ 2] 0x60007953 nnotes._STMFillPinAccessDescNSF@24+67 (d09cce8,1,0,0,6bdf090,ad857c)
# 06bdf4e8 06bdf548 60006ad6 0d09cce8 00000001 |H....j.`........|
# 06bdf4f8 00000000 00000000 00000000 06bdf514 |................|
# 06bdf508 4248b186 4248b182 00000000 06bdf528 |..HB..HB....(...|
# 06bdf518 60003d20 000005dc 00000001 00000000 | =.`............|
# 06bdf528 06bdf538 60005e8a 0d09ccea 06bdf5f8 |8....^.`........|
# 06bdf538 06bdf554 60005e08 00000000 06bdf584 |T....^.`........|
@[ 3] 0x60006ad6 nnotes._BUFPin@24+166 (6bdf5f8,1ccc05,60c6d020,1,6bdf090,ad857c)
# 06bdf548 06bdf578 60006d7a 06bdf5f8 001ccc05 |x...zm.`........|
# 06bdf558 60c6d020 00000001 00000000 4248b186 | ..`..........HB|
# 06bdf568 00000000 06bdf5f8 4248b0c4 00000000 |..........HB....|
@[ 4] 0x60006d7a nnotes._iNSFSlotPin@24+122 (6bdf5f8,1ccc05,60c6d020,1,6bdf090,ad857c)
# 06bdf578 06bdf5d4 60006ccb 06bdf5f8 001ccc05 |.....l.`........|
# 06bdf588 60c6d020 00000001 00000000 4248b182 | ..`..........HB|
# 06bdf598 00000000 4248af24 00010001 00000264 |....$.HB....d...|
# 06bdf5a8 06bdf5f8 00000000 00000000 00000000 |................|
# 06bdf5b8 00000007 0127c80c 00f46005 00000000 |......'..`......|
# 06bdf5c8 4248af24 4248b0c4 00000000 06bdf618 |$.HB..HB........|
@[ 5] 0x60006ccb nnotes._NSFSlotPinCtx@24+235 (6bdf5f8,1ccc05,60c6d020,1,6bdf090,ad857c)
# 06bdf5d4 06bdf618 60016cc1 06bdf5f8 001ccc05 |.....l.`........|
# 06bdf5e4 60c6d020 00000001 00000000 4248b182 | ..`..........HB|
# 06bdf5f4 4248af24 1341e2f4 427c60e8 00000264 |$.HB..A..`|Bd...|
# 06bdf604 00000000 00000000 00000000 22c18704 |..............."|
# 06bdf614 00000000 06bdf644 6002993f 00000264 |....D...?..`d...|
@[ 6] 0x60016cc1 nnotes._NSFSlotPin@24+49 (264,1ccc05,60c6d020,1,6bdf090,ad857c)
# 06bdf618 06bdf644 6002993f 00000264 001ccc05 |D...?..`d.......|
# 06bdf628 60c6d020 00000001 00000000 4248b182 | ..`..........HB|
# 06bdf638 02b7e0fc 4248af24 4248b0c4 06bdf668 |....$.HB..HBh...|
@[ 7] 0x6002993f nnotes._NIFLockAtomTable@8+95 (4248af24,4248b0c4)
# 06bdf644 06bdf668 60029aa5 4248af24 4248b0c4 |h......`$.HB..HB|
# 06bdf654 00000000 06bdfdba 4248af24 00000000 |........$.HB....|
# 06bdf664 00000000 06bdfc98 600639a5 42480001 |.........9.`..HB|
@[ 8] 0x60029aa5 nnotes._InitCtx@8+149 (42480001,1)
# 06bdf668 06bdfc98 600639a5 42480001 00000001 |.....9.`..HB....|
# 06bdf678 02b70000 4248af24 02b7e0fc 00000000 |....$.HB........|
# 06bdf688 1bc0d0c2 1bc0d0c2 2b2cf174 0000000e |........t.,+....|
# 06bdf698 06bdf6b0 00000001 1bc0cfb0 2b2cf112 |..............,+|
# 06bdf6a8 00000000 06bdf6c4 00000000 00000000 |................|
# 06bdf6b8 1bc0cfd0 2b2cf148 00000000 06bdf6e0 |....H.,+........|
# 06bdf6c8 06bdf6a0 2b2cf174 06bdffdc 7c5c216c |....t.,+....l!\||
# 06bdf6d8 7c571e98 ffffffff 06bdf714 6097c548 |..W|........H..`|
# 06bdf6e8 00000fe4 2b2cf174 0000000e 06bdf70c |....t.,+........|
# 06bdf6f8 6097c550 00000000 00000000 2b2cf112 |P..`..........,+|
# 06bdf708 00000fe4 0000000e 0005f112 06bdf730 |............0...|
# 06bdf718 60976d1a 0003f112 00000000 2b2cf112 |.m.`..........,+|
# 06bdf728 00000fe4 00055ce0 06bdf764 600c2e2e |.....\..d......`|
# 06bdf738 0003f112 00000001 4269b650 65940008 |........P.iB...e|
# 06bdf748 86d40001 00000000 00000000 00000000 |................|
# 06bdf758 00000000 00000000 62614550 06bdf788 |........PEab....|
# 06bdf768 600c2335 2b2c0003 4269b650 86d40001 |5#.`..,+P.iB....|
# 06bdf778 86d40001 0000000c 4269bcf0 4269bcf0 |..........iB..iB|
# 06bdf788 06bdf79c 600c24dc 86d40001 65940008 |.....$.`.......e|
# 06bdf798 65940008 06bdf7bc 600b746b 60ce98a0 |...e....kt.`...`|
# 06bdf7a8 65940008 0002c5d5 4269b650 00000000 |...e....P.iB....|
# 06bdf7b8 000006c0 06bdf7ec 600b7060 86d40001 |........`p.`....|
# 06bdf7c8 0000004c 65940008 4208e8f4 ffff0000 |L......e...B....|
# 06bdf7d8 00000000 0000004c 013ff4f8 00000000 |....L.....?.....|
# 06bdf7e8 4269bcf0 06bdf818 1001e986 65940008 |..iB...........e|
# 06bdf7f8 0002c5d5 4208e8f4 06bdfe44 65940008 |.......BD......e|
# 06bdf808 00020001 004108d3 10080001 2b2cf180 |......A.......,+|
# 06bdf818 06bdf83c 1001bf01 4208e8f4 06bdfe44 |<..........BD...|
# 06bdf828 00000001 4208e8f4 2b2c0000 65940008 |.......B..,+...e|
# 06bdf838 84ec0003 06bdfe48 10020716 4208e8f4 |....H..........B|
# 06bdf848 06bdfe44 00000001 00000000 4208e8f4 |D..............B|
# 06bdf858 00000001 06bdfd94 15f60380 01217df0 |.............}!.|
# 06bdf868 600011cc 010c35b0 010c356c 06bdf884 |...`.5..l5......|
# 06bdf878 029cb168 014e3218 06bdf964 014e3218 |h....2N.d....2N.|
# 06bdf888 029cb168 00000011 00000001 029cb168 |h...........h...|
# 06bdf898 00000000 029ce8e8 00000000 00000002 |................|
# 06bdf8a8 00004000 00004380 00004dc8 6000110d |.@...C...M.....`|
# 06bdf8b8 6000114a 600011cc 06bdfe70 60ad50c5 |J..`...`p....P.`|
# 06bdf8c8 000000ff 600010be 60001216 6000127b |.......`...`{..`|
# 06bdf8d8 01219282 06bdf9ac 607809e7 06bdfd94 |..!.......x`....|
# 06bdf8e8 00000000 06bdfa04 06bdf930 00140280 |........0.......|
2ND FATAL THREAD:
############################################################
Exception code: c0000005 (ACCESS_VIOLATION)
############################################################
[ 1] 0x6b79686f (95600001,4b7f,308f0000,308f1938)
@[ 2] 0x60709a29 nnotes.CNetSession::SendInternal_Q+329 (1492dc4,40,1531836,0)
@[ 3] 0x60708e52 nnotes.CNetSession::SendstrWait+530 (1531836,308f0050,308f0048,393efcb4)
@[ 4] 0x60708270 nnotes.CNetSession::WaitProc+496 (308f1938,1531836,60475b8,434c28)
@[ 5] 0x0041c39b nSMTP.CBaseTask::StateMachine+1195 (308f0048,60475b8,434c28,60475d4)
@[ 6] 0x00401c24 nSMTP.CSMTPSrv::OnConnect+180 (434c28,60475d4,110c0001,60475b8)
@[ 7] 0x0041873c nSMTP.CIServ::ServerTaskProtocolMachine+268 (434c28,60475b8,3,1530084)
@[ 8] 0x00417f9c nSMTP.CIServ::ServerTaskIOCP+1052 (0,0,60116d74,0)
@[ 9] 0x0041740d nSMTP._ServerThread@4+29 (0)
@[10] 0x7c57b396 KERNEL32.lstrcmpiW+197 (60116ca0)
############################################################
Exception code: c0000005 (ACCESS_VIOLATION)
Subject: RE: Fatal Thread
Hi,
Take a look at this technote. Has a very similar call stack. Could be a regression??
Kjeld
Subject: RE: Are there a lot of server crashes today?
Hi Client,
your FATAL THREAD is nsmtp?
Greg’s one is nserver.
Thank you,
JYR
Subject: RE: Are there a lot of server crashes today?
Between everything I am seeing in the logs and what lotus is telling me from my PMR that I opened with them, yes, it’s SMTP. We actually were about to set up clustering for failover so we had a 7.0.2 server that was on stand by. The server that kept crashing yesterday was 6.5.5 BTW. On the suggestion of IBM, we took SMTP down on the 6 server. Then while waiting for a response back we decided to start SMTP on the 7 server and see what happened. No crashes. We then used our 7 server for the SMTP process then routed it to the 6 server. Not a single crash last night or today so far. I am still waiting on a call back from IBM with a fix. Let me know if there is anything else I can answer.
Subject: RE: Are there a lot of server crashes today?
Hi Clint,
be sure to check:
SMTP task crashes on Domino 7.0.2 when TNEFEnableConversion=1 is enabled
JYR