Server crashes every day

hcl-bot · January 22, 2007, 4:07pm

couple of weeks back, our NT team installed a minor patch on the mail gateway server. Then the windows server got crashed because of the bad disk. later they have got that corrected and Microsoft fixed the registry for that server. When they restarted, the domino server didn’t restarted and given an error which needed me to remove the log.nsf. Then server started crashing like almost every hour at that time… the server crashed whenever it tried to update the fulltext index. I have removed the update task on the server tasks. contacted IBM and they told that I need to upgrade to 6.5.5 fix 2. But … couple of days back… the SMTP is failing to send mails… but able to receive external mails… I don’t know what exactly causing this.

hcl-bot · January 22, 2007, 4:17pm

Subject: Server crashes every day

Back up your certs, databases, and settings and reinstall bud.

hcl-bot · January 22, 2007, 5:25pm

Subject: RE: Server crashes every day

Hi,You need to post more infos,

Post the FATAL THREAD section , and the few lines under, of your nsd.

Ok look in the forum FAQ, FAQ of FAQs you will find links under NSD - crashes and hangs.

JYR

hcl-bot · January 24, 2007, 11:57am

Subject: RE: Server crashes every day

Attached the Fatal thread info from the NSD

############################################################

FATAL THREAD 4/6 [ nAMgr:08f4:0e00]

FP=0x0c16fdd0, PC=0x60191907, SP=0x0c16f258, stksize=2936

EAX=0x0114088c, EBX=0x00000000, ECX=0x00c10000, EDX=0x00c10000

ESI=0x0c16f964, EDI=0x136a84c4, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x00000038, GS=0x00000000 Flags=0x00010202

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x60191907 nnotes._Panic@4+631 (609f0016,5,bfbb550,c16fe04)

@[ 2] 0x600028bc nnotes._LockMemHandle@12+348 (f01c1d9f,c16fe00,c16fe0c,5)

@[ 3] 0x600026c7 nnotes._OSMemoryLock@4+23 (60c40a40,5,bfbb550,101)

@[ 4] 0x6000573f nnotes._OSUnlockVARRAY@4+15 (f01c1d9f,3,0,628b2210)

@[ 5] 0x628b2319 namgrdll._SecondCheckDbForTasks@4+265 (0,0,0,0)

@[ 6] 0x60115334 nnotes._ThreadWrapper@4+212 (0,0,0,0)

[ 7] 0x7c57b396 KERNEL32.CreateThread+530 (60115260,0,0,dddd04d2)

############################################################

PASS 2 : FATAL THREAD with STACK FRAMES 4/6 [ nAMgr:08f4:0e00]

FP=0c16fdd0, PC=60191907, SP=0c16f258, stksize=2936

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

---------- Top of the Stack ----------

   # 0c16f258  00000001 136a849f 00000005 54200a0a  |......j....... T|

   # 0c16f268  61657268 305b3d64 3a344638 35303030  |hread=[08F4:0005|

   # 0c16f278  3045302d 530a5d30 6b636174 73616220  |-0E00].Stack bas|

   # 0c16f288  78303d65 37314330 34393030 7453202c  |e=0x0C170094, St|

   # 0c16f298  206b6361 657a6973 33203d20 20363536  |ack size = 3656 |

   # 0c16f2a8  65747962 41500a73 3a43494e 6f6f4c20  |bytes.PANIC: Loo|

   # 0c16f2b8  4870756b 6c646e61 68203a65 6c646e61  |kupHandle: handl|

   # 0c16f2c8  6f6e2065 6c612074 61636f6c 0a646574  |e not allocated.|

   # 0c16f2d8  00000000 0c16f460 1384336d ffffffff  |....`...m3......|

   # 0c16f2e8  0c16f2fc 623e0727 0c16f338 00000000  |....'.>b8.......|

   # 0c16f2f8  1384336b 13843372 ffffffff 0c16f318  |k3..r3..........|

   # 0c16f308  623e0727 0c16f6ec 00000000 0106177c  |'.>b........|...|

   # 0c16f318  0c16f32c 60009774 00000000 00000002  |,...t..`........|

   # 0c16f328  00000001 4c5c3a66 7375746f 6d6f445c  |....f:\Lotus\Dom|

   # 0c16f338  5c6f6e69 61746144 6462645c 616d7269  |ino\Data\dbdirma|

   # 0c16f348  736e2e6e 13840066 13843362 13843363  |n.nsf...b3..c3..|

   # 0c16f358  13843368 13843369 1384336a 1384336b  |h3..i3..j3..k3..|

   # 0c16f368  13843373 7c586be5 0000063c 0c16f3b4  |s3...kX|<.......|

   # 0c16f378  0c16f384 0c16f38c 00000001 ffff0001  |................|

   # 0c16f388  ffffffff 00000001 00000000 0c16f3c8  |................|

   # 0c16f398  7c586b4f 0000063c 00000000 00000001  |OkX|<...........|

   # 0c16f3a8  00000000 0c16f3b4 011d2724 00000000  |........$'......|

   # 0c16f3b8  00000000 6000110d 6000114a 04ee3e51  |.......`J..`Q>..|

   # 0c16f3c8  04ee3e51 04ee3e2c 0c16f3e0 600010be  |Q>..,>.........`|

   # 0c16f3d8  04ee3e51 00000001 60001216 6000127b  |Q>.........`{..`|

   # 0c16f3e8  04ee3e51 04ee3e2c 0c16f3fc 60193ab3  |Q>..,>.......:.`|

   # 0c16f3f8  6000110d 6000114a 01dcf576 01dcf576  |...`J..`v...v...|

   # 0c16f408  01dcf551 0c16f41c 600010be 01dcf576  |Q..........`v...|

   # 0c16f418  00000001 60001216 6000127b 01dcf576  |.......`{..`v...|

   # 0c16f428  01dcf551 0c16f438 60001461 60008847  |Q...8...a..`G..`|

   # 0c16f438  60008888 01179bf8 048a1f04 0145375e  |...`........^7E.|

   # 0c16f448  0c16f898 607568ed 01179bf6 014530e8  |.....hu`.....0E.|

   # 0c16f458  138430e8 0c16fb54 6c5c3a66 7375746f  |.0..T...f:\lotus|

   # 0c16f468  6d6f645c 5c6f6e69 61746164 6d6f645c  |\domino\data\dom|

   # 0c16f478  2e676f6c 0066736e 000000a7 01170001  |log.nsf.........|

   # 0c16f488  49414d49 58452f4c 41424d49 60004b4e  |IMAIL/EXIMBANK.`|

   # 0c16f498  0c16f4b4 60009626 031dd5e8 00083005  |....&..`.....0..|

   # 0c16f4a8  000000a7 00000000 0c16f6d4 6003cd19  |...............`|

   # 0c16f4b8  00000000 00000000 000000a7 014530e8  |.............0E.|

   # 0c16f4c8  00000000 00000000 60abcea8 00000000  |...........`....|

   # 0c16f4d8  00000000 000000a7 01170001 00000000  |................|

   # 0c16f4e8  0049ec00 0c16f634 6007ca37 00000400  |..I.4...7..`....|

   # 0c16f4f8  0000000a 0000033c 00000000 1389ece8  |....<...........|

   # 0c16f508  014530e8 00000008 00083005 0049ec05  |.0E......0....I.|

   # 0c16f518  0c16f580 600045d2 014532d1 000008f4  |.....E.`.2E.....|

   # 0c16f528  014532d1 00000e00 0c16f54c 60003789  |.2E.....L....7.`|

   # 0c16f538  014532d1 000008f4 00000e00 0c16f55c  |.2E.........\...|

   # 0c16f548  014532d1 0c16f56c 13843373 00000000  |.2E.l...s3......|

   # 0c16f558  00000000 00080000 1384336b 00000000  |........k3......|

   # 0c16f568  00000000 00000000 00000000 00000000  |................|

   # 0c16f578  00000000 33630005 00001384 00000000  |......c3........|

   # 0c16f588  00000000 00000000 00000000 00000000  |................|

   # 0c16f598  00000000 00000000 00000000 0000008f  |................|

   # 0c16f5a8  0000008f 0000008e 00000000 00000000  |................|

   # 0c16f5b8  00000000 00000000 00000000 0000008f  |................|

   # 0c16f5c8  00000002 0c16f808 00000000 000a0000  |................|

   # 0c16f5d8  13843369 00000000 00000000 00000000  |i3..............|

   # 0c16f5e8  00000000 00000000 00000000 00000000  |................|

   # 0c16f5f8  00000000 00000000 0c16f618 60009697  |...............`|

   # 0c16f608  031dd5e8 0000008f 00000008 00000000  |................|

   # 0c16f618  0c16f624 600096cc 0c16f63c 60009626  |$......`<...&..`|

   # 0c16f628  031dd5e8 00083005 0000008f 00000000  |.....0..........|

   # 0c16f638  0c16f85c 6003cd19 00000000 00000000  |\......`........|

   # 0c16f648  0000008f 014530e8 00000008 00000000  |.....0E.........|

   # 0c16f658  60abcea8 00000000 014530e8 00000001  |...`.....0E.....|

   # 0c16f668  0c16fb54 00000000 00000000 00000000  |T...............|

   # 0c16f678  00000000 00000000 00000000 00000000  |................|

   # 0c16f688  00000000 00000000 00000000 00000000  |................|

   # 0c16f698  00000000 00000000 00000000 00000000  |................|

   # 0c16f6a8  00000000 13843360 13843360 00000001  |....`3..`3......|

   # 0c16f6b8  00000001 0c16f8b4 0c160006 00000008  |................|

   # 0c16f6c8  0c16f6ec 605f7d1b 1384336b 0c16f8ba  |.....}_`k3......|

   # 0c16f6d8  00000008 000000ff 0c16f8b4 ffffffff  |................|

   # 0c16f6e8  00000000 0c16f888 60034ed4 00000000  |.........N.`....|

   # 0c16f6f8  00000000 00000000 0000002f 1384336b  |......../...k3..|

   # 0c16f708  336b0008 0c16f8b4 000000ff 0c16f884  |..k3............|

   # 0c16f718  13843360 014530e8 0c16fb54 00000000  |`3...0E.T.......|

   # 0c16f728  00000000 00080000 1384336b 00000000  |........k3......|

   # 0c16f738  00000000 00000000 00000000 00000000  |................|

   # 0c16f748  00000000 33630005 00001384 00000000  |......c3........|

   # 0c16f758  00000000 00000000 6c5c3a66 7375746f  |........f:\lotus|

   # 0c16f768  6d6f645c 5c6f6e69 61746164 6d6f645c  |\domino\data\dom|

   # 0c16f778  2e676f6c 0066736e 00000000 00000000  |log.nsf.........|

   # 0c16f788  00000000 00000000 00000000 00000000  |................|

   # 0c16f798  00000000 00000000 6000110d 6000114a  |...........`J..`|

   # 0c16f7a8  6000110d 6000114a 04e5c0b7 04e5c0b7  |...`J..`........|

   # 0c16f7b8  04e5c092 0c16f7cc 600010be 04e5c0b7  |...........`....|

   # 0c16f7c8  00000001 60001216 6000127b 04e5c0b7  |.......`{..`....|

   # 0c16f7d8  00000000 01138938 00008aa5 6000114a  |....8.......J..`|

   # 0c16f7e8  6000110d 6000114a 04ee3e51 031dd5e8  |...`J..`Q>......|

   # 0c16f7f8  00443001 0000008e 0000008f 0000008e  |.0D.............|

   # 0c16f808  0000008f 0000008e 0000008e 0000008e  |................|

   # 0c16f818  00000000 6000114a 60001441 01178954  |....J..`A..`T...|

   # 0c16f828  60008808 01178952 00000000 0c16fb54  |...`R.......T...|

   # 0c16f838  0000008e 0117b622 0117b622 0c16f854  |...."..."...T...|

   # 0c16f848  600014e3 0117b622 0117b620 000008f4  |...`"... .......|

   # 0c16f858  60001461 60008847 60008888 493d4e43  |a..`G..`...`CN=I|

   # 0c16f868  4c49414d 453d4f2f 424d4958 004b4e41  |MAIL/O=EXIMBANK.|

   # 0c16f878  0117893a 00000000 0c16fb54 0c16fb54  |:.......T...T...|

   # 0c16f888  0c16f89c 6003b4bc 0000008f 014530e8  |.......`.....0E.|

   # 0c16f898  138430e8 0c16fb30 6003cae3 0c16fb54  |.0..0......`T...|

   # 0c16f8a8  00000000 0000008f 01170000 49414d49  |............IMAI|

   # 0c16f8b8  58452f4c 41424d49 60004b4e 0c16f8e0  |L/EXIMBANK.`....|

   # 0c16f8c8  60009626 031dd5e8 00083005 0000008f  |&..`.....0......|

   # 0c16f8d8  00000000 0c16fb00 6003cd19 00000000  |...........`....|

   # 0c16f8e8  00000000 0000008f 014530e8 00000000  |.........0E.....|

   # 0c16f8f8  00000000 60abcea8 00000000 0c16ff35  |.......`....5...|

   # 0c16f908  0000008f 01170000 00000000 0000002e  |................|

   # 0c16f918  00002f00 003a0000 59000000 65747365  |./....:....Yeste|

   # 0c16f928  79616472 00000000 00000000 00000000  |rday............|

   # 0c16f938  00000000 00000000 54000000 7961646f  |...........Today|

   # 0c16f948  00000000 00000000 00000000 00000000  |................|

   # 0c16f958  00000000 00000000 54000000 6b6f6f4c  |...........TLook|

   # 0c16f968  61487075 656c646e 6168203a 656c646e  |upHandle: handle|

   # 0c16f978  746f6e20 6c6c6120 7461636f 00006465  | not allocated..|

   # 0c16f988  8525726b 0c160017 0a880001 02020004  |kr%.............|

   # 0c16f998  0c16fbac 601d2216 05d40001 0c16f9ac  |.....".`........|

   # 0c16f9a8  0c16fe34 72616553 6e696863 4c5c3a66  |4...Searchinf:\L|

   # 0c16f9b8  7375746f 6d6f445c 5c6f6e69 61746144  |otus\Domino\Data|

   # 0c16f9c8  6d6f645c 2e676f6c 0066736e 00007374  |\domlog.nsf.ts..|

   # 0c16f9d8  00000000 00000000 00000000 00000000  |................|

   # 0c16f9e8  00083005 00083005 60003232 01003130  |.0...0..22.`01..|

   # 0c16f9f8  00000000 00000000 014530e8 00000000  |.........0E.....|

   # 0c16fa08  00000000 00000000 00000000 0000008f  |................|

   # 0c16fa18  0c16faac 00000000 00000000 00000000  |................|

   # 0c16fa28  00000000 00000000 00000000 00000000  |................|

   # 0c16fa38  00000000 00000000 00000000 00000000  |................|

   # 0c16fa48  00000000 00000000 00000000 00000000  |................|

@[ 1] 0x60191907 nnotes._Panic@4+631 (609f0016,5,bfbb550,c16fe04)

   # 0c16fdd0  0c16fde4 600028bc 609f0016 00000005  |.....(.`...`....|

   # 0c16fde0  0bfbb550 0c16fe04 600026c7 f01c1d9f  |P........&.`....|

@[ 2] 0x600028bc nnotes._LockMemHandle@12+348 (f01c1d9f,c16fe00,c16fe0c,5)

   # 0c16fde4  0c16fe04 600026c7 f01c1d9f 0c16fe00  |.....&.`........|

   # 0c16fdf4  0c16fe0c 00000005 0bfbb550 00c10014  |........P.......|

@[ 3] 0x600026c7 nnotes._OSMemoryLock@4+23 (60c40a40,5,bfbb550,101)

   # 0c16fe04  0c16fe1c 6000573f 60c40a40 00000005  |....?W.`@..`....|

   # 0c16fe14  0bfbb550 00000101 0c16ff94 628b2319  |P............#.b|

@[ 4] 0x6000573f nnotes._OSUnlockVARRAY@4+15 (f01c1d9f,3,0,628b2210)

   # 0c16fe1c  0c16ff94 628b2319 f01c1d9f 00000003  |.....#.b........|

   # 0c16fe2c  00000000 628b2210 6c6d6f64 6e2e676f  |.....".bdomlog.n|

   # 0c16fe3c  00006673 0066736e 00000066 00000000  |sf..nsf.f.......|

   # 0c16fe4c  00000000 00000000 00000000 00000000  |................|

   # 0c16fe5c  00000000 00000000 00000000 00000000  |................|

   # 0c16fe6c  00000000 00000000 00000000 00000000  |................|

   # 0c16fe7c  00000000 00000000 00000000 00000000  |................|

   # 0c16fe8c  00000000 00000000 00000000 00000000  |................|

   # 0c16fe9c  00000000 00000000 00000000 00000000  |................|

   # 0c16feac  00000000 00000000 00000000 00000000  |................|

   # 0c16febc  00000000 00000000 00000000 00000000  |................|

   # 0c16fecc  00000000 00000000 00000000 00000000  |................|

   # 0c16fedc  00000000 00000000 00000000 00000000  |................|

   # 0c16feec  00000000 00000000 00000000 00000000  |................|

   # 0c16fefc  00000000 00000000 00000000 00000000  |................|

   # 0c16ff0c  00000000 00000000 00000000 00000000  |................|

   # 0c16ff1c  00000000 00000000 00000000 00000000  |................|

   # 0c16ff2c  00000000 00000000 00008f00 00000000  |................|

   # 0c16ff3c  8b14f000 8b258062 00000162 00000300  |....b.%.b.......|

   # 0c16ff4c  00000300 00000100 00000000 00000000  |................|

   # 0c16ff5c  60001100 011424a0 0bfbb14c 00000234  |...`.$..L...4...|

   # 0c16ff6c  00000001 600922b1 00000234 60001461  |.....".`4...a..`|

   # 0c16ff7c  01142494 600bf119 01142492 00000003  |.$.....`.$......|

   # 0c16ff8c  01215e28 00000000 0c16ffb4 60115334  |(^!.........4S.`|

@[ 5] 0x628b2319 namgrdll._SecondCheckDbForTasks@4+265 (0,0,0,0)

   # 0c16ff94  0c16ffb4 60115334 00000000 00000000  |....4S.`........|

   # 0c16ffa4  00000000 00000000 00000000 00000000  |................|

@[ 6] 0x60115334 nnotes._ThreadWrapper@4+212 (0,0,0,0)

   # 0c16ffb4  0c16ffec 7c57b396 00000000 00000000  |......W|........|

   # 0c16ffc4  00000000 00000000 7ffdb000 c0000005  |................|

   # 0c16ffd4  0c16ffc0 0c16eea4 ffffffff 7c5c216c  |............l!\||

   # 0c16ffe4  7c572b08 00000000 00000000 00000000  |.+W|............|

[ 7] 0x7c57b396 KERNEL32.CreateThread+530 (60115260,0,0,dddd04d2)

############################################################

thread 5/6: [ nAMgr:08f4:0d94]

FP=1926ff20, PC=77f88f13, SP=1926fefc, stkbase=19170000, stksize=262144

############################################################

[ 1] 0x77f88f13 ntdll.RtlValidAcl+344 (56c,1f4,0,600a7e07)

[ 2] 0x7c57b3e9 KERNEL32.CreateThread+613 (83,0,60c34aac,0)

@[ 3] 0x600d5f9d nnotes._OSWaitEvent@8+29 (121a0cc,1f4,0,60115334)

@[ 4] 0x601a2638 nnotes._AutoFDGCleanupThreadProc@4+40 (0,0,0,0)

[ 5] 0x7c57b396 KERNEL32.CreateThread+530 (60115260,0,0,dddd04d2)

############################################################

thread 6/6: [ nAMgr:08f4:0e18]

FP=237bffb4, PC=77f9193d, SP=237bffa8, stkbase=23780000, stksize=8192

############################################################

[ 1] 0x77f9193d ntdll.RtlDeNormalizeProcessParams+341 (0,0,0,0)

[ 2] 0x7c57b396 KERNEL32.CreateThread+530 (0,0,0,0)

hcl-bot · January 24, 2007, 12:15pm

Subject: RE: Server crashes every day

Hi,

Do you have a fixup running when the crash occurs?

6.5.4, do you have FP1?

Error: ‘AMGR - PANIC: LookupHandle: handle out of range in SecondCheckDbForTasks’

http://www-1.ibm.com/support/docview.wss?&uid=swg21203204

JYR

hcl-bot · January 29, 2007, 11:55am

Subject: RE: Server crashes every day

I have updated the server to 6.5.4 FP2 on last tuesday. Early this morning the same server got hang. I have restarted the server and again got the problem with out going mails. I have killed the server withj nsd -kill and restarted. then the problem turned into not able to send mails to external domains. I restarted again with nsd -kill. Then mail started flowing but the INotes part stopped working. So, I called the IBM again and they told me to restart the HTTP task again ( I tried that earler) and then it started working now. trying to find THE reason for this server’s frequent crashes and hangups. PLEASE HELP ME!!

hcl-bot · January 29, 2007, 12:36pm

Subject: RE: Server crashes every day

Hi,

Does IBM told you to add Notes.ini settings to your server?

Troubleshooting router and SMTP crashes

From the link

With the server down, add the following debug parameters in the server’s Notes.ini:

smtpdebug=3

Restart the Domino server and monitor incoming SMTP traffic.
If applicable, have the firewall/gateway/relay server transfer the messages it has queued back to the Domino server.
If the server crashes on the SMTP task, the crash may be the result of an incoming message from the gateway server. Check the Domino log and the gateway server queue to identify the problematic message.
If the server does not crash, the crash may have been due to insufficient resources. A reboot of the server (thus dropping all sessions on the server and freeing up resources) may have resolved the issue.
If the server continues to crash, review and compare the NSDs and console logs to try to identify the source of the incoming message. Block or reject messages from the source, using Configuration document restrictions.
If a source cannot be identified, or the server continues to crash, add the following additional debug parameter:

smtpsaveimporterrors=n

This parameter will capture incoming SMTP messages in a STXXXXXX.TMP file into the OS temp directory. This may cause the TEMP directory to grow large and should be used only for a short time. This parameter can be set to either 1 or 2:

1 - saves only messages that fail to be imported by the SMTP task
2 - saves all messages received by the SMTP task.

Note: If server mail rules are being used, they should also be disabled to further troubleshoot. Also, third-party applications should be investigated as they may conflict with the SMTP task.

JYR

hcl-bot · February 12, 2007, 8:57am

Subject: RE: Server crashes every day

THANK YOU ALLOn saturday we have migrated to a new box from win 2000 to win 2003. unable to route the outbound mail on that server. First I found that port 25 is blocked by the McAfee. Later found that It was a problem with the NIC or Network cable isssue between our mail router and firewall. They did a some kind of reset on the router and now… IT WORKS GREAT. Now its a happy ending.