Domino Server constantly crashing - help!

hcl-bot · August 28, 2008, 9:09pm

Hi

I have a Lotus Domino server here, version 6.5.5, and it’s constantly and randomly (or seemingly random to me) crashing.

By crashing i mean it just exit’s, i log into the server and all the event log shows is “domino server service entered stopped state”.

I start it again and it works fine, until the next crash!

I need an expert’s hand to decipher the nsd file, because it’s all double dutch to me!

By chance I was logged on to the console one time and saw a crash message flick by saying “PANIC: Cannot attach to shared memory region, due to insufficient access (probably owned by another user or group)”

I did a few searches on that error and didn’t find anything that helped. I changed some notes.ini settings such as:

ConstrainedSHM=1

ConstrainedSHMSizeMB=“512”

NSF_DOCCACHE_THREAD=1

Which didn’t help.

I also ran a fixup -f and a compact -c, deleted the mail.box, pid.nbf, and let notes re-create them.

If anyone can help i’d really appreciate it!

Can someone tell me what part of the nsd file will help? It’s 5mb and i can’t paste the whole thing in here!

EDIT

i did a search for “fatal” and “error” in the nsd and these came up (if it helps):

<@@ ------ Notes Memory Analyzer (memcheck) → MM/OS Structure Information (Time 07:38:08) ------ @@>

Start Time = 28/08/2008 10:58:31 PM

Crash Time = 29/08/2008 07:37:36 AM

Error Message = PANIC: Cannot attach to shared memory region, due to insufficient access (probably owned by another user or group)

SharedDPoolSize = 4194304

FaultRecovery = 0x00010010

Cleanup Script Timeout= 300

Crash Limits = 3 crashes in 5 minutes

StaticHang = [nsmdreal: 6912:   12]/[nsmdreal: 6912: 6504] (0x1b00/0xc/0x1968)

ConfigFileSem =  (  SEM:#0:0x010d) n=0, wcnt=-1, Users=-1,  Owner=[        :    0]

FDSem         =  ( RWSEM:#11:0x410f) rdcnt=-1, refcnt=0 Writer=[        :    0], n=11, wcnt=-1, Users=0,  Owner=[        :    0]

############################################################

FATAL THREAD 7/14 [nsmdreal: 6912: 6504]

FP=0x0def5fb8, PC=0x60193297, SP=0x0def5440, stksize=2936

EAX=0x010f088c, EBX=0x00000000, ECX=0x008e0000, EDX=0x008e0000

ESI=0x0def5b4c, EDI=0x00000000, CS=0x0000001b, SS=0x00000023

DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010202

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

@[ 1] 0x60193297 nnotes._Panic@4+631 (60a30016)

@[ 2] 0x60192fcc nnotes._Halt@4+28 (def01b1)

@[ 3] 0x6011587d nnotes._AccessAllProtected@0+77 ()

@[ 4] 0x6004fedf nnotes._AccessAll@8+47 (1,1)

@[ 5] 0x60050dd5 nnotes._ProcessGlobalEvent@4+21 (d22cfc)

@[ 6] 0x60050ce1 nnotes._OSProcessShouldQuit@0+49 ()

############################################################

PASS 2 : FATAL THREAD with STACK FRAMES 7/14 [nsmdreal: 6912: 6504]

FP=0def5fb8, PC=60193297, SP=0def5440, stksize=2936

Exception code: c0000005 (ACCESS_VIOLATION)

############################################################

---------- Top of the Stack ----------

   # 0def5440  00000001 0def01b1 00000000 54200a0a  |.............. T|

   # 0def5450  61657268 315b3d64 3a303042 43303030  |hread=[1B00:000C|

   # 0def5460  3639312d 530a5d38 6b636174 73616220  |-1968].Stack bas|

   # 0def5470  78303d65 46454430 43313236 7453202c  |e=0x0DEF621C, St|

   # 0def5480  206b6361 657a6973 33203d20 20303635  |ack size = 3560 |

   # 0def5490  65747962 41500a73 3a43494e 6e614320  |bytes.PANIC: Can|

   # 0def54a0  20746f6e 61747461 74206863 6873206f  |not attach to sh|

   # 0def54b0  64657261 6d656d20 2079726f 69676572  |ared memory regi|

   # 0def54c0  202c6e6f 20657564 69206f74 6675736e  |on, due to insuf|

   # 0def54d0  69636966 20746e65 65636361 28207373  |ficient access (|

   # 0def54e0  626f7270 796c6261 6e776f20 62206465  |probably owned b|

   # 0def54f0  6e612079 6568746f 73752072 6f207265  |y another user o|

   # 0def5500  72672072 2970756f 0000000a 77e6b487  |r group).......w|

   # 0def5510  00000000 00000000 00000000 00000000  |................|

   # 0def5520  0def5548 0000002b ffffffff 00000008  |HU..+...........|

   # 0def5530  0def55ac 00000000 0def5570 03f0e5ce  |.U......pU......|

   # 0def5540  0def5550 03f3f151 00000000 00000000  |PU..Q...........|

   # 0def5550  00000000 00002a40 00000000 00000000  |....@*..........|

   # 0def5560  00000000 00000000 00000000 00000000  |................|

   # 0def5570  00000000 00000000 00000000 00000000  |................|

   # 0def5580  00000000 00000000 00000000 00000000  |................|

   # 0def5590  00000000 00000000 00000000 00000000  |................|

   # 0def55a0  00000000 00000000 00000000 00000000  |................|

   # 0def55b0  00000000 00000000 00000000 00000000  |................|

   # 0def55c0  00000006 08ef2b79 ffffffff 0def55e0  |....y+.......U..|

   # 0def55d0  623e0787 0def561c 00000000 08ef2b72  |..>b.V......r+..|

   # 0def55e0  08ef2b7e ffffffff 0def55fc 623e0787  |~+.......U....>b|

   # 0def55f0  0def59d0 00000000 00fac018 0def5610  |.Y...........V..|

   # 0def5600  60009a34 00000000 00000002 00000001  |4..`............|

   # 0def5610  0def59e0 60018910 3e1bb5b8 00000b71  |.Y.....`...>q...|

   # 0def5620  3bf664b8 00000b71 8b79feca 2e9300c8  |.d.;q.....y.....|

   # 0def5630  0def5658 7c83805e 3bf664b8 00000bb3  |XV..^..|.d.;....|

   # 0def5640  fa67b90e c6d750eb 0000001a 00001968  |..g..P......h...|

   # 0def5650  0012efa0 03f16f84 0012efa0 00001968  |.....o......h...|

   # 0def5660  00000000 0000003c 0012efa0 00000007  |....<...........|

   # 0def5670  00000000 00000007 0000006c 77e41f4f  |........l...O..w|

   # 0def5680  000000f1 00000388 0def5778 03f0e820  |........xW.. ...|

   # 0def5690  48b71a9e 00000008 0000001d 00000007  |...H............|

   # 0def56a0  00000025 00000022 00000000 00000000  |%..."...........|

   # 0def56b0  00000000 00000000 00000000 00000000  |................|

   # 0def56c0  00000000 00000000 00000000 00000000  |................|

   # 0def56d0  0000000a 77e6b487 03f32d71 0400e1dc  |.......wq-......|

   # 0def56e0  00000002 0def5708 03f10133 00000000  |.....W..3.......|

   # 0def56f0  00000002 0def5740 0def5734 00000000  |....@W..4W......|

   # 0def5700  00000000 77e6b487 00000000 00000000  |.......w........|

   # 0def5710  00000000 00000000 0def5740 0000002b  |........@W..+...|

   # 0def5720  ffffffff 00000008 0def57a4 00000000  |.........W......|

   # 0def5730  0def5768 03f0e5ce 0def5748 03f3f151  |hW......HW..Q...|

   # 0def5740  0def5784 00001968 0def5847 7fffffd3  |.W..h...GX......|

   # 0def5750  0def581c 00000042 0def57a4 00000014  |.X..B....W......|

   # 0def5760  03f32d6c 0e6e85e8 0012ef70 03efe98e  |l-....n.p.......|

   # 0def5770  0def581c 03f3f144 040042e4 040041e4  |.X..D....B...A..|

   # 0def5780  03efe9a9 0012efa0 00001968 48b71a9e  |........h......H|

   # 0def5790  38303032 2f38302f 30203932 37333a37  |2008/08/29 07:37|

   # 0def57a0  0034333a 38303032 39323830 01131900  |:34.20080829....|

   # 0def57b0  00000000 0def58d8 00000000 00000000  |.....X..........|

   # 0def57c0  00000000 00000007 00000000 01130a7c  |............|...|

   # 0def57d0  0def5a54 00000000 00000000 00000000  |TZ..............|

   # 0def57e0  2f0000a3 00000000 00000105 00000000  |.../............|

   # 0def57f0  00000000 00000000 00000d45 00000000  |........E.......|

   # 0def5800  00000000 00000000 00000000 00000001  |................|

   # 0def5810  00000000 00000000 000000a4 6e5c3a65  |............e:\n|

   # 0def5820  7365746f 7461645c 6d732f61 6d657464  |otes\data/smdtem|

   # 0def5830  536e2f70 6572444d 325f6c61 30383030  |p/nSMDreal_20080|

   # 0def5840  2e393238 00676264 00000002 00000000  |829.dbg.........|

   # 0def5850  08ef2b6d 00000000 00000000 00000000  |m+..............|

   # 0def5860  2b630006 000008ef 00000000 00000000  |..c+............|

   # 0def5870  00000000 00000000 00000000 00000000  |................|

   # 0def5880  00000000 00000000 00000000 00000000  |................|

   # 0def5890  00000000 00000000 00000000 00000000  |................|

   # 0def58a0  00000000 00000000 00000000 00000000  |................|

   # 0def58b0  00000000 00000000 00150000 08ef2b6a  |............j+..|

   # 0def58c0  00000000 00000000 00000000 00000000  |................|

   # 0def58d0  00000000 00000000 00000000 00000000  |................|

   # 0def58e0  00000000 00000000 00000000 00000000  |................|

   # 0def58f0  00000000 00000000 00000000 00000000  |................|

   # 0def5900  00000000 00000000 00000000 00000000  |................|

   # 0def5910  00000000 00000000 00000000 00000000  |................|

   # 0def5920  00000000 00000000 00000000 00000000  |................|

   # 0def5930  00000000 00000000 00000000 00000000  |................|

   # 0def5940  00000000 00000000 00000000 00000000  |................|

   # 0def5950  00000000 00000000 00000000 00000000  |................|

   # 0def5960  00000000 00000000 00000000 00000000  |................|

   # 0def5970  00000000 00000000 00000000 00000000  |................|

   # 0def5980  000000d9 0def5ec4 00000000 74b30073  |.....^......s..t|

   # 0def5990  08ef2b60 08ef2b60 00000001 00000001  |`+..`+..........|

   # 0def59a0  0def5b98 0def000a 0000000d 0def59d0  |.[...........Y..|

   # 0def59b0  6062636b 08ef2b72 0def5ba2 0000000d  |kcb`r+...[......|

   # 0def59c0  000000ff 0def5b98 ffffffff 00000000  |.....[..........|

   # 0def59d0  0def5b6c 60035534 00000000 00000000  |l[..4U.`........|

   # 0def59e0  00000000 0000002f 08ef2b72 2b72000d  |..../...r+....r+|

   # 0def59f0  0def5b98 000000ff 0def5b68 08ef2b60  |.[......h[..`+..|

   # 0def5a00  00000000 0def5e38 00000000 00000000  |....8^..........|

   # 0def5a10  000d0000 08ef2b72 00000002 0def5e04  |....r+.......^..|

   # 0def5a20  60011b0c 0def6020 0def5fcb 60ad805c  |...` `..._..\..`|

   # 0def5a30  000000fc 000000fc 00000000 00000000  |................|

   # 0def5a40  00000000 00000000 00000000 533d4e43  |............CN=S|

   # 0def5a50  322d4459 554f2f30 2f50413d 76413d4f  |YD-20/OU=AP/O=Av|

   # 0def5a60  44797265 696e6e65 006e6f73 00000000  |eryDennison.....|

   # 0def5a70  00000000 00000000 00000000 00000000  |................|

   # 0def5a80  00000000 00000000 00150000 08ef2b6a  |............j+..|

   # 0def5a90  00000000 00000000 00000000 00000000  |................|

   # 0def5aa0  7c8220c4 77e725bd 00000820 0def5aec  |. .|.%.w ....Z..|

   # 0def5ab0  0def5abc 0def5ac4 00000000 ffff0002  |.Z...Z..........|

   # 0def5ac0  ffffffff 00000001 00000000 0def5b00  |.............[..|

   # 0def5ad0  77e72555 00000820 00000000 00000001  |U%.w ...........|

   # 0def5ae0  00000000 0def5aec 00de061c 00000000  |.....Z..........|

   # 0def5af0  00000000 6000110d 6000114a 013b58dc  |.......`J..`.X;.|

   # 0def5b00  013b58dc 013b58b7 0def5b18 600010be  |.X;..X;..[.....`|

   # 0def5b10  013b58dc 00000001 60001216 6000127b  |.X;........`{..`|

   # 0def5b20  013b58dc 013b58b7 0def5b34 60001833  |.X;..X;.4[..3..`|

   # 0def5b30  013b58dc 0def5b44 60002b87 013b58b7  |.X;.D[...+.`.X;.|

   # 0def5b40  00de0000 0def5b6c 600105df 6e6e6143  |....l[.....`Cann|

   # 0def5b50  6120746f 63617474 6f742068 61687320  |ot attach to sha|

   # 0def5b60  20646572 6f6d656d 72207972 6f696765  |red memory regio|

   # 0def5b70  64202c6e 74206575 6e69206f 66667573  |n, due to insuff|

   # 0def5b80  65696369 6120746e 73656363 70282073  |icient access (p|

   # 0def5b90  61626f72 20796c62 656e776f 79622064  |robably owned by|

   # 0def5ba0  6f6e6120 72656874 65737520 726f2072  | another user or|

   # 0def5bb0  6f726720 00297075 00000000 00000000  | group).........|

   # 0def5bc0  00000000 00000000 00000000 7c8210b4  |...............||

   # 0def5bd0  7c81e708 ffffffff 0012ef84 00000688  |...|............|

   # 0def5be0  00001008 00000018 00875f40 0012f184  |........@_......|

   # 0def5bf0  7c82fda6 7c82fb23 0000236c 60c0b530  |...|#..|l#..0..`|

   # 0def5c00  00875bc8 0012efa8 7c81e86a 00000688  |.[......j..|....|

   # 0def5c10  10873bc0 00000000 0000046f 00870178  |.;......o...x...|

   # 0def5c20  00870000 0012ef01 0012f030 00000688  |........0.......|

   # 0def5c30  00002378 00000219 00873bc8 0def5dc4  |x#.......;...]..|

@[ 1] 0x60193297 nnotes._Panic@4+631 (60a30016)

   # 0def5fb8  0def5fc8 60192fcc 60a30016 00d22cfc  |._.../.`...`.,..|

@[ 2] 0x60192fcc nnotes._Halt@4+28 (def01b1)

   # 0def5fc8  0def5fd8 6011587d 0def01b1 00000000  |._..}X.`........|

@[ 3] 0x6011587d nnotes._AccessAllProtected@0+77 ()

   # 0def5fd8  0def5fe0 6004fedf 0def6100 60050dd5  |._.....`.a.....`|

@[ 4] 0x6004fedf nnotes._AccessAll@8+47 (1,1)

   # 0def5fe0  0def6100 60050dd5 00000001 00000001  |.a.....`........|

   # 0def5ff0  00000000 00d22cfc 00000000 00000000  |.....,..........|

   # 0def6000  00000000 77e6bafd 00000000 04a52340  |.......w....@#..|

   # 0def6010  00000000 00000024 00000001 00000000  |....$...........|

   # 0def6020  322f3830 30322f39 30203830 37333a37  |08/29/2008 07:37|

   # 0def6030  2035333a 3031455a 30314500 7ffd8000  |:35 ZE10.E10....|

   # 0def6040  7ffd8000 00000000 00000000 0def6008  |.............`..|

   # 0def6050  0def60ac 0def60b0 7c822124 7c822124  |.`...`..$!.|$!.||

   # 0def6060  77e6bad8 7c822124 77e6bad8 00000710  |...w$!.|...w....|

   # 0def6070  8b02013c fffffff6 00004d41 004d5000  |<.......AM...PM.|

   # 0def6080  00240000 2c000000 00000000 0000002e  |..$....,........|

   # 0def6090  00002f00 003a0000 59000000 65747365  |./....:....Yeste|

   # 0def60a0  79616472 00000000 00000000 00000000  |rday............|

   # 0def60b0  00000000 00000000 54000000 7961646f  |...........Today|

   # 0def60c0  00000000 00000000 00000000 00000000  |................|

   # 0def60d0  00000000 00000000 54000000 726f6d6f  |...........Tomor|

   # 0def60e0  00776f72 00000000 00000000 00d28d1c  |row.............|

   # 0def60f0  00000000 00000000 00000000 0def6120  |............ a..|

@[ 5] 0x60050dd5 nnotes._ProcessGlobalEvent@4+21 (d22cfc)

   # 0def6100  0def6120 60050ce1 00d22cfc 00d28d1c  | a.....`.,......|

   # 0def6110  00d28d24 601d4652 0012efa0 00001968  |$...RF.`....h...|

@[ 6] 0x60050ce1 nnotes._OSProcessShouldQuit@0+49 ()

############################################################

PASS 3 : FATAL THREAD with PARAMETER DATA 7/14 [nsmdreal: 6912: 6504]

############################################################

@[ 1] 0x60193297 nnotes._Panic@4+631 (60a30016)

   Parameter 1:

   # 60a30016  4446534f 656c6946 74697257 30203a65  |OSFDFileWrite: 0|

   # 60a30026  20782578 6f727265 25203a72 25722565  |x%x error: %e%r%|

   # 60a30036  7325006e 6e257225 55530000 4f203a54  |n.%s%r%n..SUT: O|

   # 60a30046  616d5353 70557472 64617267 61725465  |SSmartUpgradeTra|

@[ 2] 0x60192fcc nnotes._Halt@4+28 (def01b1)

   Parameter 1:

   # 0def01b1  00000000 30000000 80600010 00063fc5  |.......0..`..?..|

   # 0def01c1  04000000 610def03 a4600014 c30122ca  |.......a..`.."..|

@[ 3] 0x6011587d nnotes._AccessAllProtected@0+77 ()

@[ 4] 0x6004fedf nnotes._AccessAll@8+47 (1,1)

@[ 5] 0x60050dd5 nnotes._ProcessGlobalEvent@4+21 (d22cfc)

   Parameter 1:

   # 00d22cfc  00001b00 0000157c 6e5c3a65 7365746f  |....|...e:\notes|

   # 00d22d0c  4d536e5c 61657244 58452e6c 00000045  |\nSMDreal.EXE...|

@[ 6] 0x60050ce1 nnotes._OSProcessShouldQuit@0+49 ()

hcl-bot · August 29, 2008, 3:28am

Subject: Domino Server constantly crashing - help!

A quick google for nsmdreal gave me the following link:http://www-01.ibm.com/support/docview.wss?rs=463&context=SSKTMJ&dc=DB520&uid=swg21205687&loc=en_US&cs=UTF-8&lang=en&rss=ct463lotus

Seems helpful to me…

/Peter

hcl-bot · August 29, 2008, 1:48am

Subject: Domino Server constantly crashing - help!

I found this article when looking for “accessallprotected” and “nnotes” (note that the first six lines of the fatal thread dump seem to correspond with yours):

http://www-01.ibm.com/support/docview.wss?uid=swg21244016

Having read the workaround I think I’d check what memos are being routed with large attachments. It may be worth implementing a max message size for mails to block routing of anything above, say, 20mb.

FYI I also found this (below). I expect you’re already aware of this one though as it was a bit easier to find:

http://www-01.ibm.com/support/docview.wss?uid=swg21179667

Hope the above are of some help. I’d be interested to know if you find the cause…

Phil

hcl-bot · August 29, 2008, 7:23am

Subject: RE: Domino Server constantly crashing - help!

Like Peter said, it seems to be related to ScanMail form TrendMicro. I would look on the their Websitehttp://esupport.trendmicro.com/support/search.do

or contact their support center.

Which version are you running?

JYR,