Note, I have confirmed the problem below on Windows 2003 with v 6.5.4 and 6.5.5. There is nothing to make me believe it does not apply also to any other W32 platform such as 2000 or XP (or even perhaps other platforms)
Reported to me by Lotus (PMR 07697,379,000) there is a flaw in the full text updating engine (nupdate) causing higher than usual corruptions. When Nupdate hits the FT corruption it panics, consumes resources, and eventually causes the server to crash.
Review of the NSD file will find the word “FATAL” for nupdate and in the hex dump associated with nupdate you will find the encoded filename that nupdate was processing when it panics.
There is no fix available for this at this time. The best Lotus was able to do for me was a partial workaround. It does not address WHY the indexes are becoming corrupt so often, but it does prevent the server from crashing.
This ini parameter switches on a mode where nupdate automatically deletes damaged FT indexes and recreates them. This avoids the crash but could cause other problems (lotus was not forthcoming here but I can imagine that the problems are likely less severe than a server crash).
This may cause performance hits on very active servers.
Note that while Software support swears the problem does not exist in 7.0, they implored me to include the ini parameter anyway if I chose to upgrade to 7. (The default value for the parameter is =0… needs to be =1)
Maybe this will save you a support call.
Here is the text of the workaround.
Published
Lotus Software Knowledge Base
Title:
Domino 6.x Server Crashes with the Error “…Message Queue is Full”
Product: Lotus Domino > Lotus Domino Server > Version 6.0, 6.5
Platform: Platform Independent
Date: 06/07/2005
Doc Number: 1158595
Problem
Your Domino 6.x server crashes with the error “Chronos: Error full text indexing mail\xxx.nsf: Message Queue is full” and the following fatal stack:
fatal_error() at 0x7a5a8eb8
__zerros() at 0x11d0e09c
.() at 0xb30a65e
CGtrPosWork::ReadNext(unsigned char)() at 0xb30a65e
CGtrPosShort::InsertDocs(CGtrPosSh() at 0x1306b5f8
CGtrPosBrokerNormal::Externalize(KEY_R() at 0x13019e5c
gtrMergeMerge() at 0x13016138
gtr_MergePatt() at 0x12ff267a
GTR__mergeIndex() at 0x12ffaab6
GTR_mergeIndex() at 0x12f93f2e
cGTRio::Merge()() at 0x12f66840
FTGMergeGTRIndexes(FTG_CTX*,int)() at 0x12f5d210
FTGIndex() at 0x12f4a398
FTCallIndex() at 0x7bd3ba28
FTIndexExt() at 0x7bd3e950
FTIndex() at 0x7bd3ed14
UpdateCollections() at 0x11a0f566
Update() at 0x11a0c0fe
AddInMain() at 0x11a0d0ee
NotesMain() at 0x11a18078
main() at
0x11a1827c
Content
This issue was reported to Quality Engineering as MTMY5J2JXB and has been addressed in Domino 6.0.5 and 6.5.4.
Excerpt from the Lotus Notes and Domino Release 6.5.4 and 6.0.5 MR fix list (available at IBM Developer):
Full-text indexing
SPR# MTMY5J2JX8 - Corrupt indexes Panic the server. This fix identifies corrupted data contained in any of the GTR KEY files. Without identifying the bad data, the updating process continues processing until the crash occurs. When bad data is identified, then processing stops and the index is automatically rebuilt. By default, this feature is off. To enable this feature, use the notes.ini “FTG_ENABLE_GTR_KEY_CHECK=1”.
As temporary workaround, identify the problem database and delete the index.