Index All stuck at 100% but not doing anything?

HCL Domino Domino Forum
1

We have 3 on premise Domino 9.0.1FP9HF139 all suffering the same issue - each has a different function (1 main mail server (approx. 100 concurrent users), 1 backup ‘hot spare’ with no active users and 1 WebMail external facing (max 4 concurrent users).

We are running Windows Servers longhorn 64.6.3 (patched on a monthly rota) and have plenty of capacity (Processors, disk space and RAM)

The majority of the mail dbs are 10-30Gb with about 25% above or below this.

We have been using the same R9 mail template with just minor ‘right click menu’ additions for over a year without issues.

We are experiencing some strange Indexing issues on all 3 servers.

The Index All task keeps getting ‘stuck’ on a mail db, the console reports Index All at 100% for this db, the index tab on the db report an index of 2-300Kb which never increases.

Updating the index from the db index tab logs that the indexer is already busy, deleting the index from the tab clears the reported size of the index - but does not stop the ‘stuck’ Index All task.

Trying the above from the Admin client also logs the indexer is busy.

Stopping or restarting the Index Task from the Admin Client does not complete and the servers need restarting.

Restarting the server via services has a long delay and results in a restart notification (mail.boxes, log and the mail db checked). The logged shutdown does not get past HTTP Server stopped.

This all seems to have started since FP9 and we apply any HF about 1 week after they are released.

We perform a clean restart after the crashed restart but the issue returns after about 3 or 4 days for a different mail db.

We OS delete the FT folder of the problem mail db while doing the clean restart and create a fresh index.

We run a compact on a Friday night on all servers (and to trigger full backups over the weekend on the ‘hot spare’ Server) followed on Sunday by fixup and updall - these have been in place for many years.

Does anyone have an idea of what is going on or have similar issues ?

Any pointers to a fix for this are very much appreciated.

Ian

2

Subject: I’d run a full fixup on the DB.

Make a backup copy from the OS. Better yet, make 2 backups. Then fixup one of them.

Database corruptions can occur in the design notes as well as the indexes and document notes. As you obliterated the index (through a purge I assume), that index isn’t corrupted; it could still be either the design note for that view, or the design index, causing Updall to infinite-loop through a design elements.

3

Subject: If it were me…

…and i suspected a fixpack or a hotfix, I’d pull it off one of the servers and check to see the results.

I can tell you that I’m at FP8HF238 on 6 servers and do not see that issue anywhere.

After I pulled the ‘fix’ and can confirm it no longer occurs I’d open a PMR with IBM…

4

Subject: Problem found - hunting for a solution

Sorry for the delay getting back on this (sick and vacation),

After much testing, fixups and NSD searches I have identified a common theme.

Our CERT Department had some CSFalcon software put onto all our servers and it seems, when it and the indexing are by chance both looking at the same mail db, the indexing gets ‘stuck’.

Now I just need to figure out how to stop it happening !

Thanks for the suggestions

Ian