Hi,
I am running an 8.5 upgrade in a test environment for a customer with a lot of large databases. Main reason for the upgrade will be the DAOS implementation.
After running the compact -C -DAOS ON command, it generated 10 subdirs with 40000 .nlo files each. The next day I found the following message in the log file:
DAOSMGR: DbDelete started
The DAOS catalog cannot be updated: Entry not found in index
DAOSMGR: DbDelete completed
After looking at the status of DAOSMGR, it said that it needed a resync, so i started that.
In that process I received a LOT of messages like “Duplicate Notes Large Object (filepath/filename.nlo) has been deleted” on te server console. They do not get in the log.nsf, so counting them is not possible. If I look in the DAOS directories, the following number of files are still there:
0001 - 40000
0002 - 40000
0003 - 40000
0004 - 40000
0005 - 40000
0006 - 40000
0007 - 35882
0008 - 19196
0009 - 18504
0010 - 15071 (never reached the 40k limit)
So in total 46417 files got deleted in the 7/8/9 folders.
Searching for some file names that appeared on the server console in the daos folder, and it appears that they do exist in higher folders.
Big question: should I be worried that files are now missing? The problem is either while moving the attachment to DAOS, or by naming the file and not checking it properly.
Best regards,
Marco
Subject: Duplicate NLO deleted
DAOS is designed to err on the side of caution. When a database saves an object to DAOS, we consult the index in daoscat.nsf to see if we already have an existing copy of the NLO. If we do, we can add the current object request to the reference count of the existing object and we do not need to save the object again.
If DAOS detects any problem in the index (which seems to have happened here ), we will save a second copy of the NLO and start a new reference count. This is a redundant safety mechanism.
When you run resync, we rebuild the index and consolidate the reference counts for the objects and, because we detect that we had saved a second copy of the NLO as a redundant, extra safe copy, we clean up the extra object. There should be no data loss - this is the absolute highest design objective for DAOS. This is really just DAOS being cautious, because we absolutely never want service to be interrupted.
So, the real question from our point of view is what caused the index to have a problem. What specific build of Domino are you using ? Were there any noteworthy events around this time ( system crash, Domino crash ??)
Subject: Server specs
Hi Gary,
I used the 8.5 Win32 installer executable as downloaded from partnerworld (the file properties says 4.0.100.1124, 601.719kB), applied hotfix 84 to it and installed LEI 8.5. NSD says build 8.5.00.8318. The server runs on Win2003 5.2, Win32, Build 3790, SP2, dual processor. Somewhere in the process I installed Sametime 8.0.2, but is not DAOS enabled.
The server upgraded from 6.5.6 in one go, where the Domino Directory template was already on R85 for some time. From there I basically followed the steps as described in the documentation of Lotusphere09’s SHOW101, as I have done a few times by now (thanks again PaulM:-).
There were some other issues on the server after the upgrade that I did not come across earlier, but they should not be related to physical databases. The server itself is only different from regular servers in that it holds over 40 databases with 32Gb+ sizes, which generated 328653 .nlo files with a total of 80.0Gb within a day.
Please let me know if you need more info, I hope it is not the sheer size of the environment. I am not sure what sizes you come across at other locations. This is only the development server; the first production server we want to upgrade holds a lot more data.
Thanks & regards,
Marco
Subject: Update
Update on my previous post after checking some other documentation:- I did a ‘tell daosmgt listnlo missing (last compacted large db)’, and it reported 6 files missing in directory 10. I found these filenames in folder 0001.
- Did a ‘tell daosmgr resync force’ to rebuild the catalog from scratch, but gives the same result result; still 6 missing files.
Subject: please send…
your daos.cfg file. There is no company specific info in hereyou can send it to me directly, gary_rheaume@us.ibm.com
thanks.
this volume of files is not a problem