How to repair defective DAOS references

HCL Domino Domino Forum
,
1

Hello,

I came into a new Domino environment, where we have a lot of error messages pointing to missing DAOS objects. After researching the scope of the problem, I found that there are about 12k .nlo files missing. The causing incident must have been a while ago (several years), so that no backup can bring those files back.

What do I with the documents with the defective DAOS references?

Those cause a lot of errors in the domino console AND prevent the documents from replicating (which has become important, since it was my task to add a new duplicating server to the domain)

Thanks for any thought,

Manuel

2

Perhaps doing a copy style compact would assist in this?

https://help.hcltechsw.com/domino/10.0.1/tune_compactoptions_r.html

Not sure what the result would be but thinking the copy style compact with ignoring errors and turning off DAOS might do the trick to remove the references?

Afterwards, you would do another copy style compact and turn DAOS back on ( so legitimate DAOS objects can be reinstated ).

3

Thank you for your thoughts.

Unfortunately

compact -c -daos off

already fails with "file not found"

adding -i (for ignoring errors) purges the notes with incorrect DAOS reference from the database.

This is not exactly what I was hoping for ;)

Thanks for the attempt though.

4

The fixup task has an option to repair these broken references. Unfortunately "repair" can also mean: Purge the documents that point to missing NLOs from the database. The good thing: If there is a replica where the document is not corrupt, it will be replicated back from the intact replica.

The additional parameter "-N" can help in case of deletion but you need to test, if then the documents are fixed at all.

The option is described in the admin help:

DAOS option for Fixup

You can run the following DAOS option (-D) for Fixup from the command line only. As with other Fixup options, you can add the -N option to prevent deletion of documents.

Because transaction logging is required in DAOS-enabled databases, you must include the -J option for the -D option.

lo fixup -J -D dbname.nsf

Purges or fixes documents in the specified database containing objects stored in the Document Attachment and Object Service (DAOS) in three cases: when a document is corrupted, when the DAOS ticket (reference) is outdated, or when the associated object (.NLO file) is missing.

The -D option also purges any references to orphaned objects in the DAOS repository, but unlike running Fixup without -D, using the option preserves the actual objects so that they can be later cleaned up by using the DAOS resynchronization command

A second option could be to create "dummy"- files for the missing attachments: Just create empty textfiles that you name as the missing files and put in the right 000x sub directory. Then the error messages go away but of course clicking on such a faked attachment will not open anything...

5

Thanks Thorsten,

I did not know about those fixup options.

-D purges the documents with the incorrect reference from the database.

Adding -N had two different results in two tests.

The first test lead to the database being marked as corrupted and therefore unusable.

**** DbMarkCorrupt(DbFixup: invalid slot found, could not be repaired), DB=C:\Program Files\HCL\Domino\data\path\filename.nsf  TID=[2F6C:0002-08A8] File=dbfixup.c Line=15459 ***

The second attempt ended with:

Database Fixup: Unable to fixup database C:\Program Files\HCL\Domino\data\path\filename.nsf: 03:F5

So it seems that there is no way to "heal" the documents with Domino tools? :(

I will try your suggestion with dummy files.

Thanks a lot,

Manuel

6

Just use the LISTNLO -Parameter to get a file with missing attachments for a specific database and then write a Powershell-Script or LotusScript- Agent to create these files from the resulting mymailobjects.txt- File:

tell daosmgr listnlo -o mymailobjects.txt MISSING mymail.nsf

7

I tested creating dummy files, but unfortunately that does not help either.

The message now reads:

The database F:\Notes\Data\path\filename.nsf was unable to open or read the file F:\Notes\DAOS\0001\0554A05164E9CA540E291A16139A205C880D299700150C0D.nlo: File truncated - file may have been damaged

The message reads differently, but the effect is the same: The database cannot be replicated.

So, is there really no way to replicate those database then to destroy/purge the documents first?

8

Hello @Manuel Nientit

Since the .nlo files are missing. reference to these objects is not valid for the respective mail documents in mail files.

If there is any cluster server available, kindly try to replicate from there. If not, then it would be difficult to recover these missing nlo files.

To avoid the error message coming for mail files on the server - kindly perform below maintenance task on mail files.

lo fixup -J -D mailfilepath

this will remove the reference of deleted attachments from mail file.

If possible, kindly turn off DAOS for mail files by running command

lo compact -C -daos off mailfilepath

post successful completion of mentioned task, kindly remove the DAOS depository and then again turn on DAOS by running command

lo compact -C -daos on mailfilepath

Regards,

Pramod

9

Thank you Pramod,

unfortunately compacting with -daos off just fails with error "file not found"

Adding -i to the command will purge the documents with incorrect reference. This is not what I hoped for.

Thanks for the attempt, though.

/Manuel

10

Hello @Manuel Nientit ,

You mentioned that the issue is preventing replication of documents, So, is it ok to understand that you have replica copies of databases on other servers as well?


Is the NLO missing issue limited to specific server or the issue is on other servers(Where replica of DBs located) also?

As per your earlier replies, you do not want to purge NLO missing documents from databases.

If you have good replica copies of DBs on other servers, then you can resolve the issue by recreating the replica copy of databases from other servers.

For this you need to determine the list of database having missing NLOs.

You can use "Tell DAOSMgr Dbsummary" command to see the status of DAOS enabled databases to understand which DB is not in sync and you can re-create the respective databases on the server where NLOs are missed.

Thanks & Regards,

Chaitanya Y

11

Hello Chaitanya,

thanks for your response.

Unfortunately, I do not have good replica on other servers. Until now, the databases only existed on this one server and in backup.

Now, I want to move those databases to another server, but this fails for above reasons. And I cannot restore the missing .nlo because they are not even in backup (any more?). We do accept that the data in the .nlo is lost for good, but I do not want to accept, that the mail content will be lost, too, when committing compact -c -daos off

Thank you for your thoughts,

Manuel Nientit