Get internal DAOS information about a given .nlo file

HCL Domino Domino Forum
1

I an writing a kind of logging module.I need (really need) to know know many “tickets” an attachment has. So given an object id (RRV) that holds an attachment in a DB, I can retrieve the .nlo file name (I apply SHA1 on the object content thus obtaining the main part of the .nlo file name). I need to know how many tickets this .nlo file has… I know this is something internal but I could not find this information.

There are APIs in the nnotes.dll file like:

DAOSIDFindEntry

NSFDAOSIDTableFindEntry

NSFDAOSObjectIndexScan

DAOSFindDeletionTicket

NSFDbGetDAOSTicketList

NSFDbEnumerateDAOSTicketList

DAOSListNLOCallback

NSFDAOSObjectIndexFindEntry

and many others

but they are not documented.

Thanks!

2

Subject: Re: Get internal DAOS information about a given .nlo file

Manipulating individual DAOS objects and getting information about them is not part of the Domino API, nor is it likely to be. The ‘tell daosmgr listnlo [all|missing] somefile.nsf’ command provides an enumeration of all/missing NLO files referenced by a single NSF as part of backup/restore, but that’s it. As is the case with many other areas of Domino, the internals of the DAOS implementation are not published because we may want/need to change them without affecting an installed base of 3rd party applications.

3

Subject: I disagree

I disagree with you. I perfectly understand that you want DAOS to be 100% transparent for the server tasks, but there are always situations where we (developers) really need to know whether a given attachment is inside DAOS or not.

We do attachment statistics and we really need (due to the nature of the statistics) to know if a given attachment is in DAOS or not. Actually this cannot be done. Even if I compute the SHA1+FileSize I obtain the .nlo file name and I can search this path in the DAOS repository but it may happen that this .nlo file is found in the repository but not belonging to this attachment because this attachment was not included in DAOS (e.g. NSF was converted to ODS51 but the attachment was not included in DAOS repository at that time, and another DB included this other .nlo file).

That’s why it’s important to know whether an attachment belongs or not to DAOS and how many tickets it has.

There should be a new API function like:

STATUS LNPUBLIC NSFDbGetDAOSTicketCount( DBHANDLE hdb, DWORD dwObjectId, DWORD *pdwRetTickCount )

*pdwRetTickCount returns the number of tickets or 0 if it is not in DAOS

Unfortunately you are not helping ISV to make our application to be DAOS-compatible… Another example of this is the fact that NSFDbSpaceUsage returns the physical .nsf size while NSFDbQuotaGet returns the logical size… thus if I want to obtain the logical used-KB then it does not work.

Also if you want to see if a DB is DAOS-enable you have to research by yourself (actually it can be done by inspecting the $DAOS flag in the icon document) because you didn’t document it!

4

Subject: Re: I disagree

First off, I’m sorry we disagree on this point, but…I guess we’ll have to agree to disagree.

Secondly, I don’t think we have the information you are looking for. At the object level, the attachment is either inline, or it’s a ticket. If the same attachment is ‘stored’ in 5 objects throughout the NSF file, each of the 5 will have a ticket that refers to the same single NLO file. No ticket has knowledge about the existence (or not) of any similar tickets in the same NSF file, or in any of the other NSF files on the server. All the ticket tells you is that the NLO was last seen in subdirectory nnnn, and the key (NLO name) is kkkk. That’s it.

At the system level, the daoscat.nsf does keep track of how many tickets refer to any given NLO file, but we do not keep backpointers to the individual referencing tickets, or even to what NSF file they were referenced in. All that is stored in daoscat for a particular NLO is the key, and the count.

Why don’t we keep track of more? It’s expensive. Part of the savings of DAOS is just raw disk space, but it also reduces I/O…savings which would be more than consumed with bookkeeping I/O if we maintained more info.

At best, given an object ID, you could find the total number of references in the system to that same key. Finding out how many references there are to a specific key (NLO) in a given NSF would require traversing all of the objects in the NSF file to count occurrences in response to each request. That gets real expensive real fast.

As far as the quota info, I believe there was a bug in the 8.50 that was corrected in 8.51 to yield the same answer (logical size) for both DAOS and non-DAOS NSF files. I know we made some changes there, I’d have to double check what the impact was. We’re considering enhancements to the quota system to allow for different ways of space accounting for attachments stored in DAOS vs inline, but I can’t make any promises or give details, as that hasn’t been worked out yet.

Lastly, DAOS participation info isn’t published yet, and I’d advise against peeking ahead. The DAOS participation flag only controls the disposition of new additions to the NSF. An NSF file can definitely have the DAOS participation flag OFF and still contain valid DAOS tickets. All you have to do is a ‘compact -daos off’ on an NSF file that contains DAOS references. Unless you do a compact -c, all of the previously created tickets will still remain…and will continue to be serviced by DAOS.

5

Subject: reiterating Pat’s message

the use of DAOS is transparent to Domino applications and user’s. We will support the need to do backup and restore. If you build an application based on observed behavior it will only work for the specific version of Domino. This mechanism will certainly be changing ( and has changed subtley in 8.51 ).

6

Subject: about “backup and restore”

Hi Gary,When you say that you will support backup and restore, I guess you are talking about providing APIs for backup and restore utilities. Will these APIs list all the .nlo files of a given DB?

thanks

7

Subject: Re: about “backup and restore”

Yes, but you can already get that information today using the daosmgr LISTNLO command.

8

Subject: NSFDbEnumerateDAOSTicketList

Hi!I know I can use this command but if I want to use it from a server task (through NSFRemoteConsole) it will display stuffs in the console and this is not an option.

I hooked DaosMgr and I’ve seen that when you make “tell daosmgr listnlo” you call NSFDbEnumerateDAOSTicketList with 5 parameters:

2 params are zero

1 is the callback function (DAOSListNLOCallback)

2 are pointers to structures

Will you publish this API in the 8.5.1 API documentation, or will you provide another API to list all the .nlo files beloging to a DB?

9

Subject: Re: NSFDbEnumerateDAOSTicketList

That is not a published API, and the parameters can and will change without notice. I can’t promise one way or another about what the published API will be at this time.

I would suggest you use daosmgr LISTNLO with the -o option to write the results to a file, and then parse the resulting file.

10

Subject: When will this new API be available?

When will the API that allows you to know the NLO files from a given DB be available?If I want to use it from a workstation (e.g. an R8.5 wks) instead of a server, can I use it?

Also regarding the modification you did (or will do) in NSFDbSpaceUsage, rightnow it shows the physical usage and you will change it to logical usage… but both informations will be useful (that is, physical and logical used space).

Thanks

11

Subject: Re: When will this new API be available?

I can’t promise any specific schedule, but at this point it will likely not be in 8.51