Increasing Notes Mail Database (NSF) Size

HCL Domino Domino Forum
1

We’ve a Lotus Notes Mail Database originally in Notes 5. Now we have notes server 6.0. We’re accessing the same database using Lotus Notes C++ APIs.

The opertaions which we are performing on the database are

  1. Opening Database.

  2. Accessing each document.

  3. Opening each document using document ID.

  4. Reading the Value of each document field.

  5. Closing the document and database.

Note: We are not performing any write operations.

In this situation, why my notes database size is increasing?

It is not that the database size has increased just once, but it is increasing each time (if We’re performing steps 1 - 5, mentioned above)

2

Subject: Increasing Notes Mail Database (NSF) Size

Hitesh,

I would imagine that opening the db results in a view index being created or updated. This would not explain why the size increases each time, however. Can you document the delta each time?

I wonder if some of your actions result in an id table being updated in the database?

regards,

raphael

3

Subject: Re: Increasing Notes Mail Database (NSF) Size

Hi,

I’m from same team as Hitesh. Database in which size increase is noticed is originally based on R5 (ODS 41) release and is used with Domino6 server. R6 databases are not showing this problem.

View index being the reason is already analysed and in an IBM article ‘Impact of maintaining R5 format ODS 41 databases: Database size increases’ it is written that size of an R5 database used on R6 server will increase due to view indexes for both R5 and R6 format maintained. But according to article, this should only occur first time when database opened. Also it is seen that total view sizes is only 800 KB and is fixed even when database is growing in size.

On referring to C++ API code, it is noticed that whenever we are opening a document (LNDocument::Open) containing attachments, each open call is increasing size.

Here are the statistics…

For a document with 5 attachments, 4MB size in total, db size increases by 1.75 MB or 2 MB in a pre-defined sequence… 1.75->2->2 …and then back.

Overall, a database 60.5 MB in size growed to 120 MB on reading all documents.

For these documents, MIME-CD conversion is done at this time and window is popped showing this operation. As read in a net article, its means that some MIME content is converted to RichNotes format at this time.

But why is MIME-CD conversion increasing database size? Even if so, it should have stored RichText object in it this time, why is it creating object again and that too not disposing the previous one? Is there a way to disable MIME-CD conversion via C/C++ Notes API code?

Both C and C++ Notes API calls to open document has been tried but both results in increase in database size.

Another thing noticed is that when we do ‘Compact’ on this database from Domino administrator, it is restoring back to its original size. But it is undesirable to do compaction each time we do database operation.

4

Subject: RE: Re: Increasing Notes Mail Database (NSF) Size

Are the docs encrypted?What is the C API code?

5

Subject: RE: Re: Increasing Notes Mail Database (NSF) Size

No, there is no encryption set on database. Even for document i am monitoring, $EncryptionStatus field shows 0 value, which probably means no encryption. The code is mostly on C++, but when it shows to be increasing the size, i also tried C equivalent call of lnDoc.Open (document opening call), but that’s also increasing database size.

Here is the code snippet…

LNNotesSession session;

session.Init()

//Opening Database Logic

session.GetDatabase(strDBName, &lnDB, strServerName);

lnDB.Open();

UNID unidDoc;

//below unid is unid of notes document with 4 MB attachments

StringToUnid(L"b1959afa2bdd03ba6525709a003ebad3", unidDoc); (app. defined function converting string to UNID)

{

LNDocument lnDoc;

HANDLE hNote = NULL;

lnDB.GetDocument(&unidDoc, &lnDoc);

lnDoc.Open(LNNOTEOPENFLAGS_DEFAULT);

//below is C equivalent API

//NSFNoteOpenByUNID(lnDB.GetDatabaseHandle(), &unidDoc, OPEN_NOVERIFYDEFAULT, &hNote);

}

The MIME-CD message that is shown on console window is as follows…

01/17/2007 12:28:16 PM Begin MIME to CD Conversion (Process: ? (00000894:000000

02), Database: CN=dominoserver6/O=hcl!!amresh_small.nsf, Note: 00000B1A)

01/17/2007 12:28:19 PM End MIME to CD Conversion (Process: ? (00000894:00000002

), Database: CN=dominoserver6/O=hcl!!amresh_small.nsf, Note: 00000B1A)

-Harvijay

6

Subject: RE: Re: Increasing Notes Mail Database (NSF) Size

Can you only use C API code and not C++ API code? remember that C++ API is a wrapper of the C API so if there is a bug then it will be out of your scope…

Also, are you closing the document? (NSFNoteClose)

Do you have something (like an extension manager) installed in your server?

Do you have “don’t overwrite free space”?

Is your wks R5 and your server R6/7 and the atts are in LZ1?

7

Subject: RE: Re: Increasing Notes Mail Database (NSF) Size

I have found the problem. The problem is basically occurring when we open a document which was not closed properly last time it was used. MIME-CD conversion is done every time we open such document, but if the most recent open on this document is gracefully closed, then this conversion doesnt increase database size.

Moreover, the problem is also in new R6 (ODS43) copy of this database created from R5 database using compact/New replica/new copy.

So in summary, here are the conditions in which increase in database size will occur when we open a document :

  1. document contains attachments

  2. document is doing MIME-CD conversion. This is happenning always if we use an R5 database or its R6 copy on Domino 6 server. The databases created in R6 right from scratch doesnt found to be causing this.

  3. The most recent open on this document is not closed using lnDoc.Close(). The destructor of LNDocument object doesnt seem to be doing this and that’s why this problem is happenning.

Now let me share with you what’s going in conceptually according to me in these conditions:

It seems that such documents contains attachments stored in MIME format, but there is some flag set in such databases due to which whenever documents in such database are opened, MIME-CD conversion is done that converts document MIME attachmments to NotesRichtext format and stores the converted object internally in database for later use.

When the document is properly closed, this new object is destroyed. But in condition when we dont close document and LNDocument object goes out of scope, this new object is already commited to database but its reference is lost and hence new NotesRichText object is created for each subsequent open on this document and hence size increases.

Moreover, it is seen that if we iterate all notes documents in such database without closing them, whenever database object is destroyed (when it goes out of scope), all objects created by these documents are stored in database consistently. This results in a larger increase in database size. But now if we do same iteration again, all objects are already in place and hence no increase in size noticed.

Now consider another interesting condition in above scenario, if after iterating all documents, you kill the process before database object goes out of scope, all these object’s references are lost and they will again be added in next iteration, resulting in increase in database size again and again.

That’s the case which was happenning with us, in a test which i have done, 13 MB is added to a 49.5 MB database each time such iteration is done without letting database object to go for destruction using ‘Kill process’ before it goes out of scope.

…(can cause DoS attack on Notes database by exhausting the hard drive)

-Harvijay :slight_smile: