DAOS: No Database volume reduction? (and some other minor problems)

hcl-bot · June 2, 2008, 7:23am

I’ve enabled DAOS on a Domino 8.5 server with ~95 GB test data.I’ve set Create_R85_Databases=1, configured TL as circular & Shared Mail = None and configured DOAS with a base path C:\DAOS

I’ve switch on DAOS on every database on the directory mail

I’ve restarted the server an every database got a new DBIID.

Although Create_R85_Databases=1, the DAOS Catalog (daoscat.nsf) was created as ODS48 database and not as ODS50 database

To be sure, I’ve waited for TL activation to end and restarted the server once again.

I’ve fired lo compact -C mail, then

When the databases were compacted, I’ve seen “Informational, DAOS is enabled in database …”

I’ve seen lots of

DAOSIncrementFileCount: <Date/Time>

DAOSDecrementFileCount: <Date/Time>

When compaction was finished, C:\DAOS was ~45 GB big, but

there was no volume reduction on the mail databases in the directory mail!

I’ve restarted the server once again and run compact -C mail again, but the same result:

C:\DAOS was filled, but no volume reduction in mail

What went wrong?

hcl-bot · June 2, 2008, 6:45am

Subject: Re: DAOS

Hello Christian,

You have reported that your C:\DAOS directory has had ~45GB of data transferred to it since enabling DAOS on the mail databases of your server, yet the mail directory has seen no relief. What means are you using to confirm the size of these directories?

I look forward to your response!

hcl-bot · June 2, 2008, 5:04pm

Subject: Explorer & CLI to get the size of the directories

Hello!

Well, very simpe:

To get the Size of C:\DAOS & \mail, I just looked into the Windows Explorer… Before DAOS was enabled, only the mail directory was the with ~96 GB of data.

I’ve checked the database ODS, it is ODS50 for all databases and the compression rate in the db properties is >99 %. Domino Attachment & Object Service is enabled for all databases, too.

I’ve checked it once again and after numerous compact -C -D mail runs, C:\DAOS is now 132 GB of size, 460.700 files.(!)

I’ve done some test before starting experiments with DAOS:

I’ve written a script that will count all attachments and the total size of these attachments. The total count is/was 275388 attachments and a total size of 72,54 GB

I’ve written a MD5 analyzer to analyse how often a very attachment is found in the data.

The total data volume of attachments were 72,54 GB and in average, every attachment was in the data twice.

Therefor I would estimate a volume reduction of 50 % max, resulting in 35 GB of DAOS store max and a corresponding reduction of the mail database size.

The awaited data volume after DAOS redundancy reduction would have been ~30 GB in C:\DAOS and ~ 65 GB \mail

Hypothesis:

I think with every compact -C -D the DAOS store is filled with the attachments in the mail databases again and again, because DAOS does not remove the attachments from the databases and DAOS does not identify the attachments that are already in the DOAS store.

I think, after the first run, it might have been round about 30 GB, and with every compact -C -D, the amount of data in C:\DAOS is increased, because the attachments were not removed from the mail databases.

I think, there is a bug in DAOS introduced in this very build which was not there in older builds, i.g. pre Beta 7.

I don’t want to sent detailled data to IBM via this public forum, so we may discuss further steps IBM internally?

I’m using a defined amount of test data (~96 GB) of mail databases cloned from a productive system.

So, I am able to repeat test with the identical data, if it may help.

Sorry, if I’ve only done a dumb mistake, but I think I didn’t done it and there is a Bug in DAOS in Public Beta 1 which was not there before.

Best Regards

Christian

hcl-bot · July 18, 2008, 7:35am

Subject: Need more info

to diagnose this problem

Can you display the DAOS columns in the admin and paste the size information from there ?

for example

looking for the size, daos count, daos size

hcl-bot · June 2, 2008, 11:27pm

Subject: works for me. I did a quick test with my current test data - not sure what went wrong in your case

I am working on a couple of test right now and I took my mail-files for testing (the current one and all old ones …)

here you see the NSF files before DAOS enablement

02.06.2008 23:26 808.452.096 nsh.nsf

02.06.2008 23:27 901.251.072 nsh2.nsf

02.06.2008 23:28 843.317.248 nsh3.nsf

02.06.2008 23:31 1.995.440.128 nsh4.nsf

           4 File(s)  4.548.460.544 bytes

than I did

load compact -daos on -C testmail/nashcom

here are the same files after DAOS enablement

03.06.2008 00:08 151.519.232 nsh.nsf

02.06.2008 23:58 61.079.552 nsh2.nsf

03.06.2008 00:00 81.002.496 nsh3.nsf

03.06.2008 00:05 281.542.656 nsh4.nsf

           4 File(s)    575.143.936 bytes

and here is the summary of the DAOS Directory with all the NLO files. before it was almost empty. just a couple of files

10280 File(s) 3.584.922.813 bytes

I don’t have a lot of overlap because it is my own mail-file only.

DAOS will reduce the space a lot more with multiple users.

I also got the following debug messages.

Still testing we should talk offline tomorrow…

I could do another step by step run. First enabled DAOS on the database and than run a compact -C. I did both in the same stepan compressed files I got re-compression to LZ1 working.

But most of the objects have not been recompressed.

– Daniel

DAOSIncrementFileCount: 02.06.2008 23:56:44

DAOSIncrementFileCount: 02.06.2008 23:56:45

DAOSDecrementFileCount: 02.06.2008 23:56:45

DAOSIncrementFileCount: 02.06.2008 23:56:45

DAOSDecrementFileCount: 02.06.2008 23:56:45

DAOSIncrementFileCount: 02.06.2008 23:56:45