A business case question : database size optimization of a huge documents container database

HCL Domino Domino Forum
1

Hi,

A business case question : database size optimization of a huge documents container database:

We currently use a huge Lotus Notes database as a big documents container for our related companies information.

This db contains by company one “Master Document” and from 0 to N subdocuments (mainly with “attached files”).

(average N=10)

For this purpose, I designed ONE db containing all the documents types, because it was the simpliest way to do it (it was an emercency development request!) and because the user wanted to have a categorized view (By Company), showing all documents types (with “Master Document” in first position in the category)

All Data’s have to be kept during 3 years ;

an automatic archive program will periodically erase older data’s

Up to now,

this database contains ± 80.000 “Master Documents” (this is complete - all these documents were programmatically imported from our mainframe);

this database contains ± 40.000 subdocuments (many of them with attached files) but all the documents are not yet in the database. (These subdocuments come from different sources and have to be pushed by users from others Notes databases, via action buttons).

This database should contain about 240.000 subdocuments when it will be fully operational.

The size of the db is currently 8GB and the responses times are fully satisfying.

But I’m afraid the size becomes too big (more than the 64GB limit or too close to this limit to be practically used).

A db size of 55GB (full-text indexed) can be useable without dramatic responses time consequences?

If I have to split this database, what would be the best way?

Solution 1. :

Programmatically remove all (or selected) attached files and replace them by a field containing link(s) to files detached on file system?

(The disk space on the server is not a problem)

In this case, the size of the db will reduce and not be a problem any more, but the NUMBER of documents should stay very high: 320.000 documents. Isn’t it too much?

Solution 2.:

Split the db in 3 dbs:

  • dbase0 containing the Master" Documents"

  • dbase1 containing Subdocuments of type 1

  • dbase2 containing Subdocuments of type 2

In this case, I’d have to replace the categorized view by

  • in dbase0: a form containing 2 embedded views ( 1 coming from dbase1 + 1 coming from dbase2), with ‘show single category’ restricted to 1 company = ± 80.000 documents

  • in dbase1: 1 embedded view containing all documents (=SubDocuments of type 1) = ±120.000 documents

  • in dbase2: 1 embedded view containing all documents (=SubDocuments of type 2) = ±120.000 documents

Is it possible to use a form containing so big embedded views

(even if the ‘show single category’ displays only ±10 subdocuments)

Could this principe also used via the Web, as this information would have to be seen from another application?

Remark:

Of course, furthermore, in this case, I’d have to modify the UI, to automatically route the documents to the good db, when the documents are created…

Solution 3.:

Solution 1+ Solution 2

What would be the best solution amongst these 3?

Or do somebody know a better one?

Thank you by advance for your help,

Philippe Pierlot

Lotus Notes Designer

Belgium

2

Subject: A business case question : database size optimization of a huge documents container database

Your solution number one looks the best to me. One possible would be rather than copy them to the FS place them in a document in a second DB the a doc link from the parent, particularly if there is any replication then Domino makes the Attachments available.

The main issue I would say is get the attachments out of the Main DB one way or another, because every time the user accesses a document the whole attachment is sent over to the client attachments and all. This represents a tremendous amount of bandwidth useage.

3

Subject: RE: A business case question : database size optimization of a huge documents container database

Hi Bill,

I think i’ll use solution 1. (attached files replaced by link to file system).

Your suggested solution with link to another container database (for the attached files) seems nice and gives the advantage to make possible the use of the search engine (which it’s not possible with link to file system, unless with the use of ‘domain search’ including files system).

But I’m afraid it would be more difficult to implement than links to files on file system, as the application will have to be reached via the web. (I want to use a simple ‘show single category’ html embedded view for this purpose).

Thank you for your help…

Regards,

Philippe Pierlot

Lotus Notes Designer

Belgium

4

Subject: A business case question : database size optimization of a huge documents container database

Hello Philippe

I understand your problem, but as I see this, I would never split this in 3 databases. Or, never say never… But, it would make the userinterface less interactive, and the maintenance between the databases and their documentrelationship etc could be a nightmare.

So speaking, I like your solution 1 the best. But again, are indexes of the attached files a requirement? If so, solution 1 is not a good idea afterall. If not, and you find a goold “link-to-file-solution”, that should be a way I would prefer.

Now, having said that, - what kind of attachments are there? if they are mainly word document, then you have the problem exactly there. They consume alot more diskspace than necessary. Have you considered trying ZIPMAIL? I have never tried it in a non-mail-database, but I think that should work, and it will decrease the size of the attachments dramatically.

A big database, doesn’t necessary need to be slow. That depend on how your views are designed, and how, when and what are indexed on the server. Even if you need to keep docs for 3 years, maybe you could have views that are optimized for day-to-day work, which only include docs that are necessary for these purposes. This would give the experience of a faster database.

Summarized:

  1. I like Solution 1 best

  2. Check out zipmail

  3. Try reorganizing your views

  4. Link to filesystem if needed

Rune

5

Subject: RE: A business case question : database size optimization of a huge documents container database

Hi both

we currently have exactly the same problem, and we decided to split de DBs into 5 new ones.

BUT, considering that you have many attachments, why do not use a more efficient architecture,

such as Domino.Doc ?

That way, you could keep your master documents, and share your attachments within specialized databases,

and e.g. a connector for windows explorer…

HTH

6

Subject: RE: A business case question : database size optimization of a huge documents container database

Hi Yannick,

Thank you for you answer.

For only one application, we’d like to avoid to have to buy and set up domino.doc.

But it the near future, it seems that we’ll have more and more applications related to document management (it’s a logical trend).

So I’m finally examinating the pro and cons of upgrading to domino.doc.

Thank you for your suggestion,

Regards,

Philippe Pierlot

Lotus Domino Designer

Belgium

7

Subject: RE: A business case question : database size optimization of a huge documents container database

Thank you Rune,

I think I’ll first set up solution 1 (with link to file system).

According to the answers I received in this forum : a great number of documents can be useable, if these documents are small.

=> so with the attached files removed, the database’s size will seriouly reduce => so it should be OK

and i’ll see later if a split is finally needed

To optimize the views :

unfortunatelly, documents have to stay active during 3 years and some of them have sometimes to be manually updated after (let’s say) 35 months. So all documents must be available in the ‘daily work’ views.

=> I’ll try to optimize the views (indexing, columns,…) to make it as fast as possible.

Zipmail : we’d like to avoid the use of another third part product (it would be one more product to have to ‘sell’ to our managers).

Moreover, with the java classes, it seems easy to implement programmed zipping and unzipping on demand or on the fly…

Thank you for your clear answer who makes me more confident for the future of our document management application.

Regards,

Philippe Pierlot

Lotus Notes Designer

Belgium

8

Subject: A business case question : database size optimization of a huge documents container database

Keep it as one database, and archive more frequently f.x.Active db (this years doc’s)

Archive 2005

Archive 2004

Archive 2003

Include a muldi db search feature, and you are good to go.

9

Subject: RE: A business case question : database size optimization of a huge documents container database

Hi,

Thank you for your fast answer.

OK - archiving by year is another possible ‘split solution’.

Maybe it’s not the best one, as the documents are considered as active during 3 years, and the users want to see simultaneously all the active documents for one company.

(Archive procedure will already be used for documents older than 3 years…)

I don’t think I will use this solution, even if I have to split the database and use the multidb search features…

But it is an interesting proposal anyway…

Regards,

Philippe Pierlot

Lotus Notes Designer

Belgium