Maximum number of documents (Database size)

HCL Domino Domino Forum
1

I have an application were i divided it to 7 notes databases.

These databases will replicate with a db2 database to get records from it to my single Notes database.

  1. Number of records in db2 database is 53 million records where each record contains average 20 text fields.

  2. Way of transfering is via LEI.

  3. I got around 10 views where each view contains 3 sorted columns, and very simple view selection formula (select form = “formname”)

where formname represents a db2 record.

  1. My views wont be available to users, they wont view documents via views, but my application will search for a certain document via lotusscript agent and display data

from it on a form i designed, and i think this is a good issue.

I want to calculate my notes database size, for these 53 million records when they will be 53 millions notes document??

Will the database be able to cover this number of documents??

Will each view be capable of continaing these records???

does each view take a certain space???

Sorting, number of columns take space?

Important: i search for the document via “getdocumentbykey” , will it be slow or fast?

I created a dummy database, and i runned an agent to create 2000000 documents of the same type of the one above, i found the

document size 286b, and the database size around 2 giga.

So how to i get answers for all previous Questions?

Hope someone can answer me asap , or give me a refrence to these answers, i got the application performance pdf , but

still dont have the answers for it.

2

Subject: Maximum number of documents (Database size)

Hello,

Nice to meet you!

I have some opinions for your cases.

  1. Why must we migrate a large amount of records from DB2 to Domino database ? If you still continue to maintain DB2 system, I think you should use DECS/LEI, or DB2 Information Integration.

  2. Please pay attention that Domino Database is not a datawarehouse-supported system.

  3. No problem when you apply LEI into this case, but you should take advantage of the existing DB2 system. You can consider DECS/LEI as a data access bridge. DECS/LEI can become a bridge to access into DB2 system through “Virtual Field/Virtual Document”. Please refer DECS/LEI documentation.

3.1. If your requirement is a reporting system, you can use Crystal Enterprise 8 or 9, and build a Domino Web-based application for reporting. End-users can access/collect information through your Web-based application which get data from DB2 system.

  1. How many average size of each record? How many are categories of 53 milion records ? Make a plan to architect a Domino Database appropriating with DB2 database properties and the businesses/benefits which it has solved.

  2. Please build some views to classify documents (records). Whether you like it or not, end-users are still able to create some private views. I’d rather use FTSearch than “GetDocumentByKey”. FTSearch s the best tool to search documents in a view, database or domain, and applicable to Notes clients or Web clients.

  3. The maximum of Domino Database is about 64GB.

Anyway, you should consider your case as a real software project and build a full-plan. You can contact with a IT Consulting Company.

Good luck!

Best regards.

3

Subject: Maximum number of documents (Database size)

If users are never accessing the documents directly, and you are simply processing that data via Lotuscript and presenting in another form, why move it to Notes at all? Just use LEI to access the data in DB2 realtime.

If you must move it to Notes, you’re probably not going to be satisfied with the performance of a db with 53 million docs. You said yourself that 2 million docs yielded a 2GB file size. It’s going to be pretty linear past that, so you can expect a 53GB NSF, which is probably not going to perform well. Once all your views are built, you might get acceptable performance, if the data doesn’t change frequently, and you have a powerful server with excellent disk I/O. But I wouldn’t count on it.

This is not a sound strategy for the application. You really should run a real time query against the DB2 source, and leave data of that volume on the platform that will manage it best.

4

Subject: Maximum number of documents (Database size)

Thanks all for your reply,

but the idea is that i want to know how to calculate the notes database size. i created a database

Initial size: 387k

after creatin 400 documents doc size = 24b database size was 578k

a nothre solution was when removing $Updatedby field it also decreases in size but there is something errevelant to doc size to database size

there is something hidden that increases the databse size every time you create anumber of documents

for ex:

1 DB: 500 k ---- 200 doc

2 DB 2500 k — 400 doc

where is the increase?

5

Subject: …and we’re telling you that you’re probably doomed to failure with a 58M record design

6

Subject: RE: Maximum number of documents (Database size)

Hello,

I think that is caused by:

-Full-Text Indexed Database.

-Some hidden fields, or default fields which are auto-created when a document is created. With the same data, a document size is not equal to a response document size.

-And the compressed mechanism of Domino Database, such as a rich-text field containing a Word document will be compressed. Word data size is equal about 2/3 the original size.

So, there are many things effecting to database size.

Nice to meet you!

I’m the team leader of Saigon Information Technology Company - CINOTEC Co.,Ltd in Vietnam.

nhatnamnv.

7

Subject: RE: Maximum number of documents (Database size)

Well guys,

thanks all for your responses, but i think i will continue with my plan Notes database (53 m document) because we have already started in the progress, and since lotus mentioned that the maximum size is 64G upto , then for sure they have tested its performance, or then we wont believe them in anything they say and it will be prefferable to work with another tool!.

I created a database with 12 million document till now, and i keeped searching for a document with different criteria’s on a web form, and it returns the result very very quickly, a second or less.

So i will continue and i will start to enhance in my database design so that i can reach to minimum size i can.

Again thanks all, and if anyone has more support or any info can be useful to me, then i wil appreciate that.

Thanks,

Aiman Eltahawi

8

Subject: If you’re going to do this…

Hi there,

That’s a big database. Very big. I’d suggest you consider pruning your data before it gets into Domino. If you’re going to do reports and the like, you can aggregate the data in advance and drastically cut down on the documents in Domino. Whatever path you take (and I know a bunch of the gurus here will jump in), you’ll need some very experienced help…and I strongly encourage you to get help as your project will likely suffer seriously without it.

…Neil

9

Subject: Maximum number of documents (Database size)

Lotus isnt the solution!!!, i will use db2 database, thanks guys for your advise