I posted few days ago, and the answer for having separate databases for different data, and a summary database is not possible. I am thinking of having a sales vs budget database, which involves > 100k documents and may be talking about 100gb of database size with indexes etc. Think this is bit crazy? or this is dealt with by notes using db2? which I am not quite sure currently.
Regards
Will
Subject: ND is easy to build an application, but
it still requires design. I have been called in to try and rescue a number of Notes systems that just grew and added views and data without considering the implications on performance. The tendancy is to do incremental design with no over all plan for where the project might end up. I got called into a shop where the whole Notes/Domino install was about be turfed out because of one huge poorly designed application. Started checking it out and they had added a new view every time someone had a new idea. Many of the views contained formulas with @Today in them. There were over 250 views.The first thing we did was determined which views were needed and which ones we could consolidate (Single Category views saved us a bundle). Next we write a couple different agents that ran once a day that adjusted some fields in the documents so that the views did not have to recompute dates etc. After a couple weeks work cleaning things up they were amazed at how well it was running, we were heros (thank you very much) and it all revolved around knowing where ND takes a hit and the things to avoid.
Building solid applications is an Art not a Science!
Subject: The mammoth 100gb notes database-too crazy?
Hi,
Just a thought, but have you looked at using a product like Domino document Manager (formally domino.doc)
I used it at a previous company, and was a good way to manage huge amounts of documents (as it looks like you’ve got). You probably can build something along the lines of what you’re thinking about, but buying a proper document management tool might be the way forward for you, and make you loose a lot less hair.
James
Subject: The mammoth 100gb notes database-too crazy?
What’s in your docs? 100k documents is not that many, but each doc must be 1Mb to make it that large a db (think my maths is right).
You say sales vs budget, so surely it’s just fields with numbers, or are you storing contracts and the like in the db as well?
There must be a way of making this db smaller. If it’s mostly index then you should look at the way your design is set up. This doc might be of interest to you:
Dan
Subject: RE: The mammoth 100gb notes database-too crazy?
I have about 30 views, and each view has at least 10 categorised columns, 7 time dependeant columns. I have a database with 40000 documens, and it is running at 12 GB already.
The problem is that each month I will have around 1m transactions to be loaded into this database, (currently I am only testing with one region, but once I included all the region in the world, it will go humongous).(sorry, not 100k documents, I underestimated).
the form has around 20 fields with dblookup formula.
perhaps i can create 12 views instad of one view which will take YTD based on current month.
But other than that, I don’t know why from 300mb it grows to 12 gb currently, perhaps I have too many views ? but I dont have much choice, perhaps I can do replication to other database, but with different view yeah?
any other comments are appreciated.
Subject: Rethink your design…
All of the design decisions you describe scream ‘rethink your design’. Too many categories, too many dblookups in a doc (you can do it in 1, search around for how to do that), real time data in views - all very very bad for performance. Rethink, rethink, rethink.
If you’re being pressured to drop Notes (one of your prior posts) and go to a MS product, this kind of design will help others help you make the decision. It’ll be a pig and it’ll be Notes’ fault even though it’s most likely poor design (sorry, but you need to understand the problems sometimes belong to the designer).
If you’re not able to think of a new/better way to handle the design, you should buy some help.
Having said that, 1 million new docs/month is its own special problem. Notes may not be the answer. Again, some local outside help would be userful. As I have told you before, find your local IBM or business partner contact and ask for their help.
Rethink the design. So you add 1 million docs a month. Are all of those docs needed at any one time? Could you take snapshots of the data every hours/days/weeks, save the a couple of snapshots and then push the data to an archive db? Could you store the bulk of the data one place and just have a db of links? The link db allows users to figure out what they want to see, they hit the link and get pushed to the actual data?
Understand why your users need a zillion views. Maybe they’re stuck in a view mindset and that’s not really what they want or need. Challenge their assumptions.
Rethink…
Stuck and need some fresh perspective?
RE-THINK
Doug
Subject: RE: The mammoth 100gb notes database-too crazy?
Will,
Find someone in your area who you trust and hire them for a few days to hash out a better architecture. The few thousand bucks will be money well spent at the end of your project.
There are domino user/developer groups in many areas, so maybe you can hook up with one for a month or two to check out individuals and find someone you can trust.
At Lotus, there is (or was) a group called Field Support Sevices. If your company has a big support contract then you can use a couple days of an expert from that group to go over architecture with you. Back when I was young I worked in that group and frequently did this kind of work. Again, it’s a few thousand bucks which is just peanuts for a big project.
I’m not saying they’ll come up with something perfect, but they’ll offer you a lot of expertise and experience which I think would be valuable.
good luck,
raphael
Subject: RE: The mammoth 100gb notes database-too crazy?
How I wish that the view could select documents from other database, than, I don’t have to worry so much on view index formula. Perhaps something for Lotus v9? 
Subject: RE: The mammoth 100gb notes database-too crazy?
When you say time-dependant columns, do you mean they’re doing calculations based on current time? This is a big no-no, and you need to get rid of them and find another way of doing it.
Embedded single category views are the normal way of doing it, so look at those.
1million(?) transactions each month? That’s a lot, not something that Domino is ideal for to be fair so you might want to look at DB2 or something, but that’s not my area of expertise (if I have one:))
Also, does the database get compacted regularly (using a -b switch). Check the disk usage and see if it’s very low. Lots of updates to the db could be causing a problem.
Other than that, 30 views isn’t a big problem, the more categories you have does affect things so see if you can get rid of any of those.
If you’ve got multi regions you could quite easily set up a different db for each region. Linking between different dbs is pretty easy, and creating scheduled reports from data in different dbs is also possible with a fair amount of lotusscript and a separate reporting database that can amalgamate the totals (or look into a reporting tool).
hth
Dan
Subject: 100GB exceeds Windows file limits
The 64 GB limit on a database size is imposed due to Windows restriction, not Notes. Unless you are on AS400 or the like, you will probably receive a Windows file size exceeded error.
Subject: RE: 100GB exceeds Windows file limits
From an AS400 guy, thanks… 
We have an app that is 40 GB with 208k documents. Each document has a scanned PDF. The largest view index is 108 MB and has no categories or date/time columns. That view isn’t used much, but I just opened it in under 1 second. However, a view categorized by date in the same db takes much longer for an initial load (2 minutes!).
But it seems that Will is just trying to kill the performance of the application by having so much in there that goes against conventional wisdom (many dblookups in docs, many categories and dates in views)…
Subject: The mammoth 100gb notes database-too crazy?
As a point of reference, I have a number of dbs with at or over 100K docs, including lots of attachments, and they’re all under 3 gig. Why do you think you’re going to go over 100 gig?
Why not just test and see how your performance is. Create a db with all of the views, forms, whatever you expect to see. Make sure you’ve done adequate design (no docs with a zillion dblookups on load, no time columns in views, you know the drill). Add in a few docs representative of the kinds you think you’ll get. Write a little agent to create 100 copies of the docs and then start using the db - open all the views, etc. It’d take you less than a day and you’d KNOW how the system performs.
Doug
ps - I think a 100 gig db is nuts. For the user who only occassionly opens the db (or some view in the db), the initiall view load times are going to be killers. If 'twere me, and I really knew the db would go over 100 gig, I’d rethink my design. Why do I need all of those docs in one place? Do I need 100 views with lots of categories? Can I rethink the design so the stuff people need is in one place and the stuff they need sometimes is somewhere else?