We have a database which contains of 6000 documents. In every document there are scanned .pdf’s. This causes the database to an overwhelming 23 GB. Obviously, this messes up the indexing and performance of both the database and the server.
We have been thinking of creating a seperate(or multiple) databases or ‘attachment-repositories’, and move the scanned attachments there instead . This will dramatically reduce the size of the original database but leaves us with several databases to handle instead of just one.
Another idea is to move the attachements to a fileshare, with links pointing there from the database.
In either case, there are multiple challenges here
How to update the attachments
how to access via web
how to maintain security
how to not affect end-user experience
Have anyone any past experience on how to handle such large attachment databases?
Maybe you can try compressing the attachments… if you can afford a tool, take a look at ZipMail (www.notes-compression.com). There are many compression tools, and you can perform FullText search over .zip files too.
I was involved in a application like this that was only accessed by web users which made it easier from a UI point.
We split it up in one meta data database and several attachment databases.
Upload via web attached the file into the main database.
We had a agent detached it and attached to the preferred attachment database (which one was preferred was configurable, i.e create a new one when the current one starts to grow to big)
The agent stored the database name and a ID for the attachment and the UI created a link to it from the main document.
Security was maintained on a document basis in the application and the agent just created the attachment document with the same reader/author fields as the main document, so even if you got the attachment link, the same access control was in effect as the main document.
The reason we didn’t use a file share was that it would have been necessary to not just link to the file share since then we would have needed to manage access control on the share also on a file by file basis.
We thought of having a agent read the file from the share (i.e not give the end user direct access to the share) and then present it to the user but it was to much of a hassle. We also wanted a clustered environment and we didn’t need to set up a file share replication.
Searching for file contents is more work in a multi database solution (we didn’t have that need)
Before going on with other scenarios, examine this.Split into several “archive” databases, but don’t put attachments in separate database. It will be easier to maintain them.
I may sound ridiculous, but check once again settings of the scanners. 23GB for only 6000 documents is quite a volume.
Check views settings and sortings. But once again it smells like bad settings of user scanners except if attachments are drawings, photos, etc.
keep in mind with this solution (Archive DB), you will lose any Notes Links that you have in the document as the linked document MAY not exist in the current archive.
Ever wanted to offload all the attachments from a bunch of email messages to your local or shared drive in order to reduce the size of your mail file? Well, this is a very user-friendly solution to your problem. The agent is built on code first written by Ulrich Krause and posted on Notes.net (LDD), as well as Julian Robichaux’s excellent Java Status/Progress box implementation which he presented in the JMP201 session at Lotusphere 2006 (Julian’s sample db can be downloaded here: http://www.nsftools.com/misc/JMP201JavaSamples.zip). It will enable a user to remove the attachments from selected mail message and save them to the file system, and place a notation on each message indicating where the attachments were saved.