I have just been given half a dozen large boxes of server backup tapes that cover all the servers within the organisation for the past 10 years (!). The task is to produce one deduplicated set of all unique documents. Any ideas?
Cheers
Ron
Subject: De-duplication across many databases
If you don’t know LotusScript then hire someone who does because you are talking about a massive project there. Not only to weed out the duplicates but what about conflicting versions, author/reader field considerations, etc.
Jeff Dayton, PCLP, CLI
Subject: RE: De-duplication across many databases
This will be an overwhelming task! I would love to help but it will need way more than a thread on notes.net.
You might want to consider defining a project on openntf.org
There are so many options to consider on how to ID a doc as unique.
DocID is only the beginnning. Are some of the DB’s replicas? Do
you just need to track the docs or do you need a copy of it? Will the Design elements need to be kept?
I have a buddy who wrote some code for his job to turn every notes document into a tiff or gif. His code was in java and was very fast.
He might be a good place to start. I will check in with him about their prices if that is something you think can work for the purposes of this.
Shawn
Subject: De-duplication across many databases
Couple of questions:Are you talking about mail files, apps, or both?
Is this a ‘discovery’ thing?
Has anybody defined ‘unique’?
Is there a purpose to this exercise?
I’m not sure the task, as you describe, is actually possible. Let’s look at an app that includes workflow and maybe documents moved between a couple of different dbs as part of the flow. A doc goes through 3 states, then moves from Db1 to Db2. Is ‘the’ unique doc, the doc in db2 at the end of the workflow or are you required to treat copies of a doc in different states as found on different tapes as unique? If your applications generate audit trails, maybe you could simply take the doc in it’s final state as ‘the’ cod, then make sure you have the full audit trail so you can show the status across time.
Mail - total pain. Doc is sent and saved by user1 to users 2,3,4, and 5. Are each of those docs different enough to be considered unique? What if the sender didn’t save his/her copy? Do you need to collect all of the receiver copies? They all have different UNIDs so how would you know you have the set of unique docs?
Cheers? Something to drink? I think you may want to get a industrial size tub of KY…
I guess the point of this little ramble is to suggest that all parties in this project define some terms and agree on what is required and how you’ll know you’ve been successful. You may be able to work out something that is less painful to you and still meets the requirements.
Best of luck - and let us know how it turns out.
Doug
Subject: RE: De-duplication across many databases
Thanks for all the comments. The definition is to some extent up to me. I think that a document is unique based upon an MD5 of the body, I do need to track the the various copies of each unique doc. Most (?all) of the docs will be mail. There will be backups of replicated databases. To sum up: I create a mail memo, send it to three people and one passes it on to a forth person and each of the servers is backed up, perhaps more than once before the mail is deleted; I would like to have one copy of that memo with a note of the sender and all the recipients and perhaps a few dates. I am only interested in the human readable data and, perhaps, some of the metadata within the doc.
Yes it is to do with discovery.
Is it worth doing??? Who knows, not my prob. I get paid by the hour.
Subject: RE: De-duplication across many databases
Ron,
A friend of mine works for a company that has this exact product.
Except the output is tiff images. Would you like to speak to them?
Is a tiff image acceptable?
Shawn
Subject: RE: De-duplication across many databases
Thanks, that is interesting. I shall have to find out if we can use tiff images.
Thanks to everybody for their interest and help.
Ron
Subject: RE: De-duplication across many databases
Thanks, that is interesting. I shall have to find out if we can use tiff images.
Thanks to everybody for their interest and help.
Ron
Subject: RE: De-duplication across many databases
If most docs are emails, you could use $MessageID for checking if it’s the same email. In the case of fwding, the email will be different, but there may have been additional data added which might also be relevant, e.g. just the – fwd by – line could be important
Cheers
Peter
Subject: RE: De-duplication across many databases
Hi Peter
Thanks for that
Regards
Ron
Subject: De-duplication across many databases
Ummm. make sure you have a stiff bottle of something?