Need to delete 500k documents

HCL Domino Domino Forum
1

Hi,

This is going to sound ugly, but I need to delete 500,000 documents from a database. We need to keep a full year, which is why it is so big.

I have written an agent which copies them to an archive database and then deletes it from the master database. However, after deleting around 97k, the master database became un-usable for my users. Nobody could login to the database, either using the web or the ui.

After about 5 hours, the datbase came good and people could re-start their work.

How can I achieve this bulk deleting and not have the database hang ?

THanks

Chris

2

Subject: Need to delete 500k documents

Hi,

A-

Run your agents on weekend.

Try to split your agent for example in 5 parts (100 000 docs each)

Run an updall between each process.

B-

Why not create a new replica on another server(Inaccessible for your users), and then add a replication formula to remove the old documents from your master db?

C-

Or do the opposite of your current agent. :slight_smile:

1- Create an empty db (new replicaID)

2- Put the good documents in the new db. You will then have the old documents in your master db. (Actual replicaID use by your users)

3- Remove the master db containing the old documents from your server. (Put it in archive)

4-Just switch the replica Id of the db created in step 2 by your old replicaID from your master db. So your users will not loose there icons.

It’s not clean but it might work.

JYR

3

Subject: RE: Need to delete 500k documents

Thanks.

A. Weekend is slower, but still causes problems, as the database is 24/7. Also, the problem first surfaced at 97,000 documents - and I am not quite sure when it really started - could have been a few thousand before that. I could try (say) 20,000 lots, but that would take be 25 days !

B. What impact on server speed/availability would this have ?

C. Sounds too dangerous - would all the references from the web interface also carry over ? I might try it on a smaller db first.

Thanks

4

Subject: RE: Need to delete 500k documents

Don’t archive more than 1000 documents at once when the agent runs on a 5 minute schedule. When you delete too many documents at once, it takes too long to index the views. Frequently done small changes won’t make the indexer use noticable time.

5

Subject: Solution !

I created an agent that runs every 30 minutes and deletes 500 documents at a time. So, it will take me 25 days to get through the list, however I am not experiencing any web slowdown.

Thanks for the idea.

Chris

6

Subject: RE: Need to delete 500k documents

Hi Chris,

Just repeating what others have said here, don’t delete documents en masse from a production database. I tried this recently and ran into big diffs.

What I ended up doing was creating a local replica of the database with a replication formula to pull down only the “good” documents, deleted the database from the server, and replicated back up the local database once I was happy that all the right documents were there.

Don’t know if that is suitable for your circumstances though.

Cheers,

Andrew.