Thousands of old docs suddenly reappeared in most users’ inboxes one morning. Stuff going back to 1998 that users swear they had deleted. The usual suspect is, of course, some replica not having replicated for a long time; however, that isn’t the case as many users have one and only one replica. There is no test box anywhere with mail files that a developer might have accidentally turned on replication. A check of user activity does not show another server writing large numbers of documents. It almost looks like some agent “unarchived”, but that couldn’t be the case as most users had no archive. A change was made in the mail template during this period, and that was thought to have caused the problem by somehow causing trash to get put back in the inbox; however, that’s seems impossible because the mailfiles increased dramatically, which wouldn’t be the case if docs just went from trash to inbox. As well, it’s hard to imagine trash not being emptied back to 1998. On the 17th, all the mail servers were brought down uncleanly, and recovery manager kicked in on restart for an undetermined number of files on each server. Is it possible that recovery manager rebuilt mail files with all the old deleted docs coming back to the inboxes?
Subject: old docs reappearing
I agree with pretty much everything you stated. The usual suspect would be old replicas, but it is unlikely for that many users. Unless they were on an old server (which you ruled out). A template change doesn’t fit, because you’d have to still have the deletion stubs.
I’m not familiar with how recovery manager works, but if it had access to copies of the mailfiles when they had the 1998 data in them, then I would definitely suspect that’s where the data came from. That would mean either a problem with how recovery manager is configured, or with the data copies it has access to. (Should there even BE a copy of stuff that old?) So you should definitely look at what is going on there. You might need to tweak and do some more test runs to be sure this won’t happen again in the future.
My choice of solution to the current problem from the user’s POV: If you want to spare your users, get their mailfiles back from a recent backup…but then you have to figure out how to get the newest messages since then into their mailfile. The alternative is to have them delete all over again.