Corrupt images/Save Conflicts

I have been dealing with an issue or several issues for the past two weeks and I am out of ideas of what to look at for resolution. I have a database that has been in production for 3 years with no problems. We have done no major updates to any of the code in over a year. On Monday, November 18th, we began to experience several different problems that we had not really seen before. Nothing was changed in the application or in Notes in general. The server is running at 6.5.3 and the client version is 6.5.5. The designer client that we are using is 7, but there are no 7 specific features used in the database. To make our Network Engineering group happy we even recompiled the code in version 6.5.3 with no issues.

Here are the details of the issues that we are dealing with and my questions. Any ideas that would help us determine why these issues just began out of the blue would be extremely helpful.

For the last week I have worked full time to fix replication/save conflicts or corrupted documents in ACE. We have never had these kinds of problems in this database before. This database has been in production since 2005 and no major updates have been made to it in over a year. There are several different types of problems occurring but they are all happening to only one type of document - a document that contains Rich Text Fields where a narrative write up is entered. This write up can contain formated text as well as tables, graphs, maps, etc. These problems are not happening to all users. Also, a user may experience the problem with one document but not another that is very similar.

  1. Replication/Save Conflicts - I know this generally occurs when two people edit a document at the same time. The second person who tries to save the document will get a conflict. Notes does a pretty good job of locking documents so that this doesn’t happen. Also, in our application we have a second layer of document locking that we programmed to catch any scenario that Notes does not catch automatically. We may receive 1 to 2 save conflicts a week. In the last week I have fixed about 100 a day.

It was suggested possibly turning off replication to the DR site to see if that replication was causing the conflicts. That can not be the cause of the problem. For DR purposes, we only do one way replication. The DR server is set to pull changes from production. Nothing ever goes the other way from DR to production.

I personally created a save conflict when editing a document this week that I know positively no one else was editing. Also, yesterday I had a document open in read mode (I am positive about that) when I closed the document, I created a conflict. This is very strange since 1.) I didn’t edit the document and 2.) I didn’t save the document.

I also had a user that created multiple conflicts while he was editing the document because he was using the save icon to be sure he didn’t lose any of his work. This is the suggested practice that all users are taught during training. I tried to duplicate this issue with no luck. Using the save icon while editing a form in the production database did not cause me to create any save conflicts.

I reviewed the code in the form and didn’t see any obvious problems. There are only 3 ways to save the document 1) manual save of the document by using the save icon or doing ctrl S 2) click of the OK button does a @Command([FileSave]) and closes the document 3) Printing the document does a doc.save before sending the document to the code used for printing (we use IntelliPrint to print the documents). I thought it was possible that it could be the print button causing the issue, but again this code has been in place since we went live in 2005 and is used on all the forms in the application. We are only seeing a problem with the one form. I tested a document where I edited, saved, printed, edited, printed, saved, edited, printed, printed, saved, etc. No matter how hard I tried I could not create a conflict.

So, my question would be why are we getting conflicts now? We haven’t changed the code, the users haven’t changed the way they create this document, we haven’t changed the Notes environment, so why did this start happening now? And, if it was a Notes/Application problem, why don’t all the users have the same problem?

  1. Users are receiving a message saying something to the effect of “Ignoring portion of document updated by later version of Notes”. In this case, the message is misleading. It has nothing to do with the version of Notes but the fact that there is a rich text field on the document that has had an image pasted into it and the field size is now too large. Although the users have always pasted tables and graphs from Excel and Word into the narrative against my recommendations, I have rarely seen this message - probably not more than twice since 2005.

To fix this problem, I have to figure out which field is causing the problem and delete the field from the document. The user loses any data in the field.

My question here is although I agree that the users are causing themselves issues by copy/pasting so many tables and graphics into the document, why has this only become a problem in the last 2 weeks? Their procedures have not changed; they are doing exactly what they have been doing for the last 3 years.

  1. Users are receiving a message “Error copying Metafile”. This is a new message to me; I have never seen this message before. This message occurs two different ways. First the user can open the document but will receive the message. This is caused by an image that is corrupted. The image appears as a blank box on the document and you can not delete it. I have found out that you can replace it and then delete the image that you replace it with. In the second instance, the user can not open the document. In this case even printing the document will not display all the data on the document. It is truncated at the point of the corrupted image. The only solution in this case is to figure out which field is the cause and delete it. Again, this results in the user losing any data in that field.

Again, my question would be why did this just start happening now?

  1. Data is being duplicated within fields. I didn’t believe this was happening at first, but I have now seen it for myself too many times. The user will edit the document and insert some text and maybe 2 or 3 images into a field. Every time he subsequently edits the document the contents of the entire field are duplicated. I personally have gone in a deleted the duplicate information, double and triple checked that it was gone, printed the document to confirm it was gone, reopened the document to be sure it was gone and then minutes later have the user call back because the data is duplicated again. The only solution I have found for this is to save the information in another document, delete the entire field off of the document, reopen the document and paste the saved information back in. This seems to have worked so far.

A lot of the times, this issue is appearing if #3 above has also been a problem. So, it seems like the corrupt image is doing something to the field to cause it to act funny.

This database contains approximately 640,000 documents and is about 11,500 MB. It is not full text indexed and we do not do transaction logging on it. We run Fixup (-F) and Updall (-R) every morning on this database. Compact (-s 10 -F) is run weekly. Our Network Engineering department is sure this is a Notes application issue possibly caused by the fact that we are using the 7 designer client for an application running in a 6.5 Notes client. They can’t explain why that would matter now after the application has been in production for over 3 years.

I know this is a long post but I wanted to try to get all the details in. Again, we have made no application changes in over a year and these problems just started happening 2 weeks ago. Has anyone had any experience with any of these issues just cropping up unexpectedly?

Any help would be appreciated!

Subject: corrupt images/Save Conflicts

I have reviewed your info and while I have no conclusion by any stretch – my sense is that these are not save conflicts but replication conflicts (they tend to appear quite similar).Opening a document for reading only (your yesterday scenario) and resulting in a conflict is not necessarily bizarre - considering that developers have been know to bury saves in things like post open. You can only know that by looking at the design elements of the form.

Duplicate entries into fields might result from attempts by Notes to merge similar conflict documents.

I’m guessing that you’ve already looked carefully at this, but I’ll throw some ideas up:

  1. Check to see what replica is being used. Are you being failed over? Check the database properties while the database is open to see what server it indicates you are on.

  2. Check the form properties. On the first tab, see what is selected for Conflict handling. You can set it to create no conflicts at all - but this could present different issues - so just be careful if you leave it there.

  3. Create a local replica of the database, disconnect from any servers and then try to simulate creating the conflicts.

  4. 11.5 GB - while this is within Notes specs, I would consider reducing the overall size of the db, perhaps by archiving some docs.

Also, running fixup and updall every morning against an 11.5 GB database must be resource intensive not to mention the time. I’m concerned that this is running while users are accessing the database.

  1. Move all users off the database completely - perhaps on a weekend. Once all users are off, run fix and updall once. Then replicate. Then just leave it for a week and see.

I have some other (more obscure) ideas but perhaps will wait until you try some of these.

Subject: RE: corrupt images/Save Conflicts

Thank you for responding.

Here are my replies to your suggestions:

"Opening a document for reading only (your yesterday scenario) and resulting in a conflict is not necessarily bizarre - considering that developers have been know to bury saves in things like post open. " I know how tricky those developers can be, but since I was the developer on this form I know there is no save buried in the post open. Only two saves in the code on the form - one in the “OK” button which is the @Command([FileSave]) and the other in the print button to save the document only if it is in edit mode before sending it to the print routine.

"Duplicate entries into fields might result from attempts by Notes to merge similar conflict documents. " This is an interesting idea that I hadn’t thought of. But, isn’t it true that conflict handling only works for replication conflicts and not save conflicts. If that is the case, I don’t know where I would be getting replication conflicts from. We only have 2 replicas of the database - the production replica and the disaster recovery replica. Replication is set up on dr to only pull from production. I have searched in vain for a rogue replica of the database and found nothing.

"Check to see what replica is being used. Are you being failed over? Check the database properties while the database is open to see what server it indicates you are on. " As stated above only two replicas exist and the disaster recovery box is set up to not allow user access. They have to be working in the production copy.

“Check the form properties. On the first tab, see what is selected for Conflict handling. You can set it to create no conflicts at all - but this could present different issues - so just be careful if you leave it there.” Conflict handling is set to Merge/No Conflicts - it has been this way since the application went into production in 2005.

“Create a local replica of the database, disconnect from any servers and then try to simulate creating the conflicts.” We are in the process of working in a test copy of the database to try to replicate the problem. No luck so far.

"11.5 GB - while this is within Notes specs, I would consider reducing the overall size of the db, perhaps by archiving some docs. " I have considered this and will go here if I have to. This application is currently being rewritten outside of Notes so I would rather not have to make any major changes if possible. But, if it is decided that the db size is a problem, we will figure out how to archive some of the information.

“Also, running fixup and updall every morning against an 11.5 GB database must be resource intensive not to mention the time. I’m concerned that this is running while users are accessing the database.”

This doesn’t seem to be an issue. They are scheduled to run very early in the morning and FixUp runs about 20 mins and Updall runs about 30 min. They are both complete well before users are accessing the db.

“Move all users off the database completely - perhaps on a weekend. Once all users are off, run fix and updall once. Then replicate. Then just leave it for a week and see.” I might just try creating a new copy of the db this weekend and see what happens next week.

Thank you again for your response. If you have any other suggestions - no matter how obscure - I would be grateful. I hate it when an application breaks after 3 years in production!