Streaming Cluster Replication misses documents

hcl-bot · November 23, 2009, 7:36am

I have a customer who runs a Domino cluster on two Linux boxes. The cluster hosts a number of department (shared) mailboxes.

A number of these mailboxes use a custom database template to provide filing, archiving etc.

We have noticed a problem with SCR, where sent messages aren’t replicated between servers, causing the database replicas to become out-of-sync.

Normal scheduled replication does not synchronize the views, unless we clear the replication history first. We have even experienced a situation where clearing the history is insufficient to provoke synchronization.

What’s strange is that the problem seems to be isolated to outgoing messages.

When a user sends a message, the message is saved to the database in order to gain programmatic access to the body field and add auto-generated information to it before the message is sent. This script is pure Lotusscipt using no undocumented features whatsoever.

If we disable SCR and revert to normal cluster replication, the problem goes away for all databases.

It makes no difference whether the database contains only a handful of documents or 150000+ documents - the behavior is the same.

Because the databases behave differently depending on whether we run with SCR or non-SCR, we opened a ticket with IBM.

After much communication back and forth (over several months) IBM finally came back and said that, since it’s a custom application, they will not pursue the issue further.

I am tired of dealing with IBM technical support, who once again disappoints, so my question now is:

Has anyone seen anything remotely similar to this problem and, if so, have you managed to resolve the issue?

Thank you very much in advance,

Alex Rasmussen

hcl-bot · November 26, 2009, 1:00pm

Subject: Bump… No one?

hcl-bot · November 26, 2009, 2:09pm

Subject: No such troubles…

Regardless of where the true fault of the issue lies, there’s no question that your save-n-update procedure causes this problem to manifest itself…right?

The solution then, is to rewrite your process.

So…first things first. Explain why Call Source.FieldAppendText( “Body” , message ) doesn’t cut it.

hcl-bot · December 7, 2009, 1:13am

Subject: First things first…

I need access to the back end body field, because I need to do more editing than simply appending text to the end of the body field.

And rewriting the code IMHO, is simply curing the symptoms.

But what scares me most, is IBM’s refusal to acknowledge the fact that there is a potential problem in the server code, when an application behaves differently depending on whether you use SCR or not.

hcl-bot · December 7, 2009, 7:30am

Subject: Got it…

“I need access to the back end body field, because I need to do more editing than simply appending text to the end of the body field.”

So you need to save the document. It’s been my experience that if I want to work with a new document immediately after saving, I need to re-open the document. If it’s in the UI, then I’ll reopen the doc for the user (they don’t even notice the doc was ever closed.) If it’s in the back-end, I need to get the document from a view, and then work on it. This is due to some odd Lotus Notes thing, where the server doesn’t seem to acknowledge that a saved document is opened for edit unless it’s been opened from a view. It has to do with the way documents are referenced internally. So I would try setting your NotesDocument variable to Nothing after saving, and then get the just-saved doc from a view to continue working on it.

"And rewriting the code IMHO, is simply curing the symptoms. "

That’s absolutely true. But it’s been my experience…and especially when one is responsible for resolving a problem…that sometimes you’ll just never know the cause of some problems. But that doesn’t absolve you from having to fix it. It sounds like in this case, you just have to fix it.

“But what scares me most, is IBM’s refusal to acknowledge the fact that there is a potential problem in the server code, when an application behaves differently depending on whether you use SCR or not.”

That’s because you’re not running Domino on a System z

hcl-bot · December 8, 2009, 1:11am

Subject: Yup…

“So you need to save the document. It’s been my experience that if I want to work with a new document immediately after saving, I need to re-open the document. If it’s in the UI, then I’ll reopen the doc for the user (they don’t even notice the doc was ever closed.) If it’s in the back-end, I need to get the document from a view, and then work on it. This is due to some odd Lotus Notes thing, where the server doesn’t seem to acknowledge that a saved document is opened for edit unless it’s been opened from a view. It has to do with the way documents are referenced internally. So I would try setting your NotesDocument variable to Nothing after saving, and then get the just-saved doc from a view to continue working on it.”

Exactly, although I don’t even need to reopen the UI document, because the user already clicked send. I suspect a timing issue due to the proximity of the two save events. We’re going to try a live test with a built-in delay in one of the databases, while keeping the other databases as a control group.

“That’s absolutely true. But it’s been my experience…and especially when one is responsible for resolving a problem…that sometimes you’ll just never know the cause of some problems. But that doesn’t absolve you from having to fix it. It sounds like in this case, you just have to fix it.”

Very true.

“That’s because you’re not running Domino on a System z :P”

LOL Please forgive my transgression

hcl-bot · December 8, 2009, 8:36am

Subject: How are you getting the doc?

“Exactly, although I don’t even need to reopen the UI document, because the user already clicked send.”

Okay…so you’re getting the doc from the backend before working on it…exactly how are you doing that?

hcl-bot · December 9, 2009, 7:32am

Subject: This is the code…

The following code is the one in question. The databases all have document locking disabled.

Sub SaveAndSend(Byval printout As Boolean)

On Error Goto errorProc

Dim ws As New NotesUIWorkspace

Dim session As New NotesSession

Dim db As NotesDatabase

Dim uidoc As NotesUIDocument

Dim doc As NotesDocument

Dim profile As NotesDocument

Dim engine As Notesdocument

Dim body As NotesRichTextItem

Dim newBody As Notesrichtextitem

Dim rtRange As NotesRichTextRange

Dim rtNav As NotesRichTextNavigator

Dim header As Notesrichtextitem

Dim signature As NotesRichTextItem

Dim fontstyle As Notesrichtextstyle

Dim unid As String

Dim includeText As notesitem



Set uidoc=ws.CurrentDocument

Set db=session.CurrentDatabase

Set engine=db.GetProfileDocument("GMEngine")

Set includeText = engine.GetFirstItem("IncludeText")



'First, check to see if the message contains at least one recipient

If uidoc.FieldGetText("SendTo") = "" And uidoc.FieldGetText("CopyTo") = "" And uidoc.FieldGetText("BlindCopyTo") = "" Then

	Messagebox "Error: You must specify at least one recipient before sending.", 16, "Error."

	Exit Sub

End If





'Get the next reference number

Set profile=db.Getprofiledocument("User Profile", session.Username)

profile.MsgRef=profile.MsgRef(0)+1

Call profile.Save(True, True)



Call uidoc.Fieldsettext("SaveOptions", "1")

Call uidoc.Save()

Set doc=uidoc.Document

Call uidoc.Close()







'Set the ref no.

doc.Prefix=profile.Prefix(0)

doc.MsgRef=profile.MsgRef(0)



'Add header to the body

'Position to the beginning of the document before the insert

Set header=engine.Getfirstitem("Header")

Set body=doc.Getfirstitem("Body")

Set rtNav=body.CreateNavigator



'Set the fontstyle

Set fontstyle=session.Createrichtextstyle()

Select Case profile.BodyFont(0) 

Case "Courier9":

	fontstyle.Fontsize=9

	fontstyle.NotesFont=body.GetNotesFont("Courier New", True)

Case "Courier10":

	fontstyle.Fontsize=10

	fontstyle.NotesFont=body.GetNotesFont("Courier New", True)

Case "Verdana9":

	fontstyle.Fontsize=9

	fontstyle.NotesFont=body.GetNotesFont("Verdana", True)

Case "Courier10":

	fontstyle.Fontsize=10

	fontstyle.NotesFont=body.GetNotesFont("Verdana", True)

End Select



'If the message body contains text, we insert before the text

If rtNav.FindFirstElement(RTELEM_TYPE_TEXTPARAGRAPH) Then		

	Set rtRange=body.CreateRange

	Call rtRange.SetBegin(rtNav)

	Call body.BeginInsert(rtRange)

End If



If includetext.Contains("0") Then

	Call body.AppendStyle(fontstyle)

	Call body.AppendText(header.Text)

	Call body.AppendStyle(fontstyle)

	Call body.Addnewline(1, True)		

End If



If includetext.Contains("1") Then

	Call body.AppendStyle(fontstyle)

	Call body.Appendtext("Ref. No.: " & profile.Prefix(0) & "-" & Right$("000000" & Cstr(profile.MsgRef(0)), 6))

	Call body.Addnewline(1, True)

End If



If includetext.Contains("2") Then

	Call body.AppendStyle(fontstyle)

	Call body.Appendtext("Date    : " & Now)

	Call body.Addnewline(1, True)

End If



If includetext.Contains("3") Then

	Call body.AppendStyle(fontstyle)

	Call body.Appendtext("Subject : " & doc.Subject(0))

	Call body.Addnewline(1, True)

End If



Call body.Addnewline(1, True)	



Call body.EndInsert



If includetext.Contains("4") Then

	Set header=engine.Getfirstitem("Footer")		

	Call body.Addnewline(2, True)

	Call body.AppendStyle(fontstyle)

	Call body.AppendText(header.Text)

	Call body.Addnewline(1, True)		

End If



'Insert filing information for auto-file replies

'Make it white on white to hide

If engine.FileTags(0)<>"0" Then

	fontstyle.NotesColor = 1

	Call body.AppendStyle(fontstyle)

	Call body.Addnewline(2, True)

	Call body.AppendText("//")

	Call body.Addnewline(1, False)

	Call body.AppendText("//FILING INFORMATION - DO NOT MODIFY OR DELETE//UNID//" & doc.UniversalID & "//")

	Call body.Addnewline(1, False)	

	Call body.AppendText("//")

End If



'Insert footer

Call body.Addnewline(2, True)

Call body.AppendRTItem(engine.GetFirstItem("Footer"))	



'Handle icons and links for forward/reply

If doc.HasItem("ReplyRef") Or doc.HasItem("ForwardRef") Then

	If doc.HasItem("ReplyRef") Then

		Set refDoc = db.GetDocumentByUNID(doc.ReplyRef(0))

	Else

		Set refDoc = db.GetDocumentByUNID(doc.ForwardRef(0))

	End If

	If Not refDoc Is Nothing Then

		lockOk = True

		If db.IsDocumentLockingEnabled Then

			lockOk = refDoc.Lock()

		End If

		If lockOk Then

			If refDoc.HasItem("$RespondedTo") Then				

				If doc.HasItem("ReplyRef") Then

					If refDoc.GetItemValue("$RespondedTo") <> "2" Then

						Call refDoc.ReplaceItemValue("$RespondedTo", "1")

					Else

						Call refDoc.ReplaceItemValue("$RespondedTo", "3")

					End If

				Else

					If refDoc.GetItemValue("$RespondedTo") <> "1" Then

						Call refDoc.ReplaceItemValue("$RespondedTo", "2")

					Else

						Call refDoc.ReplaceItemValue("$RespondedTo", "3")

					End If

				End If

			Else

				If doc.HasItem("ReplyRef") Then

					Call refDoc.ReplaceItemValue("$RespondedTo", "1")

				Else

					Call refDoc.ReplaceItemValue("$RespondedTo", "2")

				End If

			End If

			Call refDoc.Save(False, False)

			If db.IsDocumentLockingEnabled Then

				Call	refDoc.UnLock()

			End If

		End If

	End If

End If



'Remove flags

doc.Flags=""

doc.ProcessedFlags = ""



'Prepare other settings

doc.Principal=engine.UserName(0)

doc.SMTPOriginator=engine.SMTPOriginator(0)

doc.PostedDate = Now

doc.MsgStatus="S"

doc.SaveOptions="1"

doc.ServerTime = Now



'Strip any form modifiers for the body font

strMod = Instr(doc.Form(0), "_") 

If strMod > 0 Then

	doc.Form = Left(doc.Form(0), strMod-1)

End If





Call doc.Save(True, True)



doc.SaveOptions="0"



doc.Subject = doc.Subject(0) & "  //" & doc.Prefix(0) & "-" & Right$("000000" & Cstr(doc.MsgRef(0)), 6) & "//" 



Call doc.Send(False)



If db.IsDocumentLockingEnabled Then	

	Call doc.UnLock()

End If



'New In 7.2.5 - handle optional filing by subject code string

Call SubjectFiling(doc)



'Handle icons and links for forward/reply

If doc.HasItem("ReplyRef") Or doc.HasItem("ForwardRef") Then

	If doc.HasItem("ReplyRef") Then

		Set refDoc = db.GetDocumentByUNID(doc.ReplyRef(0))

	Else

		Set refDoc = db.GetDocumentByUNID(doc.ForwardRef(0))

	End If

	If Not refDoc Is Nothing Then

		lockOk = True

		If db.IsDocumentLockingEnabled Then

			lockOk = refDoc.Lock()

		End If

		If lockOk Then

			If doc.HasItem("ReplyRef") Then

				If refDoc.GetItemValue("$RespondedTo") <> "2" Then

					Call refDoc.ReplaceItemValue("$RespondedTo", "1")

				Else

					Call refDoc.ReplaceItemValue("$RespondedTo", "3")

				End If

			Else

				If refDoc.GetItemValue("$RespondedTo") <> "1" Then

					Call refDoc.ReplaceItemValue("$RespondedTo", "2")

				Else

					Call refDoc.ReplaceItemValue("$RespondedTo", "3")

				End If

			End If

			Call refDoc.Save(False, False)

			If db.IsDocumentLockingEnabled Then

				Call	refDoc.UnLock()

			End If

		End If

	End If

End If	





Exit Sub

errorProc:

Print "SendAPI Error (SaveAndSend): " & Error & " in line " & Cstr(Erl)

Resume Next

End Sub

hcl-bot · December 9, 2009, 8:15am

Subject: I don’t see where you’re retrieving the doc from a view.

I see this…

Call uidoc.Save()

Set doc=uidoc.Document

Call uidoc.Close()

And then you start working on doc. If this was a new document that’s being saved, it would be my opinion that this code is problematic.

I’d suggest creating a view by UNID, then looking up the document by UNID before working on it.

Dim UNID as String

Dim UNIDView as NotesView



Call uidoc.Save()

UNID = uidoc.Document.UniversalID

Call uidoc.Close()

Set UNIDView = db.GetView( "UNIDView" )

Set doc = UNIDView.GetDocumentByKey( UNID )

And then lock the document if using doc locking.

It may or may not help, but I find this process reduces problem for me.

hcl-bot · December 7, 2009, 12:22pm

Subject: What??!??!

After much communication back and forth (over several months) IBM finally came back and said that, since it’s a custom application, they will not pursue the issue further.

I would run that one up the flagpole in IBM.

hcl-bot · December 7, 2009, 12:57pm

Subject: Well, custom apps are such a rare thing in Notes…

just kidding…

No, I got the impression that IBM likely meant customizations to the mail template. The mail template is unbelievably complex (for what it does) and I’d guess that any update to the save process would be severely frowned upon by IBM.

hcl-bot · December 8, 2009, 1:23am

Subject: But…

This application is not a customization of the mail template.

It’s a 100% custom application written from the ground up to eliminate all the fat from the standard mail template and focus 100% on sending, receiving and filing mails for departments with large database (250000+ messages) requirements. As a result, we have gone from around 100 views to 29 views (of which only 20 index messages) and no folders - with a huge improvement in performance.

The piece of code in question is around 230 lines in total - most of which is RTF formatting, so you’d think IBM could figure it out (and yes, they asked for the code and sat on it for about a month and a half, before they came back and said “sorry, custom application”).

hcl-bot · December 8, 2009, 6:56am

Subject: Ah. Okay. I was presuming.

hcl-bot · December 9, 2009, 1:36pm

Subject: Do please keep us informed.

If you can isolate it down and send it to IBM, they should be able to fix it. “Proximity” of save events shouldn’t matter, if so then streaming replication is definitely busted.

It is a fairly new feature, so it could very well have a bug or two. Please keep us informed, SCR is awesome and I’d like to know how this problem gets resolved.

hcl-bot · March 31, 2010, 9:09am

Subject: Similar problem

Server environment : Win

Yes we have a similar problem with streaming cluster replication. It also regards saved copies of sent mails. In my case works like this:

I create the background documents with LS.
save the background document.
Open the newly saved document in edit mode in UI.
Insert one empty space in the uidocument to get the IsDirty-flag
remove the background document
…user does his stuff…
document/mail is sent and saved.

This method has worked forever and ever with SCR some (not all) of the documents are not replicated.