Hi Lotus fellows,
meanwhile 4 of our customers have a strange problem with our application when using it in a clustered Domino 8 or 8.5 environment.
After updating Domino from < 8 to 8.x the following happens (example for cluster of 2 Domino servers):
At first everything seems to be fine. After a while (a few days or several weeks) some documents in our application are invisible for the admins on one of the servers.
Reason: The $Readers field has disappeared in those documents! (In our application almost all forms are readable for members of [Admin], so this is part of the design-elements!)
The corresponding document on the other server still has that $Readers field with the correct values. But from this moment, this document pair is not being replicated any more! There are no error messages by replicator.
So far known:
-
If this phenomenon starts occuring, more and more documents are affected, which means, the error speed seems to increase.
-
So far it’s only known in cluster environments with Domino 8.x
-
The same application was running fine for several years now in R6 and R7 cluster environments.
-
Only one of our forms is affected (all other forms with same $Readers fields are never affected)
-
It appears on ODS 43, 48 and 51
-
After replacing the DB replica with the corrupted documents, the application is running fine again for several days or even weeks, then the problems start again.
Any suggestions out hints?
Subject: LocalDomainServers access
Is LocalDomainServers in the $Readers field, or another field with Reader access? If there are readers fields set on a document, the server also needs to be specified as an allowed reader…or exactly what you describe can happen…it only exists on the server where it was created and never makes it to any other server because they aren’t authorized to read it.
Subject: LocalDomainServers have access
The ACL-group LocalDomainServers ist not directly listed in the $Readers fields but the group has the ACL-role [Admin] and this role is in the $Readers field. And that methode works fine for several years now.And I want point out, that the documents are on both sides (on both servers) but sporadically on one server the field $Readers gets lost. (and only in R8 environments)
The affected documents are being replicated at first and also the first changes in normal data fields are being replicated (thus the server does have access), but then in one of a thousand cases on one server the field $Readers is being “kicked” out by the Domino! The document itself still exists (on both sides).
Subject: support case?
I’ve not seen anything like that, and can’t think of anything that would cause it. Other than, possibly, replication conflicts with the field itself…but even then…
If this is happening regularly enough that it is somewhat reproducable, I would encourage you to open a support case with IBM.
Subject: most likely a support case
I agree, it seems to be a bug in the cluster replicator of Domino 8.xI am currently collecting all usefull data from the affected customers. For example one of them has switched from cluster replication to “normal” replication and since then the problem did not occur any more so far. But because sometimes even with cluster replication the problem did not occur for weeks, it is only a consideration, that “normal” replication in 8.x doesn’t have the bug.
Sadly the occurance of that problem still cannot be forced (in a small test environment), instead you need a “living” installation with hundreds of users and thousands of document changes every day before (possibly) one document is affected. => Bad conditions for debugging.