Unscheduled Replications showing in Replication History - any ideas?

Hi,

A fellow administrator and I have been asked to investigate the case of the mysteriously replicating servers. There is no log information to provide evidence that this is happening at a server level. What we are seeing is an entry in the replication log to show that 2 servers have replicated. These servers have no connection documents in common to initiate scheduled replication nor are they clustered. We are assured that this is happening frequently and is not being initiated from a notes client by a notes administrator (either deliberately or inadvertently by scheduled replication).

Does anyone have ANY ideas as to why we might be seeing regular server to server replication in the replication history without anything appearing in the server log ?

VPN? NNN? Any undocumented features that you are aware of???

Karen

Subject: Unscheduled Replications showing in Replication History - any ideas?

Did you ever discover the source of the problem? We’re experiencing the same thing here…

Subject: Solution

Hi Steve,

Yes we did find the cause of this. Our servers are now Domino R7 and there is a feature of R7 (that I was not aware of and that does not appear to be documented in the Admin Help or the Release Notes) called ‘replication triangulation’ … that you can read about in the technote below. Basically Domino R7 servers keep a record of each server that the db has replicated with. So if Server A replicates with ServerB and ServerB then replicates with ServerC, the replication history for ServerC is added to the replication history on ServerA via ServerB. If you view the replication history with an R7 client, you can see that the ‘‘indirect’’ replication history entries are marked with an asterisk. You cannot see this with an R6 client. Here is the technote that explains it (properly):

http://www-1.ibm.com/support/docview.wss?rs=0&q1=replication+history&uid=swg21270104&loc=en_GB&cs=utf-8&cc=uk&lang=en

There was a second technote about this, that I still have the URL for but I can’t now find on IBM Web site. Here is the URL and the text of that technote, that I had kept for posterity :o)

Why does an asterisk appear in the Replication History (when viewed with an R7 + client)

http://www-1.ibm.com/support/docview.wss?uid=swg21234661

Problem

An asterisk appears next to certain entries in the Action column of the Replication History dialog box. What does the asterisk denote?

Solution

The asterisk is related to a replication performance improvement, which was introduced in Lotus® Domino® 7.0. Let’s explain the improvement with an example.

Prior to Domino 7.0

Bob uses a local replica of his mail file. He normally replicates with only his home mail server, which has excellent reliability - it has stayed up for two years without a problem. But one day, there is a power failure. So, Bob’s Notes® client fails over and locates another replica of his mail file on a backup server. When scheduled replication begins, the replication history of his database shows that he has not replicated with the backup server for years, so the replication code must check all his mail for the last few years to determine which notes to replicate. This takes awhile!

Beginning in Domino 7.0

Bob replicates with his home mail server the day before the power failure. When replication is complete, the scheduled replication code now records replication history (i.e., that Bob has replicated with his home mail server.) The new code sees that in the replication history of Bob’s home server, the server has replicated successfully with his backup server, and that it was a complete replication (i.e., not selective, or interrupted in any way.)

So, it infers that if the home server replicated completely with the backup server, then Bob’s client can mark its history as having successfully replicated with the backup server. Now, when the power goes down and Bob’s client replicates with the backup server, the replication history shows that only a few notes must be examined, and replication completes quickly.

The asterisk in the replication history indicates that replication was complete (i.e., no selective replication formula and was not interrupted in any way.) This is the indicator used to allow another replication to update its replication history with multiple servers, even though only one replication was done.


Subject: RE: Solution

OMG, thank you so much! I was running out of hair to pull out in my attempts to explain what was going on.

Subject: So, does anyone know how to switch replication triangulation off?

While this is a nice feature, I’d myself would like to turn it off. I’ve enough trouble explaining basic Domino concepts to newbies, let alone confusing them more with this sort of stuff.

Anyone know how to do this?

Subject: RE: So, does anyone know how to switch replication triangulation off?

Lotus® Domino® 7 introduced a new feature called Replication Triangulation. The purpose of this new feature is to make each server aware of every other server which maintains a replica of the same database, and which has had a successful replication. In a large environment, the number of replication history events to maintain can cause a significant performance impact to the Domino server.

Symptom

After upgrading to Domino 7 or 8 from any Domino 6 release, the REPLICA task consumes more CPU.

A review of the replication history for the names.nsf on any server in the domain shows successful replication events to servers which this server has not directly contacted.

Example:

Names.nsf on Server A, is replicated to Server B. Server B replicates names.nsf with Server C.

The next time Server A and B replicate names.nsf, Server A will show successful replication with Server C, although Server A never replicated directly with Server C.

If a customer has hundreds or thousands of servers in the Domain, the replication history can become very long.

The entire replication history must be checked and then replicated prior to document replication, as a result, replication events which previously resulted in no documents replicated now may have hundreds or thousands of events.

Tprofs, or other system trace tools which show Domino activity will show very high call events to

NSFDbWriteReplHistory. In the case of the customer reporting this issue, CPU requirements increased 400% as a result of the replication overhead related to replicating the history for 1700 servers in the Domain.

Cause

Maintaining replication triangulation history for databases which exist on hundreds or thousands of servers is too expensive

Resolving the problem

You can disable replication triangulation with the following notes.ini parameters, which are available in Domino 8.01 and later:

NSF_REPLHIST_NO_TRI=1

REPL_NO_WS_TRI_HIST=1

REPL_NO_REMOTE_TRI_HIST=1

SPR MNIR72EJHH documents this issue.

Subject: Unscheduled Replications showing in Replication History - any ideas?

Issue a “show sched” command at the console of the initiating (source) server. Does the replication show up as scheduled? If so, perhaps a rogue or hidden connection doc exists (could be a rep conflict doc). Just a thought.