Replication, missing records, can't figure out why

HCL Domino Domino Forum
1

Environment:Dom 6.5.4

One Production server

One Cluster server

One Portal server

Portal replication is one way in (reads from Production server) and pulls in only one view.

Cluster accepts read/write from production and vice versa.

No reader/author fields.

Production server is manager in the cluster and portal.

Most stuff work just fine.

We have one collection of docs (*) where only two out of 42 replicate from the production box to the cluster or portal.

Cleared rep histories on all servers and forced push from production to cluster and portal - no change.

Compacted all dbs, re-cleared rep history and re-pushed, no change.

Wrote an agent on the production box to set a junk field to some bogus value just so they’d all be ‘new’. Repushed to cluster and portal - no change…???

Reran the agent on the production box and the cluster pushed it’s two docs back to production and deleted all of the other docs…??? #$!!!

All the other docs are handled the same way and they’re working fine.

I am major stuck - any input would be helpful.

Doug

(*) This is a multi-db doc control system. We have multi-doc procedures that are grouped together using categorization around the procedure number. We copy them from an In Process db to the Released db when they are approved for use. The releasing agents touch and modify each doc sharing a procedure number so the stuff in the Released db should be all new (new rep ids even) at each release event. There is nothing in the db that should limit the replication of these new docs to the cluster and portal servers.

This design is used by multiple product lines (ie many production dbs based on a single template) and only this one seems to be having problems.

I have had one report of a case where the release process only released a subset of docs that should have been released which, in a way, mimics the replication problem I’m seeing within the Released db across replicas.

Sorry for the ramble and appreciate any pointers for things to check.

2

Subject: More data

In this system, docs are copied from the In Process db to the Released db. During this process, each and every doc has several fields modified so. Additionally, because they are copied to the Released db, they are new docs. It would be my understanding that they should all just replicate because there are no replicas of these new docs.

The replication process for this problematic subset of docs is replicating only docs where, in the In Process db, the user sets a ‘new’ flag to indicate that the content of the doc has changed between versions.

If I run a routine that allows me to make a copy of a set of doc but with a modified part number field and then release this new set of docs, replication works fine.

So, what they system seems to be doing is

Treating changes made by the release code as no change to the doc.

Treating the new docs created during release as not new (ie as if they are replicas from the In Process db).

Then replicating only those docs flagged as ‘new’ as if the real time touching by the user is different from that done by the server during release.

It’s almost like the ‘ping/pong agent’ problem is back…(anybody remember when you had to touch all docs with two different agents so the agent would run on all docs?)

Guess I’ll hit this one with a big hammer and just force some new field data into all docs in the In Process db and then try a release.

Doug

ps - I realize this all sounds like crap. This has been difficult to describe. I’ve tried to include enough information about the app framwork to be helpful without going into too much extraneous detail. If anybody needs more information, let me know and I’ll do my best to provide what you need.

Much appreciation to anyone who responds.

3

Subject: Replication, missing records, can’t figure out why

I would suggest the following:

  • check not only rep. history, but the Advanced Replication settings, specially the date limit field (‘Do not replicate incoming documents modified after:’ )

  • keep in mind that replication is based on date/time of last change for a document.

  • having this, verify also time/date on servers has not been changed accidentally, this usually screws replications in Domino environment.

4

Subject: RE: Replication, missing records, can’t figure out why

Radu,

Thanks.

I would suggest the following:

  • check not only rep. history, but the Advanced Replication settings, specially the date limit field (‘Do not replicate incoming documents modified after:’ )

Nothing is checked here, there are no restrictions based on date. The only restriction is on the portal side and that is to replicate docs in one view - and all docs of interest are in this view.

  • keep in mind that replication is based on date/time of last

change for a document.

At each ‘release’, docs are copied from the In Process db to the Released db - they are brand new never been seen before docs so they MUST be new to the replicator. In fact, all the other docs handled exactly the same way DO replicate. (more below).

  • having this, verify also time/date on servers has not been changed accidentally, this usually screws replications in Domino environment.

I have no way to verify if this has been changed. When I check the date/time right now, it’s correct and as far as I know it’s not been screwed with.

More info.

In the In Process db, I ran the following agent on all docs that were not replicating:

FIELD Bogus := “Ping”

I then had the system users re-release the docs to the released db (release sets some fields on the docs in In Process, then does a LS CopyToDatabase). POOF, Magic, everything replicates to the cluster and portal.

EVERY time a set of docs is released, they are new - brand new, never been seen ever before docs. The docs in question are just like every other doc. There is no reason they shouldn’t replicate…yet they don’t until I run a process in a totally different db that is essentially identical to the stuff that happens when the docs are processed during a release event (ie field data are set). The only difference is that the agent I ran added a field to the docs rather than set fields that are part of the form design.

Well, guess it’s off to audit the dbs and make sure this isn’t happening elsewhere…

Thanks again for the reply. If anything I posted here sparks any other ideas, I’m all ears!

Doug

5

Subject: Replication, … possible reason and fix

I had a chance to talk with Rich Schwartz about this problem and he suggested that the way I created new docs in the Released db (CopyToDatabase) was somehow causing the docs act like pseudo-replicas rather than new docs. Why this would affect only a subset of docs is unclear, but that was how the system was acting. He further suggests that this may be a server level defect in 6.5.4 rather than a coding error on my part.

He suggested I switch to Db.CreateDoc followed by CopyAllItems. In this way the new doc is absolutely positively guaranteed to have a new ID as far as the replicator is concerned.

I made the switch and released almost 3000 docs in real time, every one of them replicated to both the cluster and portal with no problem.

I pushed the design out to the db having problems and had the team re-release the problem docs. They also replicated just fine.

So, the good news is that it seems I have found a way to release docs and have them replicate properly.

The bad news is that I have no independent confirmation that there is a server level bug and that my code works around the defect. The problem is low frequency and not predicatble. In my testing I was absolutely unable to re-create the problem. There are only a small subset of docs that exhibit the non-replication error so I have to imagine trying to figure out if there actually is a bug in the server code would be difficult.

For now, I plan to monitor doc counts periodically and make sure they’re remaining the same across replicas.

If anybody is aware of a defect like this (and a confirmed workaround), let me know.

Doug