How to Search 1000 Mailfiles

All, I’m an experienced Java developer trying to learn the notes environment–so this may be easy. If it’s in the FAQ or addressed elsewhere, please include a link–I had no success searching.

We are trying to develop a means using Java to detect documents modified in the last 10 minutes across multiple (> 1000) users’ mail files stored on domino server(s). I imagine that I can use the CORBA classes and open up each mailfile for all the users I’m interested in, then use Database.getModifiedDocuments() to find recently updated documents. However, I don’t believe this will scale well if I want to poll every ten minutes (or even every hour :). I wonder if there’s a cleaner alternative, such as an agent that can be notified and make note of recently updated note IDs (I’m not sure how I can write one agent and get it applied to multiple databases) Is there some other sort of option available in the Observer-Listener pattern?

Subject: How to Search 1000 Mailfiles

In Notes agents, there is an option to run on “All New and Modified documents”. In this case, you would need to put the agent in each mail file. Assuming the mail files inherit their design from a template, this shouldn’t be a big deal - you just create the agent in the design template and the mail files are updated with the agent overnight. You would then set the agent to run every 10 minutes (or whatever), and since it only runs on new and modified docs, it won’t have to loop through all the docs in each mail db. Note: the first time (or every time you save the agent), it will in fact loop through all the docs in each database, so best to build in the logic to filter the number of docs that it “finds” so you don’t accidentally run on all documents. But, I guess a big question is: what are you going to do with this data? Post it somewhere? Send a newsletter to the owner of the mail file? Maybe it doesn’t matter, but it could help develop a better solution in case you really only want to run one agent instead of one per mail file.John

Subject: RE: How to Search 1000 Mailfiles

John,

The goal is to find address book contacts that have been updated. We would then copy their data out to an oracle table (somehow) so we can expose fresh contact data via our custom web application. Thanks for the tip about putting the agent in the template. It might be OK for a new or newly updated agent to loop thru all docs, just so I don’t have to do it every time (every 10 minutes). Do you think that looping thru all docs even once in this scheme would be bad?

I’m not sure yet how I can get the data out to oracle from an agent. My current thought is create a new database (e.g. “InterestingContacts”), each document in it would contain a mailfile reference and noteID. I could get the mailfile and noteID via CORBA, delete the doc from “InterestingContacts”, then open the actual mailfile via CORBA and fetch the document via noteID, and update oracle accordingly. I’m assuming that the number of documents in “InterestingContacts” will be small at any given time.

Sound hackish? I’m open to suggestions. Of course, it would be better just to write directly via JDBC to oracle from the agent.

Thanks,

Chad

Subject: RE: How to Search 1000 Mailfiles

Hi Chad,Just curious, the address books are on the user’s local disk, no? Usually contacts are stored in the local names.nsf file. If a db is not on the server, it means the server won’t be able to run on the data on a scheduled basis. It can’t access the files. Unless maybe it’s on a shared drive or something.

If they are on the server, then indeed, run a scheduled agent. And if you’re more comfortable in Java, no problem. Like you said, just write directly to the Oracle database from the Notes agent. Not sure I get the part about a new notes db and then writing to Oracle. Might as well skip the middle part and write to Oracle directly. Unless I’m missing something…

Hope that helps a bit

John

Subject: Product-quality approach; contacts location

I’m not entirely clear whether you’re writing a component for a commercial product or coding something for a specific environment. The reason that’s important is that there are some trade-offs that might be acceptable in a controlled environment that won’t wash for a boxed product. Let’s assume you do want to write code for a boxed product.

The next thing to look at is where these contacts documents you’re interested in might reside (and how you might be able to tell if they’ve been modified).

Traditionally Notes has stored contacts in the client’s local names.nsf, to which server code will not have access. More recently, several server-side contacts stores have emerged.

In Domino 6.5, the concept of Notes roaming was introduced, and one side effect of this is that a replica of each user’s names.nsf is stored on a Domino server. The downside is that the majority of companies have not chosen to implement Notes roaming.

Around the same time, the Domino Web Access mail template added a capability for users to sync contacts between the local names.nsf and the server-based mail database. There are two drawbacks here: users needs to be set up with the DWA template, and need to manually sync their contacts. Even so, this would be a more generally acceptable approach than requiring roaming.

Finally, in Notes and Domino 8, syncing contacts to the mail file has just about become part of the replication process. However, it still needs to be enabled for users, although this can be achieved via policies (some bugs here).

Now that we know where the contacts might be, we want to look at our options for detecting when they’ve changed. In roughly decreasing order of acceptability to enterprise environments the options appear to be:

  • Find a hook from the contact update process that can be used to trigger your code running in a separate database or via a Java program installed on the server. The key thing here is to avoid modifications to the mail and address book templates. Sounds good, but this hook may not exist.

  • An agent in a separate database or a stand-alone Java program installed on the server that scans mail files (or roaming address books) for the kind of changes you want to track.

  • Event-based code in the mail database template that hooks into an event triggered by a contact update and writes a UNID to a central database. Process new notifications in the central database either with an agent or a standalone Java program. This is like the first option, but accepts the need to modify the mail template.

  • PostSave event-based code in the personal address book template that sends some notification to a central database. This approach has two big flaws: it requires code running in every user’s names.nsf, and that code has to find a way of conveying the update data from the client to your central app.

  • A scanning agent that runs on schedule in each mail database. This one isn’t likely to fly. There won’t be many server administrators who would sanction a scheduled agent running every 10 mins in each of 1000 mail files.

Perhaps, we can get an idea of how best to approach this by looking at how others have handled it. Blackberry Enterprise Server 4.0+ can sync users’ contacts from the Domino server so long as they either use the DWA contact sync feature or are set up for roaming. As far as I remember, this is handled by the BlackBerry Messaging Agent, which is a C++ or Java executable running on the server. I imagine this executable does the work of scanning the mail and address book databases for changes on a frequent schedule (~10 mins); it doesn’t appear to be event-driven.

Hope that gives you some ideas about how to approach this!

Rupert Clayton

Chicago

Subject: How to Search 1000 Mailfiles

Given your requirement I would simply add some code into the postsave event of the form(s) your interested in working with into the mail template. The postcode would write to a central(single) database the UNID of changed documents. Finally your java code could then spin through the unprocessed UNIDs in documents and do whatever you desire and finally mark them processed.