I have a Domino 7.0.3FP1 Server with over 40,000 databases. The majority of these database have 2-3 scheduled agents. As of recently, I’ve noticed the following entry in my log file:
11/29/2011 09:11:07 AM AMgr: Finished loading all tasks. Elapsed time 538 minutes and 53 seconds
Looking back over the past week, I can see it takes Agent Manager on average 8-9 hours to update. Currently, the Refresh agent cache process in the server document is scheduled to begin at 12;00AM (midnight) which means it is still running in the morning.
Because most of the databases are the same template, I’ve talked to my developers about centralizing the same agents to run across all the databases. This would greatly reduce the number of scheduled agents from probably over 100,000 to 10-20 each night. Is it the number of scheduled agents which makes this process take so long, or is it also the number of databases? Is it possible I need to split this server into 2 with 20,000 databases on each?
Finally, does this Agent Manager scheduling process run more efficiently on Domino 8.5? Would I see a noticeable improvement if I moved to Windows 2008 64-bit running Domino 8.5 64-bit?
Subject: Agent Manager Refresh agent cache process takes too long
40,000? Wow! Totally off topic, but since most are based on one template, what exactly do the apps do?
Actually that’s not totally off topic. I guess I’m wondering if somewhere along the line you made a design/topology choice and are now paying the price. Could you rethink your design and go from 40K dbs to, 1 or 10?
It’s hard to say what’s taking all the time. Are these big dbs and the agents are processing tons of stuff in each db? If you look at the agent run times in one db, how long to they take to complete? If the individual run times are long, is there a problem with the agent design (using big arrays when a list would work instead and run incredibly faster) or is it that you’re just cranking through a lot of stuff? There are some great articles by Andre Guirard on agent efficiencies that you might want to find and read.
OTOH, 40K dbs is just a lot of dbs and you may be taxing your server’s capacity. Is the server properly provisioned to support that kind of load? More ram? Faster CPU, faster drives all could help. Are your dbs actually on the server or are you using NAS via directory links?
If this is really killing you, you might want to contract with a consultant to spend some time helping you through the analysis process.
Subject: RE: Agent Manager Refresh agent cache process takes too long
Thanks for the response.
Yes, 40,000! My zero key is not stuck. The average size is about 125MB which results in about 5TB of total data.
We have a large application which consists of a central database (catalog) which references client engagements (lots of databases). We could have multiple engagements for each client, so they add up. Technically, we have about 6 different engagement templates, but they spawned from one originally. They evolved slightly over the years.
The agents themselves don’t take long to run, it’s the daily process Domino runs to rebuild the agent list. If you look at the server document (Server Tasks | Agent Manager), the Refresh agent cache runs daily at 12:00AM (midnight). When this process runs, Domino searches every database for scheduled agents. It is this process which takes over 8 hours and runs into the business day. In some ways, I’m not surprised. Agent Manager has to scan over 83 dbs/s to search 40,000dbs in that time. Since it is building a schedule, I’m hesitant to set it to run at 10PM (not sure how spanning days will work for scheduled agents). It is also preventing agents from running until it finishes (i.e. no agents will run between 12:00AM and 8:00AM - not good). If I could, I’d rather have Domino rebuild the agent schedule on the weekends only.
I was hoping someone might know how (maybe through an INI setting) to tell Domino not to Refresh agent cache daily, but instead do it weekly.
I could call support, but I’m not sure what answer I would get since it’s still Domino 7.
Subject: RE: Agent Manager Refresh agent cache process takes too long
You may need to have the client info living in 40,000 DBs (give or take), but surely you can consolidate the agents into a more compact number of DBs (perhaps one per template variant or per file system directory). Even if the consolidation is just in the form of one or more “Agent Runner” agents, it means that you can take all of the other agents off-schedule. The actual run(s) may be intensive – something is still going to have to scan for and trigger the agents – but the Agent Manager overhead will come down.
The average size is about 125MB which results in about 5TB of total data.