After upgrading the Domino server from 6.5.x to 8.5, the scheduled Agents taking very long time to process. We have an agent which probes the previous day web access log for ?opendocument entries, then it opens the respective database, fetches few details from the document, create a new document to log the user details and the information about the document the user has accessed.
This agent usually takes 3 to 4 hours to complete on 6.5.x environment but it takes more than 12 hours now on 8.5 environment. Have anybody come across similar issue with Agent Manager which slow down after the upgrade? I doubt the agent manager has been throttled and not enough CPU been allocated to it. We have 4 agent managers running round the clock. The total CPU utilization of Domino is less than 25% during this agent execution, but the percentage cpu utilization for the agent manager that process this agent were close to 98% when it runs. Any help is appreciated.
Subject: Lots of possibilities…
For starters:
-
Did you compact your weblog.nsf after upgrade (is it ODS 51)?
-
Did you create new FTI for the log?
-
How big is the log?
-
How are you locating document in the log (ie, what script approach)?
-
Tell me you didn’t create a view who’s selection formula only grabs yesterday’s documents.
-
Why wouldn’t you just put code in the WQO of the desired forms themselves to log user activity vs. hashing the over-sized weblog?
Subject: Agent manager slow down
- The ODS is 51 for all the dbs. We are probing the accesslog.txt not domlog db. * The accesslog would be around more than 250MB
My question is why would the EXACT SAME CODE take 3X LONGER to run under 8.5 vs 6.5???
Subject: It is impossible to tell without looking at the data
The author of the part 1 of the article I posted in my earlier response, Raphael, is a consultant who shares his experiences addressing exactly that question when he goes out to the field. One of the common problems is related to rebuilding views. There are other reasons specific to each environment. The articles gives suggestions on what to look at to diagnose “slow downs”.
Subject: Agent Manager Slow Down
Hi Folks, Finally we found the root cause of this problem and it was caused by some of the group entries in our addressbook.
The following two items caused Agent Manager to clock:
-
Nested Administrator groups
-
If Administrators group is added to a group which is added to all databases given Editor access
We had couple of group documents contain Administrators group which are nested which lead to recursive lookup calls. Also, the performance of agent manager was badly impacted because of a group which is added to all our databases and this group has administrators group as its member
During GetDatabase calls, the agent manager refers the $ServerAccess view in the NAB for ACL lookup and r8.5 Agent Manager unable to determine the exact access control for the above said disputed group which lead to performance slow down.
We resolved this issue by removing the nested group entries and Administrators group entry from the disputed group.
Defiteinly there is some core level change has been introduced on r8.5 for ACL lookups, the mechanism for overcoming nested lookups was missing on the new release because the above setup works well with r6.5 & r7 with out any problem for several years.
Subject: Thank you for the follow-up!
Thank you for posting the resolution to the problem - I am sure it will help many others!
Subject: Agent Manager
There have been no code added to throttle down agents. We have added several tools to help diagnose performance problems.
I hope you will find the following two part article useful. The first part talks about
application design and things that can cause performance issues. And the second
part talks about the new tools.
Troubleshooting application performance
Part 1: IBM - United States
Part2: IBM Developer