Local agent extremely fast, server agent glacially slow

Hello, all,

I have a short agent that runs on all documents in the view to build a flat file. Running it locally on my test database, it’s lightning fast and done with 700+ docs in several seconds. But running it on 700+ docs on a server test database you can watch individual docs click by (on a Print line). And running it on the prod db with 18K docs is impossible; it stops responding.

A while back I had the same situation, but none of the same factors are present; I do not have ComputeWithForm, there is no GetNthDocument, there’s no tricky Evaluate code or any computations at all. I tried splitting the docs into thirds, using a separate agent for each third, and in the new agents I added code to delete the doc object at the end of each iteration of the Do loop, and I added db.DelayUpdates = True, but there was still no improvement in performance (except that the agent does not hang anymore; 6000 docs get processed, but it takes 20 min).

I inherited this agent from our system administrator, for whom, of course, it worked fine. (I have left a message for him but the user must be able to run this agent by Jan 31, so I am entering panic mode.) We did change servers last spring, but I believe it was last run after that. It was run on a different db because back then the data was in Access and brought into Notes solely for the agent, but the only thing this agent does is get values and print an output string, so I can’t see how that would make a huge difference.

Here is the code:

Declarations:

Dim varoutName As String * 20

Dim varuserName As String

Dim varoutCountry As String * 20

Dim varPOBCountry As String

Dim varoutSpecAcc As String * 6

Dim varSpecAccess As String

Dim varoutDOB As String * 8

Dim varDOB As String

Initialize:

Sub Initialize

Dim s As New NotesSession

Dim db As NotesDatabase

Dim coll As NotesDocumentCollection

Dim doc As NotesDocument

Dim filename As String

Dim filenum As Integer

Dim outputstring As String

Set db = s.CurrentDatabase

Set coll = db.UnprocessedDocuments

Set doc = coll.GetFirstDocument

filename ="C:\CVSReport.txt"

filenum = Freefile()



Open filename For Output As filenum



Do While Not doc Is Nothing

	varuserName = doc.LName(0)

	varoutName = varuserName

	

	varPOBCountry = doc.SubCountry(0)

	varoutCountry = varPOBCountry

	

	varSpecAccess = doc.SpecialAccess(0)

	varoutSpecAcc = varSpecAccess

	

	varDOB = doc.OPM_DOB(0)

	varoutDOB = varDOB

	

	outputstring = doc.SSN(0) & doc.SOI(0) & doc.ClrLevel(0) & doc.RecordType(0) & doc.GrantDt(0) & varoutName &_

	varoutDOB & doc.SubState(0) & varoutCountry & doc.OPMExcep(0) & doc.ClrType(0) & varoutSpecAcc &_

	doc.Standard(0) & doc.Contact(0) & doc.OPMNonUSFam(0)

	

	Print #filenum, outputstring

	

	Set doc = coll.getNextDocument (doc)

	

Loop



Close filenum

End Sub

The only substantive difference in the “one-third” agents is that I have Set docNext = coll.GetNextDocument (doc) at the beginning of the loop, and Delete doc and Set doc = docNext at the end of the loop.

Seems like more than a memory problem if there is no performance difference between deleting the doc object and not deleting it… but what is it? There’s hardly enough code to even screw up!

Can someone point me in the right direction?!

TIA

Mark

Subject: Local agent extremely fast, server agent glacially slow

Something else you can do to improve performance is to delete “doc” after each iteration. Here’s an excerpt from one of Andre Guirard’s posts (full post here: ) on why it’s necessary & how to implement it:

Every time you refer to a NotesDocument, the NotesDocument object is retained in a cache. If you process so many documents you are using up all the available memory by leaving them in the cache. Your main loop should use the Delete doc statement to clear the object out of the cache. Of course, this will require that you have a second NotesDocument object, docNext, so that you can do:

Set docNext = view.GetNextDocument(doc)

…

Delete doc

Set doc = docNext

Ever since I heard about this technique, I’ve made sure I use it in every view/collection processing loop I write - it really makes a huge difference as your doc count increases.

Hope this helps,

Stephanie

Subject: RE: Local agent extremely fast, server agent glacially slow

Stephanie,

Thanks so much - I am already deleting the doc object after each iteration, thanks to a response from Andre to a previous post of mine. (It was not in the original code given to me, but I have it in my modified agent that only runs on one-third of the docs.)

Mark

Subject: Local agent extremely fast, server agent glacially slow

Mark,

Note my previous response, I am not sure why you are not using Lset into your strings, the ones declared with th ‘* 20’. By not using this, the strings are being destroyed and re-built each time, causing lots of garbage collection.

John

Subject: RE: Local agent extremely fast, server agent glacially slow

John,

I have this feeling my answer is going to sound pretty dumb, but the C drive referenced is the local hard drive of the user. Originally, the SA was the one running the agent for the user, so that’s how he coded it. I just left it as is, in an effort to change as little as possible… but eventually I want the user to run the agent himself. So I guess even though it does create an output file on my local hard drive, that’s not the way it should be done??

  1. I am not using Lset because 1) I’d never even used the “* 20” syntax before, and it was part of the original code, 2) it seemed to work well, and 3) I was not aware of the problem with strings being destroyed and re-built each time. Oh, and 4) I’m not familiar with Lset! So I need to learn that and see if that improves things…

Mark

Subject: RE: Local agent extremely fast, server agent glacially slow

Mark,

What if you tried something like the code below.

It doesn’t actually write the data to the file.

Create this agent and see if you end up with an array full of strings that would represent your flat file.

Also observer how long it takes.

This way you should be able to see if the performance hit is related to the file or not.

My guess is it’s related to the file and it’s ability to write the data out to 'C:' and your permissions.

You might want to create the file in the server data directory.

GL,

Shawn


Declarations:

Dim varoutName As String * 20

Dim varuserName As String

Dim varoutCountry As String * 20

Dim varPOBCountry As String

Dim varoutSpecAcc As String * 6

Dim varSpecAccess As String

Dim varoutDOB As String * 8

Dim varDOB As String

Initialize:

Sub Initialize

Dim s As New NotesSession

Dim db As NotesDatabase

Dim coll As NotesDocumentCollection

Dim doc As NotesDocument

Dim filename As String

Dim filenum As Integer

Dim outputstring As String

’ store the data locally before we write it to the file

dim dataArray(1) as String

Dim counter as int

Set db = s.CurrentDatabase

Set coll = db.UnprocessedDocuments

Set doc = coll.GetFirstDocument

filename =“C:\CVSReport.txt”

filenum = Freefile()

Open filename For Output As filenum

Do While Not doc Is Nothing

’ don’t redim the first pass through

if counter <> 0 then

Redim Preserve dataArray(counter +1) ’ add

end if

the next position

varuserName = doc.LName(0)

varoutName = varuserName

varPOBCountry = doc.SubCountry(0)

varoutCountry = varPOBCountry

varSpecAccess = doc.SpecialAccess(0)

varoutSpecAcc = varSpecAccess

varDOB = doc.OPM_DOB(0)

varoutDOB = varDOB

outputstring = doc.SSN(0) & doc.SOI(0) & doc.ClrLevel(0) & doc.RecordType(0) & doc.GrantDt(0) & varoutName &_

varoutDOB & doc.SubState(0) & varoutCountry & doc.OPMExcep(0) & doc.ClrType(0) & varoutSpecAcc &_

doc.Standard(0) & doc.Contact(0) & doc.OPMNonUSFam(0)

’ Don’t write it yet.

’ Print #filenum, outputstring

dataArray(counter) = outputstring

Set doc = coll.getNextDocument (doc)

Loop

Close filenum

End Sub

Subject: RE: Local agent extremely fast, server agent glacially slow

I forgot to increment the counter.

Use this.

Shawn


Declarations:

Dim varoutName As String * 20

Dim varuserName As String

Dim varoutCountry As String * 20

Dim varPOBCountry As String

Dim varoutSpecAcc As String * 6

Dim varSpecAccess As String

Dim varoutDOB As String * 8

Dim varDOB As String

Initialize:

Sub Initialize

Dim s As New NotesSession

Dim db As NotesDatabase

Dim coll As NotesDocumentCollection

Dim doc As NotesDocument

Dim filename As String

Dim filenum As Integer

Dim outputstring As String

’ store the data locally before we write it to the file

dim dataArray(1) as String

Dim counter as int

Set db = s.CurrentDatabase

Set coll = db.UnprocessedDocuments

Set doc = coll.GetFirstDocument

filename =“C:\CVSReport.txt”

filenum = Freefile()

Open filename For Output As filenum

Do While Not doc Is Nothing

’ don’t redim the first pass through

if counter <> 0 then

Redim Preserve dataArray(counter) ’ add the next position

end if

varuserName = doc.LName(0)

varoutName = varuserName

varPOBCountry = doc.SubCountry(0)

varoutCountry = varPOBCountry

varSpecAccess = doc.SpecialAccess(0)

varoutSpecAcc = varSpecAccess

varDOB = doc.OPM_DOB(0)

varoutDOB = varDOB

outputstring = doc.SSN(0) & doc.SOI(0) & doc.ClrLevel(0) & doc.RecordType(0) & doc.GrantDt(0) & varoutName &_

varoutDOB & doc.SubState(0) & varoutCountry & doc.OPMExcep(0) & doc.ClrType(0) & varoutSpecAcc &_

doc.Standard(0) & doc.Contact(0) & doc.OPMNonUSFam(0)

’ Don’t write it yet.

’ Print #filenum, outputstring

dataArray(counter) = outputstring

counter = counter +1

Set doc = coll.getNextDocument (doc)

Loop

Close filenum

End Sub

Subject: RE: Local agent extremely fast, server agent glacially slow

Shawn,

I tried saving this code but got “ILLEGAL REDIM on: DATAARRAY.” In the Help on the ReDim statement, it says:

Do not use ReDim on a fixed array (an array already declared and allocated by a Dim statement).

Is it not like the dim’ing of dataArray?

Mark

Subject: RE: Local agent extremely fast, server agent glacially slow

Mark,

Maybe I should have asked, are you running the agent by hand on a server copy of the dB, or is the agent running on the server by a scheduled agent or something.

Lset ing into a string does not destroy the string and it will always retrained the length you set it at. Thus if you lset a 40 byte string into one defined as 20 bytes, the last 20 bytes will be truncated.

I would change your script to use the lset.

Finally, while it does not make a huge difference, you should be using the doc.getItemValue( “”)( 0) syntax, since that too is faster than what you are using. It will also help you to use the proper methods when you starting doing java.

john

Subject: RE: Local agent extremely fast, server agent glacially slow

John,

To answer your question: I am running the agent by hand on a server copy of the db. I don’t have permissions to run an agent on the server.

I declared the Lset variables in Declarations, moved setting the Lset lengths outside the loop, and re-wrote the output string to first GetItemValue for all the fields. It seems just a tiny bit faster - I can barely see each incremented number - but basically you can still see it going up one at a time.

Here is the section of modified code:

doccount = 1



Open filename For Output As filenum



w = "qqqqqqqqqqqqqqqqqqqq"  ' (length of 20)

x = "qqqqqqqqqqqqqqqqqqqq"  ' (length of 20)

y = "cccccc"    ' (length assignment of 6)

z = "cccccccc"    ' (length assingment of 8)	



Dim varSSN As Variant, varSOI As Variant, varClrLevel As Variant, varRecordType As Variant, varGrantDt As Variant, varSubState As Variant, varOPMExcep As Variant, varClrType As Variant, varStandard As Variant, varContact As Variant, varOPMNonUSFam As Variant



Do While doccount <= thirdofdocs

	

	Set docNext = coll.GetNextDocument (doc)   ' (Get a handle to the next doc in advance)

	

	percentage = (doccount/thirdofdocs) * 100

	' Print progress on status bar

	Print "Now processing doc #: " & doccount & " - " & percentage & "%"

	

	varuserName = doc.GetItemValue ("LName")(0)

	Lset w = varuserName

	

	varPOBCountry = doc.GetItemValue ("SubCountry")(0)

	Lset x = varPOBCountry

	

	varSpecAccess = doc.GetItemValue ("SpecialAccess")(0)

	Lset y = varSpecAccess

	

	varDOB = doc.GetItemValue ("OPM_DOB")(0)

	Lset z = varDOB

	

	varSSN = doc.GetItemValue ("SSN")(0)

	varSOI = doc.GetItemValue ("SOI")(0)

	varClrLevel = doc.GetItemValue ("ClrLevel")(0)

	varRecordType = doc.GetItemValue ("RecordType")(0)

	varGrantDt = doc.GetItemValue ("GrantDt")(0)

	varSubState = doc.GetItemValue ("SubState")(0)

	varOPMExcep = doc.GetItemValue ("OPMExcep")(0)

	varClrType = doc.GetItemValue ("ClrType")(0)

	varStandard = doc.GetItemValue ("Standard")(0)

	varContact = doc.GetItemValue ("Contact")(0)

	varOPMNonUSFam = doc.GetItemValue ("OPMNonUSFam")(0)

	

	outputstring = varSSN & varSOI & varClrLevel & varRecordType & varGrantDt & w &_

	z & varSubState & x & varOPMExcep & varClrType & y &_

	varStandard & varContact & varOPMNonUSFam

	

	Print #filenum, outputstring

	

	Delete doc   ' This deletes the object, not the document itself

	Set doc = docNext  ' (handle gotten at beginning of loop)

	doccount = doccount + 1

	

Loop



Close filenum



Msgbox "Report done!  See C:\CVSReport1.txt.", 0 + 64 + 0 + 0, "Completed"

Is it slowing it down to ask the server to write to a file on my hard drive?

Mark

Subject: RE: Local agent extremely fast, server agent glacially slow

Running an agent by hand on a server is always slower, especially if you treat document content.

You might consider using the columnvalues property instead of directly accessing the documents the way you do. Of course, you will have to specificaly design a view with all the needed columns in order for columnvalues to work. Usually, this increases the speed of the treatment.

Additionnaly, you might also consider user a NotesViewNavigator, which might also slightly improve performance.

Kristof

Subject: RE: Local agent extremely fast, server agent glacially slow

Kristof,

Thanks so much for the ideas. I will try using ColumnValues if I have time to re-architect the agent in the next day or so. Right now I’m almost to the point of having the user create a local copy of the db and then run the agent since I know it runs rapidly and successfully in that situation (or I will just create the report file myself). Then after this report is done I can work on a more elegant solution.

I’ve never used NotesViewNavigator. I read the Help but don’t see how it differs markedly from NotesView or NotesViewEntryCollection or what advantage it gives you??

Subject: RE: Local agent extremely fast, server agent glacially slow

The main difference between NotesView and NotesViewNavigator is that NotesViewNavigator caches the view entries. So navigating the view is faster.

There is also an other advantage, you can browse the view just like it appears, going from one category to another. If you have response documents, you can jump from parent to parent, etc.

The only drawback I’ve come accross is when debugging. When the view is on a server and contains several documents, the debugger seems to communicate a lot with the server from the moment the NotesViewNavigator object has been instanciated, so step by step debugging becomes like driving in a traffic jam. Using breakpoints is highly recommended in that case…

Once you have used them, you’ll probably never forget about them.

Here’s a sample:

set db=session.currentdatabase

set view=db.GetView(“MyView”)

set viewNavigator=view.CreateViewNav

set viewEntry=viewNavigator.GetFirstDocument ’ This is the closest to a view navigation, but there is also GetFirst if you browse through categories

while not viewEntry is nothing

print viewNavigator.Columnvalues(4)

print viewNavigator.Document.MyField(0) ’ Avoid this…

set viewEntry = viewNavigator.GetNextDocument ’ Check the help for other possibilities: category, sibling, etc.

wend

Subject: RE: Local agent extremely fast, server agent glacially slow

Thanks so much for the info. I guess this is the next thing I will try… but first I will set it up to use ColumnValues.

Subject: RE: Local agent extremely fast, server agent glacially slow

OK, I designed a new view so that I can use ColumnValues. As far as I can tell, I set it up the same way I have used ColumnValues before:

Sub Initialize

Dim s As New NotesSession

Dim db As NotesDatabase

Dim view As NotesView

Dim coll As NotesDocumentCollection

Dim doc As NotesDocument

Dim filename As String

Dim filenum As Integer

Dim outputstring As String

Dim doccount As Long, total As Long, thirdofdocs As Long, percentage As Integer

Set db = s.CurrentDatabase

Set view = db.GetView ("CVS")  ' (To try using ColumnValues)

view.AutoUpdate = False

Set vcoll = view.AllEntries

Set ventry = vcoll.GetFirstEntry

Set doc = ventry.Document	



filename ="C:\CVSReport1.txt"

filenum = Freefile()	

db.DelayUpdates = True  ' from Notes.net posting; to improve performance (doesn't actually seem to speed it up)

doccount = 1



Open filename For Output As filenum



w = "qqqqqqqqqqqqqqqqqqqq"  ' (length of 20)

x = "qqqqqqqqqqqqqqqqqqqq"  ' (length of 20)

y = "cccccc"    ' (length assignment of 6)

z = "cccccccc"    ' (length assingment of 8)	

	

Do While doccount <= thirdofdocs

’ (I took out the code calculating doccount, etc. because I know that already works fine)

	Set entryNext = vcoll.GetNextEntry (ventry)

	Set docNext = entryNext.Document

			

	' Using ColumnValues instead of accessing the doc directly:

	Lset w = doc.ColumnValues (6)

	Lset x = doc.ColumnValues (9)

	Lset y = doc.ColumnValues (12)

	Lset z = doc.ColumnValues (7)

	

	Print #filenum, 	doc.ColumnValues (1) & doc.ColumnValues (2) & doc.ColumnValues (3)_

	& doc.ColumnValues (4) & doc.ColumnValues (5) & w & z & doc.ColumnValues (8)_

	& x & doc.ColumnValues (10) & doc.ColumnValues (11) & y & doc.ColumnValues (13)_ 

	& doc.ColumnValues (14) & doc.ColumnValues (15)

	

	Delete ventry

	Set ventry = entryNext

	

	Delete doc   ' This deletes the object, not the document itself

	Set doc = docNext  ' (handle gotten at beginning of loop)

	doccount = doccount + 1

	

Loop



Close filenum



Msgbox "Report done!  See C:\CVSReport1.txt.", 0 + 64 + 0 + 0, "Completed"

End Sub

But I am getting the “Variant does not contain a container” message on the line where I set the value for w. What am I missing??

Mark

Subject: RE: Local agent extremely fast, server agent glacially slow

This is because you use a view entry collection.

In that case the columnvalues is set on the entry, not the document. make this little change:

instead of doc.columnvalues(n), write ventry.columnvalues(n)

Good luck.

Subject: SOLVED!! re: Local agent extremely fast, server agent glacially slow

Kristof,

THANK YOU A MILLION TIMES OVER! As soon as I corrected the ColumnValues code, the agent worked perfectly! Now it runs the entire 18K+ records in about a minute and a half (where before, 700 records took 20 min)!!!

I can’t believe the gigantic difference between doc.GetItemValue and viewentry.ColumnValues!

THANK YOU FOR YOUR HELP AND TIME! I can’t tell you how much I appreciate it.

Mark

Subject: RE: Local agent extremely fast, server agent glacially slow

Never mind about the server, since I had thought you were running the agent on the server.

I would get rid of the variant data types, since they too are big on how much memory they take up, and don’t build one string to use in another string, simply write both lines into one as.

lset w = doc.GetItemValue (“LName”)(0)

this stops the gargabe collection even more, you can then further reduce it by

print #fileNum, w & x … & doc.getItemValue( “contact”)( 0) & doc.getItemValue( “”)( 0)

if need be use the line continutuation of _ to make each one on its own line

John

Subject: RE: Local agent extremely fast, server agent glacially slow

I really appreciate your ideas. I got rid of the variants and combined things per your suggestions, but there is still no change in performance.

Subject: RE: Local agent extremely fast, server agent glacially slow

How about instead of using a NotesDocumentCollection, try using a view and setting the AutoUpdate to off.