Agent performance

HCL Domino Domino Forum
1

Hello,

I have a requirement where I have 2 databases. db.nsf and db2.nsf.

db.nsf has 960,000 documents. db2.nsf has 300,000 documents. I have to get a key value from db.nsf, and search in db2.nsf, and update the flags accordingly.

I have defined the following algorithm, and have a code also. The code works fine, but it runs for atleast 8-10 hrs or a day. I am not sure why. I have following questions,

Is there a way that I can improove the processing time by making changes in this code?

This line, Print "Proccessing document # - " + Cstr(count%) - Shows me Processing document # - 32767. But count then does not changes, but I can see the different fieldvalue1 upon next document. The counter works fine from 1…32767, but then stops increamenting, I am not sure why? Since the agent runs for hours and hours, I have set an email which will give me an update, how many docs have been processed? But since the count is not going further after 32767, I can not figure out how many documents have been processed?

Algorithm

Get the key from view ALL within db.nsf

Search in db2 for a key from db.nsf

If the count >= 3 then,

3.1

Update a flag “dbvalidemail” within db2.nsf

3.2

Update a flag “dbemailbounced” within db2.nsf

3.3

Update a flag “dbvalidemail” within db.nsf

If the count = 0 then,

4.1

Update a flag “dbvalidemail” within db

AGENT CODE -

Sub Initialize

Dim varray() As String	

Dim it As NotesItem

Dim i As Integer

Dim vc As NotesViewEntryCollection

Dim strProdNo As String		

Dim ns As New NotesSession

Dim ndt As New NotesDateTime ("")  'a time for our notes document and log messages

Dim theTime As String   'a string for the time

Dim errorlog As NotesLog

Dim doc2 As NotesDocument

Set db = ns.Currentdatabase	

Set errorlog = ns.Createlog("logdb")	

Set logdb = New Notesdatabase (db.server, "logdatabase.nsf" )	

theTime$ = ndt.LocalTime

Dim view As NotesView

Dim doc As NotesDocument

Set view = db.GetView( "ALL" )

Set vc = view.AllEntries

count% = 0

X = 1

Set doc = view.GetFirstDocument

While Not ( doc Is Nothing )

	count% = count% + 1

	Print "Proccessing document # - " + Cstr(count%)

	Print doc.fieldvalue1(0)

	val$ = doc.fieldvalue1(0)

	If (doc.HasItem ("fieldvalue2") And doc.fieldvalue2(0) <> "") Then			

		Call check(doc)			

	End If

	If X = 10000 Then

		Call SendMail( count% )

		X = 1

	End If

	Set doc = view.GetNextDocument( doc )

	X = X + 1

	On Error Goto this

Wend

this:

Call errorlog.Opennoteslog(db.Server, "logdatabase.nsf")	

errorlog.Logaction("Agent Started" + " at "& theTime$+ "Key is, "+ doc.fieldvalue1(0)	+" created using form, "+  doc.Form(0) +" has doc id "+ doc.UniversalID + "for user - " + doc.eu_first_name(0) + " " +doc.eu_last_name(0)) 'begin logging	

Call errorlog.Close

Print "Resume on error"	

Resume Next

Exit Sub

End Sub

Sub check ( req As NotesDocument )

Dim view1 As NotesView

Dim dc As NotesDocumentCollection

Dim doc1 As NotesDocument

Dim item, item1, item2 As NotesItem	

Dim db2 As NotesDatabase



Set db2 = New NotesDatabase( "IP ADDRESS", "db2.nsf" )

Set view1 = db2.GetView("ALL")		

query$ = req.email(0)

j = view1.FTSearch(query$, 0 )	

Set dc = view1.GetAllDocumentsByKey(req.fieldvalue1(0))

If ( dc.Count >= 3 ) Then

	Set doc1 = dc.GetFirstDocument()

	Set item2 = req.AppendItemValue("dbvalidemail", "No")	

	While Not(doc1 Is Nothing)				

		Set item = doc1.AppendItemValue("dbvalidemail", "No")	

		Set item1 = doc1.AppendItemValue("dbemailbounced", "Yes")				

		Call doc1.Save( False, True )									

		Call req.Save( False, True )									

		Set doc1 = dc.GetNextdocument(doc1)				

	Wend

Elseif ( dc.Count = 0 ) Then		

	Set item = req.AppendItemValue("dbvalidemail", "Yes")	

	Call req.Save( False, True )											

End If

End Sub

Sub SendMail (Count As Integer )

Dim emaildb As NotesDatabase

Dim emaildoc As NotesDocument

Dim emailsession As New NotesSession

Set emaildb = emailsession.CurrentDatabase

Set emaildoc = New NotesDocument( emaildb )

emaildoc.Form = "Memo"

emaildoc.SendTo = "user's name"

emaildoc.Subject = "Agent Status Update - " & Cstr(Count)

Call emaildoc.Send( False )

End Sub

Thanks

MS

2

Subject: RE: Agent performance

Where to start…The good news is, your agent can run much faster.

Someone already explained about your counter.

Please use Option Declare in all your scripts. You’ll find you make substantially fewer errors.

Adding the statement view.AutoUpdate = False just before GetFirstDocument will greatly improve performance.

The statement Set vc = view.AllEntries isn’t gaining you anything since you do not use vc thereafter.

Where possible, refer to values in the document by way of the Columnvalues array. For instance, If (doc.HasItem (“fieldvalue2”) And doc.fieldvalue2(0) <> “”) Then could maybe be written, If (doc.ColumnValues(3) <> “”) Then. Of course I don’t know which column this field might appear in. In any case, If (doc.fieldvalue2(0) <> “”) Then will also do the same thing and it is faster.

Every time you refer to a NotesDocument, the NotesDocument object is retained in a cache. If you process so many documents you are using up all the available memory by leaving them in the cache. Your main loop should use the Delete doc statement to clear the object out of the cache. Of course, this will require that you have a second NotesDocument object, docNext, so that you can do:

Set docNext = view.GetNextDocument(doc)

…

Delete doc

Set doc = docNext

The check function, every time it is called, creates a new NotesDatabase object and a new NotesView object. This is very wasteful. Create them once and store them in global variables, and you should see a dramatic increase in speed. (While it’s true global variables are a little slower than ones declared in your function, the difference is nothing compared to the amount of time to locate and open a database and a view).

Without knowing more about the structure of your view and the data in the lookup documents it’s hard to say for certain, but your call to FTSearch seems unnecessary. You are searching for documents that contain the value of the “email” field anywhere in them. I suspect there are actually only one field or a few fields that you need to look at, and the lookup view could be sorted by these fields – a keyed view lookup is faster than a FTSearch. I know, you need to lookup two keys, but GetAllDocumentsByKey lets you do that using an array. I’m not even sure that GetAllDocumentsByKey works correctly on the results of an FTSearch.

You’re calling req.Save inside of a loop, which results in your doing multiple saves of the same document.

This is not a performance issue, but are you sure that AppendItemValue is what you want to use, as opposed to, say, ReplaceItemValue?

This is apparently a server agent? All those Print statements are slowing it up. Perhaps you could just print the counter every 100 or 1000 documents. Similarly, 10000 is too frequently to send email. I don’t think you should bother to send email. Can’t you just look at the server log document to see how far it’s gotten?

3

Subject: RE: Agent performance

Once again a more experience member of the community post an excellent response.

To Andre, Stan Rogers, Ben L. (sorry, don’t want to butcher your last name) & so many others, You guys always go above and beyond in supporting the community. THANK YOU!

btw. I run a conceptually similar agent against a 9GB database of approximately 600K documents and another 2GB database of 400K documents. Thanks to sugguestions like these it finishes in less than 15 minues - it used to take over 3 hours (my server is seriously limited by drive throughput, but that’s another issue)

4

Subject: Agent performance

Consider rebuilding the view in db1 so that you can use the view.allentries instead of view.getfirstdocument.

This doesn’t “Save time” in that while it will make your agent run faster, your view indexing time may go up.

But, properly designed views will build MANY times faster than your agent can run using view.getfirstdocument.

Also, you could rebuild the view in db2 so that you can use view.getentriesbykey instead of the FTSearch. This is also MANY times faster.

5

Subject: Agent performance

For problem 2. If you want higher values you should not define count as Integer because you have hit the limit for a notes integer value. For the other part I am sorry but I am too tired today.

PS: I swear when I first looked at this there were no answers.

6

Subject: RE: Agent performance

Thank you all again for your guidance and great responses.

MS

7

Subject: Agent performance

I’ll answer the easy question first! #2:

The counter stops counting because You have defined the variable ‘Count’ as Integer, and Integer values have the maximum value of 32767. Change the definition to Long instead and You will not have the problem.

#1:

Much more difficult to answer, yet easy… Of course it will take very long time to execute if You have 960k+300k documents to handle and You do a FTSearch in every loop. This is something I wouldn’t recommend since LotusScript isn’t know to be very fast… I would seek a solution that didn’t include FTSearch and maybe a way to act on smaller amount of documents.

Have a look at C API’s to see if You can build a program that executes the same logic but outside the Notesclient using the built-in API-calls. This could be faster than using LS.

hth

8

Subject: RE: Agent performance

It’s not only the FTSearch, there are a lot of other things that drag this code down.

In the Initialize() sub, move the On Error Goto to the top. You don’t need to reset it in the loop. It may not be performance-affecting, but it doesn’t need to be there.

In the check() routine, move the req.Save outside the While loop. You don’t need to re-save req everytime you update doc1. You also don’t use the NotesItems you get back from the AppendValue call, so just use Call req.AppendValue instead. Move the db2 declaration to the Declarations section and initialize it in the Initialize procedure. You don’t have to recreate it everytime you call check, it’s not like it’s changing to a new location.

This last bit is more style than function, but please use Option Declare and declare all your variables. Throwing in random variables in the middle of an application makes it really difficult to troubleshoot.

Once you get through all that you might have something you can work with further.

Hope that helps,

Charles

9

Subject: RE: Agent performance

Thank you all for your responses. I will make a change for counter and for Problem #1 I will try other options.

Thanks again.

MS

10

Subject: Agent performance

Here is an article which talks about techniques to optimize performance:IBM - United States

11

Subject: Agent performance

  1. % is an Integer. Use a Long instead.