I need to:1] Get all entries in view in a remote DB “log.nsf”, in the view called “archive”
2] Copy them to the Db where the agent is hosted called “archivedb.nsf”
3] Delete all entries and would love to then compact the “log” db.
My code works, but it takes forever ~45 Mins (I need to work with some BIG logs, my test log has 560 000 Doc’s and is not Full Text Indexed - so I cannot use a search formula). Does anyone know of a more efficient way to code this? Any comments greatly appreciated! Please see my sample code below.
'View based Archiver
Sub Initialize
Dim session As New NotesSession
Dim thisdb As NotesDatabase
Dim thatdb As New NotesDatabase ( "", "log.nsf" )
Dim view As NotesView
Dim vc As NotesViewEntryCollection
Dim entry As NotesViewEntry
Dim doc As NotesDocument
Set thisdb = session.CurrentDatabase Set thatdb= session.GetDatabase ( "" , "log.nsf")
Set view = thatdb.GetView ("archive")
Set vc = view.AllEntries
Set entry = vc.GetFirstEntry ()
While Not (entry Is Nothing)
Set doc = entry.Document Set entry = vc.GetNextEntry (entry)
Call doc.CopyToDatabase (thisdb)
Call doc.remove (True)
Wend
Subject: Calling all LS Gurus - Code verification required: transfer & deletion of HUGE view collection
Erik’s tip about moving the remove call outside the loop will save you considerable time. The only other possible shortcut at that point would be to try to eliminate (or at least speed up) the CopyToDatabase call. If the database this is running on has transaction logging enabled, there’s not much you can do to speed that up.
However, you might be able to avoid the CopyToDatabase call altogether. If you’re absolutely fanatical about maximizing your tweaks, then you might want to try a little experiment. Pipeline a DXLExporter from the log database to a DXLImporter to your agent database. No promises, but it might be faster. The importer always seems to generate docs for me faster than an LS .Save call (which CopyToDatabase is shorthand for)
Subject: RE: Calling all LS Gurus - Code verification required: transfer & deletion of HUGE view collection
I don’t see why you don’t use the Archive functionality that’s built into Notes, accessible thru the Database Properties. You can have it select documents for archiving based on what view they’re in, and it will work with a remote archive database.
For your future reference, however:
It’s best to Delete doc when you’re done using a NotesDocument object, since otherwise it’s cached, and you can use up all the memory processing so many documents.
I’m not sure whether it also helps to Delete entry.
The NotesView.AutoUpdate property may help.
You may get better performance using NotesView.GetFirstDocument and GetNextDocument rather than AllEntries thing. Since you’re handling the document objects anyway, you’re just adding overhead by referencing the documents by their NotesViewEntries.
Subject: RE: Calling all LS Gurus - Code verification required: transfer & deletion of HUGE view collection
“I don’t see why you don’t use the Archive functionality that’s built into Notes, accessible thru the Database Properties.”
You’re quite right, but there is one good reason – because that requires a COMPACT -A for that database on the server. A given developer may not be able to ensure a periodic run of such a thing, while he might be able to ensure an agent running.
Subject: RE: Calling all LS Gurus - Code verification required: transfer & deletion of HUGE view collection
Thanks for the response and the great suggestions Andre, I will be sure to give them a go. The reason that I don’t use Compact -a or -A is that these need to be set individually on a replica, which means that it will be lost if (and when) an admin renames the log.
Subject: Calling all LS Gurus - Code verification required: transfer & deletion of HUGE view collection
Move your remove call to the end and things should be much quicker. Change:
Set vc = view.AllEntries
Set entry = vc.GetFirstEntry ()
While Not (entry Is Nothing)
Set doc = entry.Document
Set entry = vc.GetNextEntry (entry)
Call doc.CopyToDatabase (thisdb)
Call doc.remove (True)
Wend
to:
Set vc = view.AllEntries
Set entry = vc.GetFirstEntry ()
While Not (entry Is Nothing)
Set doc = entry.Document
Set entry = vc.GetNextEntry (entry)
Call doc.CopyToDatabase (thisdb)
Wend
Call vc.RemoveAll(True)
Also, if all you’re using the view entry collection for is to get document handles you may experience better performance by simply walking the view using NotesDocuments. Ex:
Set doc = view.GetFirstDocument ()
While Not (doc Is Nothing)
Call doc.CopyToDatabase (thisdb)
Set doc = view.GetNextDocument (doc)
Wend
Call view.AllEntries.RemoveAll (True)
If you go that route it might actually be faster to remove each doc as it is processed rather than go through the expense of the view.AllEntries call. Experiment and good luck!
Subject: RE: Calling all LS Gurus - Code verification required: transfer & deletion of HUGE view collection
Thanks for the great suggestions guys! Your submissions with the ammended code were particularly helpful Erik. I am currently performing more tests with several new agents.