Does anyone have or know of anyone who has a Lotusscript class that can create a JSON object called by ajax and then loaded using datatables for a very large dataset? Say 10,000 documents. I cant seem to process this much data in under 30 secs using the usual string concatenation blah…blah…which is very expensive technique to use I know. If not Lotusscript perhaps something else can create the JSON object faster? FYI: Not on Xpages yet…
Appreciate any advice.
Subject: Have you considered using the view rendering option format output as JSON?
Then you don’t need any Lotusscript, and just use the Domino view rendering engine to do all the work.
database.nsf/viewname?readviewentries&OutputFormat=JSON
Subject: just a quick update on this - FYI:
So after a couple months learning some xPages REST services, xAgents, etc… I was able streamline my current process to produce my JSON object without any LotusScript!! About time.
What i do now is on web open call an xPage … “https://domdevostx01.ten-net.net/SASE/SASENomination.nsf/GetJSON.xsp”… eseentially an xAgent like Howard explained and in the afterRenderResponse event call a Java Class that actually produces the JSON object and sends it back to me.
Oddly enough I load my data into Datatables and not xPages. I am was not happy about the out of the box xPages view and Dojo grid and our organization will not fund the Dojo grid EXTJS so I stick with Datatables. I now can load 20,000 documents in 5 seconds with all the nice filtering, paging etc that Datatables provides.
Subject: Post your code
If you post your existing code, it would be easier to help.I create plenty of JSON using Lotusscript, but I write the code for each agent from scratch.
The reason for this is that I want the code to be as fast as possible. If I process a large number of documents, I don’t want to actually open the document to read values, I use a view where I expose the values I want to include, then use the NotesViewEntryCollection and NotesViewEntry classes to read the values I need.
Subject: Code post
Karl-Henry, sorry for the cross post.
I am using NotesViewNavigtor as seen below and it’s slow as can be. To build the JSON output for 10,000 documents it takes nearly 30 sec. Thoughts?
Dim VE As NotesViewEntry
Dim Nav As NotesViewNavigator
Dim vStr As String
View.AutoUpdate = False
Set Nav = View.CreateViewNav()
’ enable cache for max buffering
Nav.BufferMaxEntries = 400
Nav.EntryOptions = Vn_entryopt_nocountdata
Set VE = Nav.GetFirst
While Not( VE Is Nothing)
If ( JSONOutput <> "" ) Then
JSONOutput = JSONOutput & ", "
End If
JSONOutput = JSONOutput & |{EmpNum: "| & VE.ColumnValues(0) & |", Name: "| & VE.ColumnValues(1) & |", NBK: "| & VE.ColumnValues(2) &_
|", NomType: "| & VE.ColumnValues(3) & |", NomDate: "| & VE.ColumnValues(4) & |", Status: "| & VE.ColumnValues(5) & |", APL: "| &_
VE.ColumnValues(6) & |", NomType: "| & VE.ColumnValues(7) & |"}|
Set VE = Nav.GetNext(VE)
Wend
Print “Content-Type:text/plain”
Print “[” & JSONOutput & “]”
Subject: Profiler
I use TeamStudio Profiler to check my code for slow spots. If you have access to that excellent tool, I would suggest using it to see if you can pinpoint the slow code.
Another thing to try is to use a NotesViewEntryCollection instead:
Dim col as NotesViewEntryCollection
Dim entry as NotesViewEntry
Set col = view.AllEntries
'Set col = view.GetAllEntriesByKey(key)
Set entry = col.GetFirstEntry
Do until entry Is Nothing
'Put yoru existing code to build JSON here
Set entry = col.GetNextEntry(entry)
Loop
You can keep your code to build JSOn pretty much the same. Using & to concatenate values is actually a little bit slower than converting the values to text and concatenating them with +, so you may want to change that.
As you can see at the beginning of my code I also added an optional line to get a smaller subset of values and just process them, not the full view.
Another way to speed up the code is to remove the if-statement where you check if JSONoutput is not blank and add a comma.
I would add a comma (and a space) to the end of every item, then remove the last two characters of the JSONoutput string when you are done:
JSONoutput = Left$(JSONoutput,Len(LSONoutput)-2)
That will speed up your code as well.
Subject: Just go to XPages for this one thing
You can create an XPage that is unofficially called an Xagent. You prevent the XPage from rendering and control all the output with the afterRenderRepsonse event. This will work much faster than a LotusScript agent.
Howard