Given the XML tools available for Domino are there any guidelines for which tools to use in particular when exchanging larger volumes of data with other dbs.
For example I have some agents that populate a Dominio db with XML data from Oracle via a DOM Parser but it all “chokes” when the number of records being processed goes up.
Is there some point at which I should be swithing to something like LEI?
Subject: Domino & XML - Volume?
The DOM parser is the first limiting factor. A parsed DOM tree can be between two and fifteen (!) times the size of the document it represents, depending on how complex the data hierarchy is that is being represented, and you have to have the whole tree in memory before you can even start. Switching to SAX can gain you a lot of headroom – it’s event-driven and so only needs to look at one tag at a time.
The XML/LEI decision shouldn’t be based on quantity, but rather on connection and control. XML has its benefits, of course, but they arise mostly from the ability to be system-agnostic. That is, you give up a lot of efficiency with XML in order to ignore the system at the other end – as long as the address and XML format of the data source remains the same, the entire foreign system can be replaced overnight and you don’t have to care. A native-type connection isn’t as versatile, but if you are “in” on the foreign system, you will know about changes ahead of time and will be able to keep up with them.
Subject: RE: Domino & XML - Volume?
Even with a switch to a SAX parser. For a large volume of data I still have a lot of issues to overcome e.g. limit on agent run time, amount of data coming across in source url, extending and maintaining the SAX parser, security of URL connection etc.
Have you had experience of similiar?
Subject: RE: Domino & XML - Volume?
Define “large volume”.
Subject: RE: Domino & XML - Volume?
At the moment 25,000 Oracle records → 25,000 Notes Documents but ill grow by 10,000 over next 12 months
Subject: RE: Domino & XML - Volume?
SAX should be able to handle that (unless there’s something about the Oracle side you’re not telling us, like that it involves multimegabyte blobs and lots of big text, etc.). I asked because some folks think that a couple-ten million rows is a reasonable single fetch. As XML, that can be like downloading the extended version of the Lord of the Rings trilogy in one shot (XML is kinda versose). Can you successfully stream the XML to a file without truncation?
Subject: RE: Domino & XML - Volume?
Are you suggesting 1st stream all the XML to a file then process the file via SAX?
Subject: RE: Domino & XML - Volume?
No, I am suggesting that you stream all the XML to a file as a test to see if the transmission is truncated. It won’t do a bit of good to try to work out the Domino-side processing problems if there is a problem with the data transmaission/reception. You need to confirm first that the data are making it from one machine to the other without damage.
Subject: RE: Domino & XML - Volume?
Ok … will try and update you … tx for all so far