where bytes is the byte to write and length indicates the length of the stream to write.
My problem:
I have a document with a single text attachment of 32MB - nothing else. I use my app to export the doc to XML, it took over 6 minutes to export with the CPU sat at 85%+.
I noticed that the callback was always writing 1 byte at a time and never more than 1 byte? which would obviously increase the performance.
I have tried the dxlexport.exe application that comes with the toolkit and it takes the same time.
Has anyone come across this? and a possible solution to get better performance figures?
Subject: Large attachments, OLE objects and imbedded images has always been slow with DXL Export
I have not be able to speed it up either
One solution I do in one of my applications, is to have DXL Processing in a separate thread, and have a user configurable “threshold value” control wheather a user want to export larger documents than the threshold value or not. Not an ideas situation!
One thing is nice about DXL in ND6 though, and that is the callbacks. You can at least inform the users about the progress! In R5, we didn’t have this feature, and the DXL export just worked within it’s blackbox.
Subject: RE: Large attachments, OLE objects and imbedded images has always been slow with DXL Export
Hi Shahid,
Yes your observations are correct in that the XMLWriter function always returns a single byte and no more. I did pose this question to the DXL design team about a year ago or so and they ‘unofficially’ indicated that it may be something that they’d look into. Given that i’m not certain what the Lotus position currently is regarding DXL as a strategic interface to/from Domino i’m not sure how much input/resource is being directed into this and related areas.
From re-collection, my research of buffering within the XMLWriter module before writing out to disk did provide a minimal improvement but nothing that would negate the performance overhead of DXL’ing large files. Remember that a 32MB file will add an approx 33% overhead when base64 encoded so you’re looking at an approx 42.5MB in your stated case as well as a resource (CPU) intensive process.
The above said, there are some additional design strategies that can be utilised to offset the ‘perceived’ performance overhead. Let me know if you’d like to cover those in greater detail.
Subject: RE: Large attachments, OLE objects and imbedded images has always been slow with DXL Export
Thanks for calling this to our attention. We do have a related SPR in the system, JHER64WPGZ, “DXL export progressively slower for larger file attachments”. I’ve been looking into this SPR and currently I do not suspect that the 1 byte at a time to the C API callback is a major performance issue. Clearly it can be improved upon, and I intend to look into it eventually, but it is not the major factor, for this SPR at least.
The case you cited where it took 6 minutes to export a 32MB attachment is absymal performance and it actually degrades badly as the attachments get larger. I believe I have tracked this down to inefficient code that decodes attachments that are Huffman encoded. The bad performance appears to be in a system class we use, and it seems to be fixed in 7.0. Can you verify the following for me? If you export a 32MB attachment that is not compressed, I believe it will take on the order of 15 seconds or so to do the export. If this is the case, a possible workaround for you is to not compress your attachments, if that is an option. Also, exporting a 32MB Huffman compressed attachment with a Beta of 7.0 should take on the order of 30 seconds or so to do the export. Obviously the times are system-dependent but mine were on a pretty fast Windows system with plenty of RAM and pagefile.
