Hi,
we are trying to automatically import “regular” files into file resources inside a db. Therefore we take the “raw” file, encode it with base64 (with the help of a ls-library, available from nsftools - Lotus Notes and Domino Tips, and encapsulate it into the necessary xml.
This XML looks like in the following example:
<?xml version='1.0' encoding='utf-8'?>
D
abc.js
mydir/abc.js
6.5
3
QUJD
345CgQ
For testing purpose we created a text based file with the letters “ABC” in it. The above is the XML-output for this file
QUJD is the right presentation of ABC in base64 (as you can proove with this online encoder/decoder: Page Not Found :: OpinionatedGeek)
If we take the same textfile and (after importing it manually, with “New File Resource”) export it with help of the designers - DXL Utilities - Exporter the base64 results in
YQAaAAAAAgADAAAAAQAAAAAAAAAAAAAAanNgABUAAAADAAQAAAAAAAAAAABBQkMW
which is way to long for the base64 represantion of “ABC” and which results in
ajs`ABC
if decoded.
QUESTION: What does Domino do to create this strange looking base64 represantation and how can we rebuild it? If we try to import our above XML (the one with QUJD) we are getting an Importer-Error:
Document has invalid structure
DXL importer operation failed
If we paste in the loooooong base64 represantation (YQAa…) it works.
TIA
Subject: base64 encoding and DXL
This is due to the fact that file resources are stored/exported in a particular way that is not exactly the same as regular attachments… (Who of you guys at Lotus dreamed this up?).
I’ve reverse engineered this partly to be able to export FileResources from a database – I haven’t gone through the trouble of fully re-engineering it to be able to import this.
The gist:
– There is a global 28 byte header that starts at beginning of the file resource.
– After this there are blocks for each 10KB of data that contain
—> An 18 byte block header
—> 10KB of file data
—> The file data is appended with 0x0000
To extract this:
-
Skip the 28 byte global header
-
Read the first 18 byte block header
2(a). Byte 3+4 of the block header indicate the amount of data following that block header.
-
Read the data for the indicated amount.
-
If there is more data, read the next 18 byte block header and goto 2(a).
-
Strip the last two bytes (0x0000) from the retrieved data.
I have no idea what the other parts of the headers are supposed to do, as far as I know they are not documented anywhere. Since there was no requirement to automate imports at that time, I’ve not given it much thought.
If you are going to reverse engineer this further, let me know.
cheers,
Bram
Subject: It’s Composite Data
The grandparent post got me all curious again, so I investigated a bit more 
I dug a bit into the Notes C API reference, and AFAIK it seems tot be a basic CD (rich text composite data) structure.
The global header is a CDFILEHEADER, the block headers are CDFILESEGMENT structures.
These are documented in the C API reference, so you should be able to cobble up a functioning implementation with this.
(If you’re not able to make this work, feel free to contact me for my consulting tariff 
cheers,
Bram
Subject: RE: It’s Composite Data
Bram,
thank you for your answer. This is valuable information, although I also can’t figure out why “they” implemented file resources in this manner. Image resources are implemented in the “right” way (without this nonsense header stuff)… And are not so much different…
Thanks again!
Subject: RE: It’s Composite Data
Image resources are not encoded as “regular” file resources, but rather as images using slightly different CD-structures (CDIMAGEHEADER + CDIMAGESECTION).
I simply assume that the only reason why the DXL format differs is due to the fact that images are used a lot within Notes apps, so a translation layer was added in the DXL toolkit just as it was for attachments.
File resources (and CSS resources, which are encoded in the same manner) simply didn’t exist for the first release of the DXL toolkit (for R5), and aren’t used that much.
While DXL is a really powerful concept, there are some parts of the design and implementation that just aren’t what they are supposed to be…
A nice example is the fact that you cannot export/import custom format settings for date fields. I’ve developed a (relatively complex) application for a client that allows them to dynamically build forms depending on a simple configuration. This allows them to extend and rework parts of the application without any designer knowledge, yet it screws up any date fields with custom format settings. Try explaining that to your client :^P
cheers,
Bram
Subject: RE: It’s Composite Data
A few thoughts often hit me when I duck my head into the N/D forums (P.S. I’m a Domino DXL developer).There are a lot of really smart developers out there who find ways to use DXL that I hadn’t even thought of. I had to scratch my head for a minute or two before I realized what you are doing. The real problem (as you know) is that DXL does not (yet!) have a higher level representation for a File Resource design element. You seem to have cleverly worked your way around that.
DXL is getting mentioned more and more in here, it seems. I’m never too sure whether people are thinking about it as an alternative, but ultimately using a different solution like Midas or the C API (which would certainly be an option for you in this case), or are really using it. And unfortunately I usually don’t have time to dive on the issues when I read them, although I would like to.
This is the most important one. REPORT THE PROBLEMS YOU FIND. (Sorry for yelling.) I believe that anyone with a N/D license can report problems up through the support channels. Eventually that will result in an SPR. That’s when it really catches our attention. I would have to say at least 95% of the DXL issues I see talked about in here, I never see an SPR for. 'Tis a bummer, because problems without a customer complaint get lower priority, as they should. I am pretty fanatical at working my SPR queue once problems have made it there. So please, report them!
Subject: RE: It’s Composite Data
Hi Dick,
I appreciate it that the actual people doing the groundwork for the DXL kit are reading our rants from time to time :-).
I consider myself to be pretty knowledgeable about DXL and (as I’ve posted before) I think it’s a really powerful concept. It would be really awesome if the DXL toolkit would be feature-complete (which it isn’t… yet ;-), and the stability (at least when using the Java API) during hard-core DXL stuff is still a bit rough around the edges.
Relatively speaking, there aren’t a lot of us Notes developers actively (ab)using the DXL functionalities. Most actual uses of DXL are further limited to some kind of one-shot XSL transformation to HTML or some other markup language (i.e.: barely scraping the surface of what DXL can do).
There are a lot of reasons why a DXL issue reported here in the forum don’t end up as SPRs in your queue. Some features are not documented in the DXL DTD (e.g.: custom date/time format), so it isn’t considered a bug but a (missing) feature.
DXL is a relatively ‘young’ technology in Notes, and a lot of developers are still considering it to be somewhat experimental, so they either work around or outright avoid bugs in the DXL toolkit.
As for the reason why I personally haven’t reported any bugs during my work on DXL, that’s pretty simple: I’m an independent consultant and I have to rely on my clients (who have the actual Passport Advantage contracts with IBM) to submit any issues to Lotus… This usually involves jumping through lots of hoops to find the right person and get them to actually report and follow-up on an issue.
It would be really nice for (certified) developers to have some channel to IBM to actually report these kinds of bugs without having to rely on your client to do it in your place.
Anyhow, since this post is evolving in a dangerously rapid pace into a rant w.r.t. Lotus support, I think it’s time to end my ramblings
If you wan’t to discuss the intricacities of a developer’s point of view of dealing with (a lot of) DXL, feel free to contact me :-).
cheers,
Bram