What XML/DXL features would you like to see post-7.0?

hcl-bot · July 25, 2005, 4:59pm

I’m taking this informal survey as to what XML/DXL features you would like to see in N/D post-7.0. If you use XML/DXL features or have investigated using them, please take a moment to ponder this list and give your top 3 (or so) answers. Replies that say “I want them all” are not really that helpful (sorry).Here is a brief list of ideas we have heard from customers and/or come up with ourselves, with a one or two sentence description. This list does not represent any commitment whatsoever from N/D Development.

If you have other ideas, please be as specific as possible as to what you’d like to see.

In no particular order…

?SearchViewEntries: Like ?ReadViewEntries, but view contents are determined by a full-text search. (Is this needed from Exporter APIs as well (in addition to the Web Server))?

?ReadDXL: URL command to get the DXL representation of a document or design element. What are your security concerns? 1+ would be a parameter that could apply an XSLT transform to the DXL and return the result – how important is the 1+?

DXL Import/Export capability from the Notes Client. Ability to import from DXL files into document(s), ability to export document(s) to DXL files. What are your security / usuability concerns? 1+ would be to apply XSL as well, “E&T” = Export to DXL, Transform, write to file, “T&I” = read from file, Transform to DXL, Import – how important is the 1+? (Note: XSLT style sheets for transforms would be user-supplied, not part of N/D.)

Ability for all of the Exporter / Importer APIs to easily include transform(s) – similar to E&T and T&I from previous item. Convenience feature. You can accomplish the same thing with existing APIs, but with a little more work because you have to code up the transformer logic yourself.

HTML generation from DXL: What level of fidelity do you require? 1+ would be to support other formats (e.g. PDF) – how important is the 1+, and to which formats? Is there alternate format that is more suitable; for example XSL:FO generation from DXL? (Then use any transforms available in the industry to go from XSL:FO to HTML et. al.) Are there open source solutions that eliminate this requirement?

XSLT Style Sheet design element: Possibly needed to support 2., 3., and 4., and maybe 5., if we want the ability for the XSLT style sheet to come from the database rather than a filesystem file or elsewhere.

“URL” class in LotusScript and Java: At a minimum, this would work in conjunction with NotesStream / Stream so that those classes could read a stream of input data from a URL. Major use would be to get input for XML parsers, DXL Importer, or XSLT Transformer from URLs.

Import DXL into some sort of a memory object: Instead of directly creating notes in a database. Pretty sketchy how this would work at this point, but is it an idea worth pursuing?

Mechanism to generate LotusScript/Java classes for working with XML: Idea here would be to take any XML vocabulary (not DXL) and automatically generate (from its schema) classes with properties and methods that allow one to work with the XML data in a language-specific way rather than dealing with the XML directly. Deserialize method would go from XML to objects and serialize from objects to XML.

Provide DXL Exporter capability for transforming notesbitmap to jpeg and/or png. (Already can do notesbitmap → gif.)

Enhance DXL to improve on some of its quirks. Which quirks of the language bug you? (One example, the structure of a list really isn’t well-represented in DXL.) May be a prerequisite to certain items like 5.

DXL Exporter Filtering: Ways to export partial notes by criteria such as “export everything but xyz” or “export only xyz”. Largest anticipated usage would be to filter out attachments and graphics, but goal would be to do something more generic than that.

Implement DXL for features that don’t yet have a high-level representation in DXL. List includes calendar view, MIME, web service, shared column, OLE, file resource, style sheet resource, layer, outline, replication formula, layout regions, database icon, viewentries, navigator. Are any of these important to you and if so, which few are the most important and how do they balance against the other features above?

(Your suggestion here)

hcl-bot · July 27, 2005, 3:15pm

Subject: Import DXL into some sort of a memory object (Yes!!)

Import DXL into some sort of a memory object: Instead of directly creating notes in a database. Pretty sketchy how this would work at this point, but is it an idea worth pursuing?Make it so! Yes! Please!

hcl-bot · July 29, 2005, 11:56am

Subject: Support for the xml-stylesheet processing instruction

I’m a big proponent for client-side xsl transformations. Currently, this is quite easy with IE and the Mozilla based browsers via JavaScript. Using the open source Sarissa project (http://sarissa.sourceforge.net/doc/index.html) makes it even easier.

However, the MAC Safari browser doesn’t currently provide a JavaScript api for xsl transformations. It does, however, support xsl transformations via a <?xml-stylesheet> processing instruction. If you could give us the ability to specify an xsl stylesheet that would be added the output of ?ReadEntries and ?ReadViewEntries, then client-side transformations can be done in the MAC Safari browser. This would also be a great an easy way to do simple xsl transformations client-side without the need of JavaScript.

Sure, doing this server-side would also be good but I think it should be much easier as a first deliverable (i.e. we get this sooner! :)) if you just provide a way for us to specify what xsl to include for a view and then let the browser do the transformation.

ReadViewEntries Output currently:

<?xml version="1.0" encoding="UTF-8"?>

…

ReadViewEntries Output proposed:

<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="dominoview.xsl"?>

There are a several places that I can think of where this information can be entered.

The properties box for the view itself
On an embedded view, the Web Access\Display dropdown could include an option to display “using XSLT” and then let the user specify the name of the xsl file.
URL argument such as http://server/db/view?OpenView&xslt=dominoview.xsl
server ini variable

hcl-bot · October 7, 2005, 7:15am

Subject: RE: Support for the xml-stylesheet processing instruction

I had a kinky thought on client-side transforms. So far, as I understand it, there have been two directions for those proposing transform support. One is to support server-side transforms, which would be most effectively done if we defined a new design element (XSLT Style Sheet) and various ways in the API/URL commands/options to specify one of these to automatically apply to a XML result document in the Domino web server, and deliver the result of the transform back to the browser/caller.

The other is to provide some simple hooks in the API/URL commands/options to specify an <?xml-stylesheet> processing instruction to be emitted into the generated XML, whose href attribute references a file on the client side. Correct me if I am wrong.

Would it be better for the hooks to specify a stylesheet stored in a Domino db that could be embedded into the XML document? That way, the transformation would still happen on the client side, but the style sheet itself would be stored in Domino, giving you the benefits that come along with that. The href in the xml-stylesheet instruction should be able to contain just a fragment id (href=‘#stylesheet-identifier’) which allows the stylesheet to be within the document.

If this were an option, I’m not sure it would merit an XSLT Style Sheet design element. I think that should be reserved for the server-side usage above. It could just reference a File Resource design element. Would that be an abuse of a file resource or a legitimate usage of it?

Any thoughts out there?

hcl-bot · October 10, 2005, 4:39am

Subject: RE: Support for the xml-stylesheet processing instruction

I’m not sure who was proposing that the client-based transform use a style sheet that’s already present on the client. That wouldn’t seem very useful to me. What’s needed is a client-side transform that uses the processor at the client, but references a style sheet that’s delivered by the Domino server.

Whether it needs a seperate design element is an open question. A file reference would be just fine. The main point is the AJAX-orientation involved in using the client to do the heavy lifting on formatting.

But that’s not the important point. The ultimate cool solution is that there should be a syntax to call a design element in an NSF that returns the design as a style sheet, that transforms standard DXL. THAT would be unbelievably powerful, as it would provide a native-mode support for the kinds of projects that Jack and Erik are talking about. If I could use ?ReadViewEntries to retrieve the DXL, and some kind of view?ReadDesign&transform syntax to get the view’s design as a STYLE SHEET, then the client could turn the standard DXL output into something really useful at the client end, but for very little cost to the Domino server (that style sheet would be heavily cachable, for instance.)

Think of it another way: the Java applet should be replaced with what would effectively be an AJAX applet. Now do the same thing with forms. Getting the idea? Domino applications suddenly become extremely lightweight. They get more portable. They automagically leverage the latest & greatest in browser technology.

Or put it another way, every homegrown app gets all the sex appeal of DWA, pretty much instantly. Or at least, a heckuva lot easier than all the effort that goes in now.

hcl-bot · July 3, 2007, 5:17pm

Subject: See openntf project NotesView2

See this project NotesView2 on openntf.org. The author is attempting to do just this, and the samples look good.

http://www.openntf.org/Projects/pmt.nsf/ProjectLookup/NotesView2

hcl-bot · October 10, 2005, 9:12am

Subject: RE: Support for the xml-stylesheet processing instruction

I may have misunderstood the proposal by Jack Ratcliff in the main note of this subthread. I thought the proposal was for the ?xml-stylesheet instruction to reference a client-side file. I too thought that would be limiting in that it would require deployment of additional file(s) on the client.

I think your other ideas are fantastic as well. Right now I am trying to split things up into shorter and longer term items and they might be longer term. One question that immediately pops to mind is if Domino is itself returning an XSLT to style a form/view/etc. it seems like it is going to be a “one size fits all” approach that inevitably people are going to want to tweak and change. What would be the mechanisms to allow this flexibility?

hcl-bot · October 10, 2005, 10:12am

Subject: RE: Support for the xml-stylesheet processing instruction

I think we’re all in agreement that referencing a client-side file would be fairly limiting.

Effectively returning design elements as XSLT would be handy, but as you’re pointing out Dick things either get VERY complex or else have to be forced into a “one-size-fits-all” approach.

For items with a limited amount of design DXL and a very predictable structure (e.g. views) this wouldn’t be very difficult. Heck, you could probably do it today with a LotusScript agent with a bit of coding. For more complex items (e.g. forms) there are a ton of attributes to consider, however – effectively all of rich text. Very cool, for sure, but obviously not something that would be very straightforward or inexpensive to do. On the other hand, once you’d done it for forms you’ve effectively done it for pages, and there would be other two-birds-with-one-stone cases.

But for something like forms, what would you get out of it other than server performance improvements? And I don’t know how much of that you would gain, either – in many cases generating the DXL behind a form is much more expensive than rendering it to HTML (and also usually much more bloated bandwidth-wise). You could cache it, but you could also have the browser cache an ?OpenForm call today… I’ll admit, though, that I haven’t thought about this in great detail as I’ve got a pounding headache.

If IBM/Lotus included an AJAX view applet alternative I would personally fly to every IBM/Lotus branch office and kiss everyone I saw. Heck, want mine? There’s no reason you couldn’t do the same with the outline applet, action bar applet, etc. I believe I even saw an OpenNTF project that was an AJAX alternative to the outline applet at one point.

As far as “short term” items, Dick, I would think an obvious choice would be incorporating some of the DWA work into Domino’s native HTTP engine. DWA is full of such fantastic web technology that has both great functionality and lots of sales-sizzle. The various DWA “picker” widgets (date, time, dialog, etc.) are a perfect example.

Consider a date field: In Notes, you get pickers. In DWA, you get pickers. But on the web (not DWA), you get an box. Many websites out there have DHTML date pickers bound to those sort of fields. Why not Domino? I would think it would be fairly straightforward to have those sort of widgets in DWA become part of the native Domino HTTP rendering engine. For backwards compatibility you could make it a new checkbox on the field in Designer ( on web access generate widget). I would think that incorporating those sort of widgets would be great short-term items that would extend Domino even further into the web. Heck, it might even make life easier for the DWA access design team – they won’t have to code/test that sort of thing themselves in the DWA template.

hcl-bot · October 17, 2005, 2:44am

Subject: RE: Support for the xml-stylesheet processing instruction

“But for something like forms, what would you get out of it other than server performance improvements?”

Sections, Tabbed tables, action buttons/bars, even dialog boxes – think of the client-based development tools available that don’t translate to the web without a truckload of custom AJAX/CSS/JavaScript hacking (which we both know all too well). Given a proper stylesheet/XML client-side transform model, these comparatively advanced UI concepts translate to web clients as high-performance tools that would be a breeze to build in the Domino environment.

Think of all the extra stuff that goes into writing, say, a date picker. If Lotus started focusing on delivering those kinds of tools to Domino developers, the platform could become the premier tool for building sophisticated web applications. Doing it via an XML/transform model protects the design investment, since those can be ultimately ported into Websphere environments with minimal trouble.

Of course, this was a scheme that I proposed, oh, nearly 4 years ago. So a lot’s happened since then (Workplace Designer?) so maybe it’s not that valuable anymore. But it would have been incredible if it had started 4 years ago and was hitting the market now.

hcl-bot · October 17, 2005, 10:48am

Subject: RE: Support for the xml-stylesheet processing instruction

Ah, the WebSphere aspect gives a different light to this More web widgets would be fantastic (I think I said this before, but DWA has several nice ones).

We’ve already got a DHTML rendering of sections in ND6, but there is a whole lot more that could be done. You could build drag-and-drop functionality pretty easily, right-click context menus (DWA again!), and a whole lot more. Obviously there would be significant complexity here, but there’s no reason you couldn’t effectively recreate the entire Notes client UI capability list for web browsers.

Pieces of this sound very similar to .NET… except it would all be based on open standards.

All of this, of course, would be a HUGE undertaking… but what an undertaking it would be.

hcl-bot · November 21, 2005, 11:27am

Subject: RE: Support for the xml-stylesheet processing instruction

Wow, I haven’t read this forum in a few months and it looks like a lot of discussions have been going on concerning dxl. This is great!

Dick, to answer your question I was NOT referring to having an xslt file on the client. The <?xml-stylesheet?> processing instruction will pull the file from the server. It is the transformation itself that will be done client-side. My thought was that it would be really simple for you guys to add the ability to specify an xslt file to be used by a view and if a ?ReadViewEntries was done, then an <?xml-stylesheet?> processing instruction would be automatically added referencing the said xslt file.

I, personally, would prefer the xslt be referenced via an url command such as:

http://server/db/view?ReadViewEntries&xsl=myxsl.xsl

If we could get this, then most browsers now a days will support this and do the transformations client-side. Then, later on you can give us the ability to have server side transformations much like .net gives with the asp:xml control.

hcl-bot · November 23, 2005, 11:56am

Subject: RE: Support for the xml-stylesheet processing instruction

Thanks, Jack, and everyone else contributing to this part of the discussion. I get it now (I don’t do nearly enough web development A couple of follow-ups for everyone:Would this mechanism need to support other types of stylesheets other than XSLT (e.g. CSS)? (As you know, you can do some basic styling of XML with CSS, I just don’t know if anyone does that in practice.)

Is it acceptable for XSLT stylesheets to be File Resource design notes? Does anyone foresee any problems with that approach? (There is a Domino design element specifically for CSS style sheets, and I’m not sure I fully understand why it merited a design element type all of its own, as opposed to a file resource.)

If the answers are Yes and Yes, additional questions are raised. At its simplest resolution, using the URL argument model, 2 arguments are potentially needed, one to specify the style sheet type (so we know which type of design note to generate a reference to), and the other the style sheet name.

Dick

hcl-bot · November 24, 2005, 3:29am

Subject: RE: Support for the xml-stylesheet processing instruction

My 2 cents…

XSLT is fine for now. Anyone looking to use this overall approach ought to be willing to venture into serious transforms. Supporting the CSS technique is a bit like worrying about supporting the tag.

File Resources are fine, unless the IDE team wants to put together a serious XSLT editor. And if they say they do, please thump them on the nose for me and tell them to get to work on improving the Lotuscript IDE first (you know, like being able to click on a routine to open the source code, or hover over a constant to find out the literal value, or show custom classes in a proper tree structure.).

(Just so everyone understands, I’m not really wishing abuse on the IDE team. I’m just saying it would be ridiculous to add a new, more sophisticated code parser to the interface without improving the one that has VASTLY more usage.)

hcl-bot · October 17, 2005, 9:32am

Subject: Another mini-poll on ?ReadViewEntries (?SearchViewEntries, ?ReadDXL, etc.)

To summarize this topic so far, a list of of potential new XML/DXL features was given, and the ones with the most interest seemed to focus on Domino URL commands – extensions to ?ReadViewEntries and possible new ones like “?SearchViewEntries” and “?ReadDXL” (to be clear, these do not exist today.)In speaking with colleagues, the question has been raised as to whether URL commands are the proper avenue for additional XML/DXL features, or if perhaps we’d be better off with a Web Services API for this sort of thing. I’d like to hear about how people feel about this, and what they see as the pros and cons of URL commands vs. a Web Services interface.

A Web Services API would not need to be limited to the generic sorts of things you can do with ?ReadViewEntries and the proposed URL commands. It might also be a proper avenue for new APIs for basic processing of mail, C&S, and contacts, and it could grow from there. Let’s hear your thoughts on this.

hcl-bot · October 17, 2005, 10:43am

Subject: RE: Another mini-poll on ?ReadViewEntries (?SearchViewEntries, ?ReadDXL, etc.)

I think you’ll probably get a pretty solid “give us ?SearchViewEntries at least” reply to this one, Dick. Part of that is because it’ll be “free” (once you define a view, you get a predictable output, without additional development effort) and there’s no need to change the code one already uses to handle the return from ?ReadViewEntries.

Not that anyone will complain about having a generalised API for creating DXL or custom XML, but there are going to be questions about relative performance and additional development effort. Would the proposed web services API be something you’d want to run in the high drain parts of an application (views in particular, which have a high signal-to-noise ratio and, in general, a higher hit rate than individual documents)?

hcl-bot · October 17, 2005, 11:02am

Subject: RE: Another mini-poll on ?ReadViewEntries (?SearchViewEntries, ?ReadDXL, etc.)

I’d vote to definitely extend the ?ReadViewEntries functionality and add ?SearchViewEntries (in that order).

?ReadDXL I’m honestly not dying for – but that’s just me.

As for a Web Services API: I’m with Stan. It would be nice as an addition to the Read/SearchViewEntries functionality, but I’m guessing there would be serious performance implications and Lotus/IBM development effort here. If the dev effort is costly, I’d much rather see some AJAX-based UI widgets from DWA make it into the Domino core.

Oh, and if nobody’s seen it yet, here’s a fun Google-Suggest-style AJAX snippet (since we’re talking about AJAX a bit):

http://www.nsftools.com/tips/ZipLookupTest.htm

hcl-bot · October 18, 2005, 1:53am

Subject: RE: Another mini-poll on ?ReadViewEntries (?SearchViewEntries, ?ReadDXL, etc.)

?SearchViewEntries is a must.

As far as Web Services implementations go, I have two responses…

If I want to go that route, do I need you to do it? I haven’t had the opportunity to get into Domino 7’s web services stuff, but now that you mention it, maybe I can just use that route to achieve my goal. Which leads to the second point…
What’s the performance implication of one vs. the other? If the services approach at an application level (ie: I write it) is appreciably slower, then I’d like you to do it. If you do it, I’d like to see you take the route that will result in better performance.

I’ll also add that it’s generally easier for novice developers to deal with handling an URL, so you’d likely get more use from something like ?ReadViewEntries. But I don’t think any of us in this discussion are looking for something targetted at novice developers. Heck, that might be a reason to stay away from the URL approach.

Honestly, the web services approach never occured to me, and the more I’m thinking about it, the more I like it. It’s just the performance aspects that will drive it. But if I can put together services in ND7 for this, and they’re faster than, say, agents would be, then it might be exactly the way to go. (Can I output a DXL pipeline direct to an HTTP response? I don’t even know. Eric?)

hcl-bot · October 18, 2005, 10:17am

Subject: RE: Another mini-poll on ?ReadViewEntries (?SearchViewEntries, ?ReadDXL, etc.)

Here’s my thoughts on a Web Services approach (I’ll discuss within the context of views since we’re revolving mostly around that):

We could almost do all of this view navigation fetching now at the application level with a run-as-web-user agent and web service, making heavy use of the NotesViewNavigator class. The biggest problem (and this is currently faced with any WS approach) is the fact that a script agent can only “Print” 64K back to the browser. Depending on your character set you’re probably talking ~30000 characters or so.

If this limit were lifted there would be significant capabilities opened up to Domino’s web delivery. You could bytestream large files, pipe big DXL streams, etc.

Getting back to the view navigation example I was referring to: The approach I described is frighteningly slow – instantiating the NotesViewNavigator class itself kills your CPU for several seconds on any decent-sized view. Overall it’s definitely not a possibility for any real production use. ?ReadViewEntries, on the other hand, is incredibly fast.

If a WS approach was taken by IBM/Lotus it must focus on performance (and also address the large-data-output limitation present today). I see it being at least slower (if not significantly more so) than a low-level URL command. For that reason I think the URL commands must be the primary means of adding these capabilities. WS capabilities would be a nice extension, but just as that – an extension.

I definitely wouldn’t want to see IBM build in Web Services extensions for mail and calendaring off-the-bat – I think the bulk of the capabilities there we could do today with reasonable performance.

hcl-bot · October 19, 2005, 3:36am

Subject: RE: Another mini-poll on ?ReadViewEntries (?SearchViewEntries, ?ReadDXL, etc.)

So Dick, what we’re all saying is: whatever makes it fast!

Or, put another way, don’t make it work slow like agent calling approaches do. (As much as I love agents, ?RunAgent as a web delivery strategy is catastrophic.)

hcl-bot · October 19, 2005, 1:57pm

Subject: RE: Another mini-poll on ?ReadViewEntries (?SearchViewEntries, ?ReadDXL, etc.)

I’d have to mostly agree with everyone on this ‘min-poll’.?SearchViewEntries would be great and is at the top of my list right now and could be used right away, with little change to current application design.

As far as employing a WS to fetch DXL, I could do this myself as needed, but as Eric mentioned we are limited to what an Agent can handle. So I would take it for free if it were quick and reliable and saved me the hassle. I think this type of interface is better than ?ReadDXL and have implemented something similar in the OpenNTF project DXLPeek.

In reqards to the other Web Services features you mentioned, I’d rather hear that a database’s design and documents could be fully exported and imported as DXL before we got into all that.

Thanks.

–Mac