What XML/DXL features would you like to see post-7.0?

hcl-bot · November 21, 2005, 12:37pm

Subject: RE: Another mini-poll on ?ReadViewEntries (?SearchViewEntries, ?ReadDXL, etc.)

?SearchViewEntries is an absolute must have

After that, I would prefer having extra url arguments to the ?ReadViewEntries command. More specifically &xsl=myxsl.xsl.

I would also like to have a way to navigate “back” as mentioned elswhere in this thread. I think someone mentioned having a &navigate={prev|next} and that would work for me.

WS for now is just a nice-to-have for me.

hcl-bot · July 25, 2005, 5:29pm

Subject: What XML/DXL features would you like to see post-7.0?

I can’t add more than Jack Ratcliff’s comments on ?SearchViewEntries. Big thumbs up if you can do it.

?ReadDXL (or ?ReadNote or what have you) would need to be disabled by default and enabled on an element-by-element basis (at the form level for documents), since hide-whens and form formulae have so far been legitimate security in web applications.

I’d like to see DXL import-to-memory, since it would (for example) allow a fairly sophisticated mail message to be created and sent without the need to create and delete stored documents. (I’m already using DXL to create a highly-formatted newsletter, but the ability to personalise the content is very limited given the number of documents that would need to be created/destroyed.) I think a single-document memory object should be all that’s required – I can’t think of too many good uses for unsaved views, for instance.

SetConvertNotesBitmapToJPEG/PNG would be a huge help too.

I’ve filtered at the DOM and SAX level, but I can see where turning things off before export would be a lot more efficient.

Two things I’d like to see added to the DOMParser – an InsertBefore (or after) method for appending new nodes, and a native means of creating a cross-document clone. There’s a project described on Rocky Oliver’s blog that illustrates the need:

http://www.lotusgeek.com/sapphireoak/lotusgeekblog.nsf/d6plinks/ROLR-6E2V8G

Rock’s eventual work-around is ingenious, but truly kludgy compared to a native method.

hcl-bot · July 26, 2005, 1:29am

Subject: 13 - Ensure that DXL can represent all elements in Notes databases

…and if you make this Dick, I won’t bother your with other points

hcl-bot · July 26, 2005, 1:12pm

Subject: What XML/DXL features would you like to see post-7.0?

I have a project on OpenNTF.org that would definitely benefit from ?SearchViewEntries. This would allow developers to process view data that has been searched in the same way as if it hasn’t been searched. I have blogged about this here > http://jackratcliff.com/jratcliff/blog.nsf/dx/SearchViewEntries.htm

I definitely like the idea of ?ReadDXL or ?ReadNote as some have referred to it. I need to think some more on how I would recommend it to be implemented.

One enhancement that I would love to see (if it hasn’t already been addressed) is the issue with &RestrictToCategory. I think it would be hugely beneficial if the other optional view arguments worked in conjunction with &RestrictToCategory. The main ones I would like to see would be &StartKey, &ResortAscending, &ResortDescending. By supporting these arguments WITH &RestrictToCategory would give us what I call “mini-views”. You can see more of my thoughts on this one here > http://jackratcliff.com/jratcliff/blog.nsf/dx/RestrictToCategory.htm I know this one is probably difficult and I’m not sure how you currently handle view indexes now, but perhaps the domino server could internally build separate indexes for each category behind the scenes. So, to domino it’s almost like there are separate views but we the developer don’t build these, domino does based off of the view selection formula PLUS the RESULTS of the category column.

Another enhancement that would be really nice would be if you could include in the viewentries element an attribute that told you what the “previousstart” should be (probably based off of the server docs lines per view and overridden if the &count arg is included). The reason for this is because of Readers fields and categorized columns, it’s not easy to determine where to start a view when a user wants to do a “previous” navigation. Doing a Next is pretty easy since you can just use the position attribute of the last viewentry as your starting point. Again, since Readers fields affect which documents a user can see, I’m not sure if you could determine this value based on the current user or not, but if you could, that would really be cool. I’ve currently come up with a workaround of sorts by passing my own &PreviousStart argument in the url to keep track of this myself.

Anyway, if you want to see what I’ve done with using xml/xslt for domino views and outlines you can download the project from here:

http://www.openntf.org/Projects/pmt.nsf/ProjectHome?ReadForm&Query=Domino%20Web%20Tools

And here’s an online demo of it in action:

http://jackratcliff.com/jratcliff/dwt/dwt-demo.nsf

Again, the way my code works is that it reads in the xml for a View or Outline using ?ReadViewEntries or ?ReadEntries respectively and then uses XSLT to transform this information into html to be displayed in the browser. For views, I also do a ?ReadDesign to get the design information for the view.

hcl-bot · July 29, 2005, 12:36pm

Subject: Exporter Output Filtering - does this pique any interest?

Of the items in the list, 12. DXL Exporter Filtering, is one that I expected to be in high demand and therefore have previously thought about in more detail. There hasn’t been much demand for it in the replies so far. I’m providing a little more detail to see if that will cause the idea to get more traction. Here is a summary of what I envisioned:Provide a generic way to filter out elements by several different criteria from a DXL output.

Provide the option of leaving placeholder “replacement elements” (in a different namespace) where filtered elements were removed. An attribute of each replacement element would be an xpath-location that uniquely identifies what was filtered.

Provide a generic way to export only specific element(s) via a (probably limited) set of XPath expressions.

Note that the design is such that the 3 capabilities may be combined to export most of the elements of a note, leaving out ones that are not immediately needed (graphics and attachments are the typical examples); parse the DXL for replacement elements to obtain a list of filtered things; re-export the note to get at parts that were initially filtered.

By way of example:

Assume that a full DXL export of some notes(s) contains a note with UNID 85256D020057187285256CF60081437A that has an attachment. Assume the DXL contains this item for the attachment (much of the Base64 content has been omitted):

20000402T135618,64-04
20000927T162822,00-04<filedata

W05vdGVzXQ0KDQpEaXJlY3Rvcnk9Zjpcbm90ZWZpbGUNCktpdFR5cGU9MQ0KSW5zdGFsbFR5cGU9
. . .

MSwNCkRFQlVHX1RSQVBMRUFLU19GSUxFPWY6XG5vdGVmaWxlXGJsa3MubmJmDQpERUJVR19CVFJF
RV9FUlJPUlM9MQ0K

Via capability 1., you ask (via an API method) to filter all item elements that contain a descendant file item, and you ask for replacement elements. Now, in place of the item above, the DXL contains this (capability 2.):

<dxlproc:filtered-element

element-name='item' descendant-name='file'

xpath-location="document[noteinfo/@unid='85256D020057187285256CF60081437A']/

			item[descendant::file]"/>

Now, via capability 3. you ask (via an API method) to export the note(s) but restrict the content to data matched by the XPath expression document[noteinfo/@unid=‘85256D020057187285256CF60081437A’]/item[descendant::file]. The result of the export would contain only the shown above.

Would there be any interest in such a feature? If yes, which features are more important than this feature?

hcl-bot · July 29, 2005, 1:55pm

Subject: RE: Exporter Output Filtering - does this pique any interest?

Now that you have described this enhancement in more detail, I would definitely be interested. However, I would still rank it lower than these three.

?SearchViewEntries
Full support for &RestrictToCategory
support for xml-stylesheet processing instruction

hcl-bot · July 27, 2005, 10:43am

Subject: What XML/DXL features would you like to see post-7.0?

“URL” class in LotusScript and Java: At a minimum, this would work in conjunction with NotesStream / Stream so that those classes could read a stream of input data from a URL. Major use would be to get input for XML parsers, DXL Importer, or XSLT Transformer from URLs.Yes, please, a Lotusscript URL class. And ditto what Nathan Freeman said about this.

I’m not sure how useful a Java url class would be because there’s some basic functionality for that in the java.net package, but a Lotusscript one would be sweet. I need to get stuff from urls sometimes, and I cannot touch the file system, so I have to use the url class in java.net and it’s a but clumsy when, as is usually the case, the rest of the work has to be done in LS.

hcl-bot · July 29, 2005, 9:30am

Subject: What XML/DXL features would you like to see post-7.0?

My top three would be 6, 1, and 2 in that order.

Regarding security for ?ReadDXL, Domino’s security must be honored. Thus if you are a Designer or Manager, than you can get DXL for a view element, for example.

If you have access to read a document, you can get the DXL for that document. This implies that data otherwise hidden on the web with the use of hide/whens would be available using ?ReadDXL. This is still consistent when you consider that 1) hide/whens are not security (even on the web) and 2) DXL is not just about web access. If you can access data via the Notes client, you should be able to get that data via ?ReadDXL.

A trickier problem that Mac Guidera mentioned is what gets read by ?ReadDXL when the “Hide Design” feature of a db is enabled. Obviously, ?ReadDXL for any design element is off limits. ACL and other database parameters should still be available. But with documents, it gets really interesting: Form and Field metadata must still be included along with the data when using ?ReadDXL on a document. Without field names, the “data” becomes just text. I’m not convinced of what to do with form names, but my instinct is to include it.

I really like this list and I think you’re heading in the right direct.

Thanks,

Ian

hcl-bot · July 29, 2005, 11:18am

Subject: Full &RestrictToCategory support - anyone else?

Anyone else want all of the optional URL args to work WITH &RestrictToCategory? I think this would really be awesome and would cut way back on the number of views that designers have to create/maintain.

Here are some of my thoughts on this issue:

http://jackratcliff.com/jratcliff/blog.nsf/dx/RestrictToCategory.htm

I know I have heard it mentioned that this would really be difficult to do and I for one do not know all of the internals on how domino currently handles view indexes. Perhaps the server could create separate indexes per category behind the scenes for us.

Don’t views currently work this way: If you have a view that has 3 columns set to sort, then there would be 4 indexes maintained? One for the view as designed and 3 more for the 3 columns that are currently being indexed. I know the concern is that if you then add a categorized column AND allow for the ability to do the sorts within the category, that the indexer will really bog down when updating the indexes for this view. But will that really be the case if, behind the scenes, domino creates a bunch of small views/indexes for each category?

If the concern is the possibility of a categorized column having so many categories that the entire server suffers by creating/maintaining all of these extra indexes, then perhaps we, the developers, can flag which categorized views we want the server to spend the extra overhead on. This flag can be turned off by default. Then, if the flag is turned off, then Notes and Domino will not show the sort icons on a column when a single category view is being displayed (since it won’t work anyway due to the extra indexes not being maintanined). We’ll also need to have this flag included in the output of ?ReadDesign for those developers who handle there own html generation of domino views.

hcl-bot · October 5, 2005, 12:38pm

Subject: RE: Full &RestrictToCategory support - anyone else?

Jack, and other ?ReadViewEntries fans that might be listening. I was playing around with ?ReadViewEntries and came up with what might be a possible workaround for wanting to mix &RestrictToCategory with &StartKey. In all likelihood, this has been thought of before, but just in case…

I set up a categorized view named TopicsCat where column number 0 is categorized. In the Column Sorting properties it is defined as Sort: Ascending, Type: Categorized. Now here comes the possible workaround. I also checked “Click on column header to sort”: Ascending and clicked on “Secondary sort column:” Topic, Ascending, where Topic is a field that I want to sort on within a category.

Now I can do a request like

http://server/db.nsf/TopicsCat2?ReadViewEntries&ResortAscending=0&StartKey=D

The TopicsCat2 view would normally automatically sort by ascending category, and &ResortAscending=0 asks to do the same thing. But it has the wanted side effect of doing the secondary sort by Topic. What the URL above doesn’t do is restrict to one category but it does start with the first category >= “D” and perhaps if used in conjunction with Count would be a possible workaround. Now that I’ve written this all up I can see that the problem is that there is no way to start at a particular subkey (Topic) value, and their are an indeterminate number of Topics within the category.

hcl-bot · October 5, 2005, 4:33pm

Subject: RE: Full &RestrictToCategory support - anyone else?

Thanks for looking further into the issue, Dick. While I do like the idea, I’m not a huge fan because (a) the functionality already exists to do this (albeit requiring numerous views) and (b) I think the work involved by the dev group would be fairly extensive.

I also want to expand on my request for pagination information in ?ReadViewEntries. I’ve got a great replacement for the Java view applet that is completely AJAX. It’s been in production for 7 months now, and has been battle-tested by several thousand Internet users with great results. It has virtually all of the advantages of the view applet, including:

Multiple views on one page
Various parameters (restrict to category, every-other-row shading, “show select column”, etc.)
“Starts with” functionality
User-definable “count per-page”
Fully CSS-enabled rendering

It pulls down the view design XML, renders icons in columns, allows click-sorting, etc.

It works with IE 5.0+, Netscape 6.1+, and Firefox 1.0+. No ActiveX required, etc., etc. I’d love to send it to you if you’d like to have a look.

The only real disadvantage with the application is with navigation. You currently have to navigate with Next/Previous buttons. This is where the ?ReadViewEntries pagination info would really come in useful. Here’s why:

How my naviation works

Next Page - the “Next Page” button simply looks at your last entry of your current result set and sets your &Start to that. No big deal. The effect is virtually identical to how the HTML-rendered views work. It also adds your current &Start parameter to an in-memory stack (more on that in a second).

Previous Page - the “Previous Page” button doesn’t have a clue as to how to go “back” 50 entries (or 100, or whatever your current &Count is). It simply doesn’t know what your new &Start needs to be, unless it can always assume there are no readers fields involved. So what it does is pop your last &Start parameter off of the stack I mentioned above. This works great except when you use the “Starts with” functionality offered by &StartKey. In that case “Previous” simply jumps you back to the beginning of the view.

How this can be vastly improved with (I think) almost no effort from IBM/Lotus Dev:

If we were to receive a “priorpagestart” attribute in ?ReadViewEntries then we could have true “Previous page” functionality, regardless of our current &Start parameter (or &StartKey, for that matter) or how we got there. I mentioned this in my other post: the logic’s already there with the HTTP engine – HTML views calculate this attribute for their own “Previous” button – so I think this could be added to ?ReadViewEntries pretty easily.

How this can be perfected

“Page X of Y” navigation - this is a simple concept, visible on virtually every web-based record navigator that we’ve all seen.

The problem with implementing such a solution with ?ReadViewEntries is that there is no way of knowing how many entries are in the view that are visible to the current user. I’ve done some hunting and experimentation, though, and discovered an undocumented attribute called “rangeentries” that appears to do so – sometimes.

If you use ?ReadViewEntries with both a &StartKey and &UntilKey parameter, you are returned a “rangeentries” attribute, which appears to be the number of entries visible to the current user for the result set.

So theoretically if you know the first and last entries in the view you can use &StartKey and &UntilKey to receive this attribute, but there are a few roadblocks:

First of all, you’ve got to get the first and last entries in the view in order to know what parameters to use with &StartKey and &UntilKey.

Second of all, &StartKey only works with text or date-valued columns (i.e. any views sorting off a number column are out-of-luck).

Thirdly, even if you do manage to get an &StartKey value (say, with a script agent) and try to assume an end-of-alphabet &UntilKey of “&UntilKey=~”, you won’t get any category entries that are “not categorized”.

In order to know the size of a view we’ll need the “rangeentries” attribute to always be returned (or a new parameter with a similar effect). Ideally it would also take into account any &Expand arguments in affect. E.g. if a collapsed, categorized view initially has 20 visible entries and a category is expanded that has 5 more inside of it, the attribute should return a count of 25.

If we can get that, then the only thing missing would be the ability to jump to a particular page, e.g. “jump” from page 6 to page 10. In order to do that, though, we would need to know the &Start parameter to use. In order to take this into account with reader’s fields the ?ReadViewEntries result set would have to return some page info, e.g.:

<— notice the sudden jump? readers fields are hiding some docs!

This could be a performance killer on huge views, though. But here’s the catch: we don’t need all of this page information – only some of it. E.g. A typical navigation interface is similar to:

Page: 1 2 [3] 4 5 … 27

We only need a bit of info for this – the prior 2 pages to the current page, the next 2 pages, and always the first and last.

One final note: the view applet uses some other undocumented arguments such as &Navigate, &SkipNavigate, and &EndView=1. I’m not sure if those would address any of our concerns, but if they could then simply documenting those would be a great help.

Whew! I’ve written a book! Thanks again Dick, and to anybody else who is taking the time to read this.

hcl-bot · October 6, 2005, 7:32am

Subject: RE: Full &RestrictToCategory support - anyone else?

I will get back to you on your offer to let me have a look at your code. It sounds pretty cool.

I understand the need for a priorpagestart attribute. I was wondering if it would be better still to provide the capabillity to navigate backward. Maybe this would be implemented via a &PrevCount=n argument, similar to &Count=n. Or maybe better, a &Naviagate={prev|next} where the default is next, like today, and it could be paired with &Count or other arguments. How does this rate vs. the proposed priorpagestart attribute? I haven’t really considered the implementation cost of either way, but it seems like the &Navigate way would perform better at an educated guess.

I will take the rest of your suggestions under consideration as well.

hcl-bot · October 6, 2005, 10:27am

Subject: RE: Full &RestrictToCategory support - anyone else?

Sure thing with the code - just e-mail me at erik&archondev&com (replace with appropriate characters) and I’ll wrap it up and send it to you. We developed it out of a need to replace the ailing Java applet while maintaining a great deal of granular control over layout and keeping server overhead low. It chews up some client-side CPU when rendering large result sets, but that hasn’t been a big deal in production for our users.

I don’t know what &Navigate currently does with ?ReadViewEntries (I’ve tried to figure it out from watching the Java applet, but it’s a bit strange in behavior). While your {prev|next} proposal would make navigation a bit easier I think a priorpagestart attribute would be a little more flexible overall. It would keep any navigation requirements on our (the end developers’) shoulders, while keeping the data delivery on yours.

I also think priorpagestart would be significantly easier for IBM/Lotus Dev to develop and test, as there wouldn’t need to be any new looping coded/tested/etc. to handle generating results in the other direction as there would with {prev}. I can see things getting very complex there, especially with &StartKey/&UntilKey, etc, whereas simply adding a priorpagestart attribute would likely be a much easier task. Hopefully it would be soooo easy that it could even be dropped into a 7.0.x MR.

hcl-bot · October 7, 2005, 10:12am

Subject: RE: Full &RestrictToCategory support - anyone else?

Wouldn’t changing the way the siblings, toplevelentries and related properties are displayed fix view navigation on the web?

I mean, does a notesview object is LS tell you that there are 100 notesviewentries when the effective user can only read 50 of them? IF it does, then it ought to be fixed too.

If the TLE count, sibling count etc. were correct for the effective user, then &start would be all that’s really needed for easy back/next navigation.

hcl-bot · October 10, 2005, 11:10am

Subject: RE: Full &RestrictToCategory support - anyone else?

You can’t change how those properties work currently because it would break backwards compatibility rules. Consider an app with a view where I (the user) can only see 30 docs out of 523: