Decoding Non-English Characters in LotusScript Agents

Blogs
, , , , , , , ,
1

Have you ever tried to send JSON data with non-English characters to a LotusScript agent only to have them appear as garbled text? It’s a frustratingly common issue, and if you’ve been working with Domino for a while, you know it’s a persistent problem. While English characters pass through just fine, anything else gets scrambled. In this post, we’ll explore why this happens and, more importantly, a practical workaround to solve it.

The Root of the Problem: LMBCS vs. UTF-8

The issue stems from a fundamental conflict in character encodings. When you send JSON data via HTTP POST request, modern applications typically use UTF-8 encoding, which supports a vast range of international characters. However, when this data reaches a LotusScript agent on a Domino server, it’s not handled as you might expect.
Domino has a legacy character encoding called LMBCS (Lotus Multi-Byte Character Set). This encoding has been the native standard for Domino since its early days, and for reasons of backward compatibility, it hasn’t been changed.

When an agent runs and tries to read the REQUEST_CONTENT CGI field, it interprets the incoming data as if it were encoded in LMBCS. Since the data is actually in UTF-8, this misinterpretation leads to the garbled characters you see. The raw data itself isn’t corrupted; it’s just being read with the wrong lens. This is a day-one behavior of Domino that cannot be changed without breaking countless legacy applications.

The Solution:

Fortunately, there’s a reliable workaround that allows you to correctly process UTF-8 data within your LotusScript agent. The key is to bypass the automatic LMBCS conversion by handling the incoming data as a byte stream before parsing it.

Here’s the step-by-step process:

  1. Retrieve the Raw Data: Instead of directly accessing REQUEST_CONTENT, you need to read the raw, unconverted data from the agent context. You can do this using the GetItemValueCustomDataBytes method of the NotesDocument class. This method retrieves the data as a byte array, preserving its original UTF-8 encoding.
  2. Use a NotesStream Object: Once you have the raw bytes, write them into a NotesStream object. This object acts as a buffer for the data, which is essential for the next step.
  3. Parse with NotesJSONNavigator: The final and most crucial step is to use the NotesJSONNavigator class. This class is designed to handle UTF-8 data natively and can read the contents directly from the NotesStream object. It can then parse the JSON content correctly, with all your non-English characters intact.
    By following these steps, you can correctly parse the JSON data and work with it as intended. You can even use the Stringify method of the NotesJSONNavigator to print the data, and it will now appear correctly in your response.

Here is the complete LotusScript sample code that demonstrates this workaround:

Sample Code

Sub Initialize
	Dim session As New NotesSession
	Dim document As NotesDocument
	Dim navigator As NotesJSONNavigator
	Dim stream As NotesStream
	Dim customdata
	Set document = session.DocumentContext
	customdata = document.GetItemValueCustomDataBytes("REQUEST_CONTENT", "Text")
	Set stream = session.CreateStream
	Call stream.Write(customdata)
	stream.Position = 0
	Set navigator = session.CreateJSONNavigator(stream.Read)
	Call stream.Close
	Print {Content-Type:application/json}
	Print navigator.Stringify()
End Sub

By adopting this method, you can build more robust and international-friendly integrations with your LotusScript agents, ensuring that your data, in any language, is handled correctly.

Disclaimer

This solution has been tested in a controlled environment and is shared as a reference implementation. It is recommended to adapt and thoroughly test the code in your test environment before deploying it in production. Please work closely with your development and QA teams to ensure it meets your organizational standards. This sample script is not officially supported by HCL Product Support. Please use it at your own discretion.

6 Likes