Try manually setting the heap sizes in the notes.ini.
NTS_Java_Parms=-Xms64M -Xmx512M
Not sure why the value would be -Xmso4k - that is certainly not formated correctly. Possibly there are some old notes.ini settings being picked up. Try the above - should work.
I installed Domino + Traveler server from the scratch on the same physical server and got working traveler on newly Domino system. Conclusion: server’s OS is ok.
Tests with clean installation notesdata and production notesdata showed that it was notes.ini fault. Manual notes.ini comparison and adding NTS settings did not work for me. Finally
I used notes.ini from clean installation, fixed as necessary and got production working. The following settings are in use now:
NTS_TEMPLATE_INSTALL=0
NTS_AUTO_CONFIG=true
NTS_ENABLE_WEB_CLIENT_INSTALL=true
NTS_MAX_HTTP_THREADS=100
NTS_CLASSES=C:\Lotus\Domino\traveler.jar
JavaUserClassesExt=NTS_CLASSES
Setting NTS_Java_Parms=-Xms64M -Xmx512M is not in use.