I have a brand new Domino 8.5 server, just installed and just finished configuration; which simply went away and hung. I had a weird conflict error on the java Domino console about conflicting streams (which I didn’t catch as I’ve never seen it before), then the host server crashed and rebooted.
Cute.
After reboot the domino server would not stay started, it went from running to stopped within 10 seconds. No logging, no diagnostics and I’m using a VM so I couldn’t see the physical console.
Eventually I restarted in user mode and the server was fine.
Immediately I went into the services in the registry and removed the “-jc” “-c” from the service and the server started and ran fine as a service. Connection by remote admin on the Domino Administrator only. Which is not what we want.
In the end I reinstalled the domino server just to remove possibilities of code corruption in the reboot. Same issue on start.
So to go back to basics I removed the directory and put back the initial notes.ini. Re-ran the server install as a first server install using the id’s created first time round. Hey presto, server works.
I have a backup of the Directory so I then went through a painstaking 2 hours of reintroducing all the configuration settings I had just put in place, with frequent backups and server restarts at each key point.
My server is working perfectly again with all configurations in place.
My questions are:
Why?
What caused it?
How can I be sure it won’t happen again?
Because until I can get answers to these questions I’m not going to be happy handing this over to the customer even if they have tested it to death.
Subject: Saw similiar symptoms with 8.0.2 test server
Can’t tell you why, but can tell you how we fixed similar issues with 8.0.2.
Deleted \Lotus\Domino\jvm. Note purging the contents of the jvm directory didn’t work. We had to delete directory itself.
Reinstalled the code over existing code. Configuration stayed intact.
In the end I suspect something funky was up with permissions on the jvm directly, but since we never experienced the problem again I never found the true root cause.
I have finally identified what the problem was. I’m split between slapping myself round the head and unamused at the lack of diagnostics data and the “helpful” way that the Java Console engine blows the server away on startup.
I found I had not put back One configuration value. The Server Console IP address.
The configuration of the servers (yes I know I should have looked), is different to what I was led to expect. There are two network adapters and the Second adapter is Natted to the IP address given to me for the server.
As such I, slap, entered the given NAT address in the Server Console field of the Domino Directory server document.
Bang, one server on it’s back and unrecoverable with no diagnostic information.
To resolve the issue I have had to put the TCPIP_ControllerTcpIpAddress entry into the notes.ini (just to be sure), as well as entering the “Correct” address into the Domino Directory server document Server Controller field.
I have also had to use the TCPIP_TCPIPADDRESS entry to force the Notes Port. Because the nice guys who natted my IP address used the Secondary Port and the Domino server binds to the Primary IP port by default.
Hence my initial problems with an invisible server controller.
This is consistent on D8.5.0 and D8.5.1, I have had the same problem on both.
Personally I’d like the Server Controller code to be just a little friendlier and get some diagnostics out of the server instead of just becoming a dead body with no apparent cause.
But at least I have answered my 3 questions in the original post.