MSClustering

I am suprised that there is so little info about using clustered OS (active/passive) since we need to patch/restart our MS boxes every month. Clustered OS would present the server to its clients as never having changed (ie:DNS name, IP address) I have more applications that want to integrate with email (Rightfax, Avaya Unified messaging, SAP) and they don’t really like Domino Clustering (works great for users! but not these apps).

got bad news today when a consultant dug up technote 1165986. all I can say is “say it isn’t so”!!

anyone familiar enough with the NSD process to figure out the ramificatoins of a “bad” state? I don’t need automatic failback,I can let the servers A & B take turns being Active for a month, so if this is something I can correct manually, please let me know what it is and what i would need to do.

Anyone have any other resources I can try? that technote doesn’t exactly “volunteer” itself at ibm.com!!

thanks,Craig

Subject: MSClustering

Check out IBM Redbook, Lotus Domino R5 Clustering with IBM eserverXseries and Netfinity servers. SG24-5141-01. It’s R5 but I found it helpful. I’ve also found some information on Microsoft web site and our hardware vendor site.

Subject: RE: MSClustering

Rebecca- does this mean, that despite what that technote says, you have successfully clustered w200x and are running dom6.5 on it for OS failover? I have those documents and v6 equivalent. they all say that I should cluster OS to accomodate the other applications… but the technote is much more current (4/15/2004) and I don’t want to buy two copies of MS2003 enterprise and $50,000 of SAN to fail miserably.

Just looking for someone with positive experience to disagree with the technote.

anyone?

thanks

Subject: RE: MSClustering

Not yet. We are looking at running domino clustering with Windows clustering and storing data on the SAN. From what I understand at this point is that MS cluster will work with Domino 6.x. However it is unsupported. Bottom line,recommendation that we’ve received so far is to run one or the other. The NSD process hangs when it is invoked in V6 with MS clustering, which will eat up resources, which may cause the server to,well,you know the story.

Subject: RE: MSClustering

Rebecca- sounds like you know more about Lotus, than I (by far) what is this NSD? I plan on using only MSCS not Domino clustering. I have some other applications that don’t failover using Domino(and the table in the lotus pdf recommends using OS clustering!)

bottom line for me is to know that when my Active #1 server crashes, the Passive#2 takes over. I plan to tell #2 to remain the Active, even if #1 comes back online. If I need to intervene on #1’s behalf, then that would be acceptable. I just need to know that the first failover will occur- not get caught in some kind of NSD bad trip (listening to Woodstock)

Subject: RE: MSClustering

NSD is the Notes/Domino application that collects information after a crash to assist customer support in diagnosing the cause of the problem. What happens is that after the crash NSD is launched and while it is busy collecting information the directories are ripped out from underneath it by the MS cluster server. Therefore, if you want to use MSCS you must disable NSD and Fault Recovery (automatic restart, since you are handing that job to MSCS). Both are done in the server doc / basics tab

If you have a problem with Domino you will not have NSD info to supply to support. You will have to decide whether to live with the problem or disable MSCS in order to collect the info.

Subject: RE: MSClustering

I am running 6.5.1 on an MS cluster and we are running into issues. I have configured Domino according to the IBM Redbook, Lotus Domino R5 Clustering with IBM eserverXseries and Netfinity servers. After about 2 weeks up, I see this error in the log:

05/24/2004 08:37:55 AM Unexpected TCP/IP error func: 0011h error Notes: 1C5B, NTI: 0006h, Stack: 00000000h

05/24/2004 08:37:55 AM Unexpected TCP/IP error func: 0011h error Notes: 1C5B, NTI: 0006h, Stack: 00000000h

Then the server hangs and crashes. Any ideas?

Subject: RE: MSClustering

There is an update to technote 1165986 that is on the way. MS Clustering is supported just not in the same way as it was for version 5.0.x

We are having similar problems with our MS Cluster and Domino 6, and we were flat out told that this configuration was not supported.

Our IBM rep arranged a meeting for us with Jim Rouleau, Eddie Bell and John Woods of IBM and were told that Domino is supported on MS Clustering so long as the MS Cluster Manager is not configured to manage Domino. (Yes, this is a reversal!) MS CLuster Manager continues to manage the fail over for the HW and OS.

So what we were told to do is create scripts to automate the commands one would enter on the Domino server to initiate a safe shutdown or restart on the other server in the cluster.

We used HP services (with support from IBM who signed off on the design knowing we were going to version 6!) to implement the MS Cluster, their servers and their SAN with Domino. HP is now working with IBM to figure out just what these scripts need to do, who will own and support them. I’m hoping IBM will step up and take ownership, but they seem reluctant to do so. If HP takes ownership then I would expect them to take the credit for the fix and rightfully so.

At the same time we asked IBM to go back and update the technote with this information as well as the release notes for Domino 6 versions currently available and those on the way. HP will also have to go back and update a white paper they have on the subject of Domino 6 on using MS Cluster under Win2003.

We also strongly suggested to IBM that they publish a new redbook on the topic of High Availability and Domino (presumably including other OSes such as Linux). IBM would not commit to this but if we all yell at them loud enough they just might give in.

Our meeting with IBM occurred on 07/19/04, so this is pretty current. As some of the issues I’ve outlined in this post are addressed by IBM and HP I’ll post updates to keep everyone current.

I’m also wondering whether the Jim Rouleau that also has a response in this thread is the same Jim Rouleau that I met with, if so he is responsible for the NSD tool and Domino Fault analysis… If so hopefully he’ll add more background on the issue.

Good Luck

Subject: RE: MSClustering

Here is a summary of the interaction between NSD, Fault Recovery and MS Cluster Manager. I will be passing this on to the support folks as an update to the Technote you reference. I hope this clarifies the current situation regarding Domino 6 and MS Cluster Manager.

Best Regards, John

What is NSD?

NSD is the core First Failure Data Capture (FFDC) Tool. In the case of an exception or Panic in the Domino code, the Domino exception handler gets control and invokes NSD to collect diagnostic information on the failure. This information includes environmental data, Process Information, Thread/Process call stacks, Local and Shared Memory Dumps, File System Information, etc… When data collection is complete, NSD will clean up memory, files and processes and generally prepare the Domino Server for a clean restart. (This is true for Windows only. On Unix the cleanup is done internally, but conceptually, the principle is the same)

What is Fault Recovery?

The Fault Recovery feature was new to Domino 6. Fault Recovery enables a server that has terminated abnormally to shut itself down and restart automatically, without any administrator intervention. When Domino encounters a situation which is considered a fatal error, diagnostic information is collected via NSD, each Domino process is terminated and all associated resources are released. There is lots of information on Fault Recovery in the Domino Administration Guide and Release Notes.

What happens when MS Cluster Manger is used for Domino Application Fail-over?

If the Domino Server fails (takes an exception) NSD gets control for FFDC as described above

MS Clustering will take away the file system while NSD is running

NSD will NOT be able to capture diagnostic data about the Domino failure

NSD will NOT have an opportunity to clean up the processes

The Domino Server will be left in a “bad” state and the MS Cluster Manager will not be able to fail back. I have heard some describe this as “ghost” Domino processes on the failed server.

There has been some discussion that NSD and Fault Recovery can be disabled on the server. This may allow MS Cluster Manager to perform application failover (although this has NOT been tested or certified by IBM) However, disabling NSD makes Domino fundamentally “unserviceable” because we capture no data concerning product failures. This is the basis of the statement that MS Cluster Manager is not supported for Domino fail-over.

Furthermore, we believe that the fastest and most efficient way to recover from an abnormal termination of the Domino server is Domino Fault Recovery. This feature allows for efficient FFDC, proper clean-up of processes and other resources and efficient restart of the server.

Can MS Cluster Manger be used for Hardware, Network and Operating System Failover?

Yes. There is no known reason why MS Cluster Manager can not be used for Hardware, Network and Operating System failover and continues to be supported. Domino development does not provide scripts to enable this functionality. If technical help is required in writing these scripts contact your IBM Representative to engage IBM Software Services for assistance.

How can I terminate the Domino server on Windows?

Quit at the console or “nserver -q”
This is the cleanest method to shut down the server. All client activity will be quiesced, all databases will be closed and the server will terminate cleanly.

nsd -dumpandkill
This method uses NSD to first take a diagnostic snapshot of the server processes, memory and environment and then forcibly terminates the server. Any databases opened at the time of invocation of the nsd -dumpandkill will require fixup when the server is restarted. The diagnostic information collected can be used by the support teams to analyze the problem when forced the abnormal termination of the server.

nsd -kill
This is similar to the above, but NO diagnostic data is collected. The server is forcibly terminated. Any databases opened at the time of invocation of the nsd -kill will require fixup when the server is restarted.

Subject: RE: MSClustering

We made a proposal in april 2004 for Domino 6 with MS Cluster based on information available at the time . The customer (govnt bid) accepted ou offer and we are now planning the details of implementation. It is with great displeasure that I just found this thread as well as the official Lotus reply : http://www-1.ibm.com/support/docview.wss?rs=0&q1=1165986&uid=swg21165986&loc=en_US&cs=utf-8&cc=us&lang=en

Does Domino 6.5.3 really works with MS Windows 2003 cluster if you disable NSD ???

Thanks in advance for replies

Subject: RE: MSClustering

What kind of scripts do you use? Can you post an example script? Do I have to use the MS Platform SDK?