Domino 6.0.3 Server Crash during Startup

hcl-bot · October 18, 2003, 1:50am

I upgraded server from 6.0.2cf2 to 6.0.3 but server is crashing at the startup itself. I tried even a fresh installation, still the server is crashing.

Any help is highly appreciated.

Here is the info when server startup & also log data from fault recoverylog.

Lotus Domino (r) Server, Release 6.0.3, September 18, 2003

Performing consistency check on log.nsf…

Completed consistency check on log.nsf

10/18/2003 21:52:36 Begin scan of databases to be consistency checked

10/18/2003 21:52:36 End scan of databases: 2 found

10/18/2003 21:52:38 Event Monitor started

10/18/2003 21:52:38 Performing consistency check on events4.nsf…

10/18/2003 21:52:38 An Adminp request has been submitted to update port information in the server document

10/18/2003 21:52:38 Server started on physical node Devlopment

10/18/2003 21:52:38 Completed consistency check on events4.nsf

10/18/2003 21:52:38 Upgrading the design and data of EVENTS4.NSF…

Stack base = 0xbedffc4c, Stack size = 8900 bytes

Fatal Error signal = 0x0000000b PID/TID = 2768/6151

Sat Oct 18 21:52:41 Running cleanup script

NSD is in progress …

Sat Oct 18 21:53:06 Running NSD

NSD is in progress …

notes@Devlopment:~$ less fault_recovery.log

Sat Oct 18 21:15:43 [15] Running external script: /opt/lotus/bin/nsd

Sat Oct 18 21:16:03 [15] Running external script: /opt/lotus/notes/latest/linux/nsd.sh

Sat Oct 18 21:16:27 [1] STARTING FAULT RECOVERY by process: 2766

Sat Oct 18 21:16:27 [2] Terminating process: PID (PPID) = 3051 ( 2766)

Sat Oct 18 21:16:44 [4] Process hasn’t died, sending SIGKILL: PID (PPID) = 3051 ( 2766)

Sat Oct 18 21:16:49 [4] Process hasn’t died, sending SIGKILL: PID (PPID) = 3051 ( 2766)

Sat Oct 18 21:16:54 [4] Process hasn’t died, sending SIGKILL: PID (PPID) = 3051 ( 2766)

Sat Oct 18 21:16:59 [5] ERROR: Some processes won’t die

Sat Oct 18 21:16:59 [7] Removed IPC: SHMEM 1310720

Sat Oct 18 21:16:59 [7] Removed IPC: SEM 196608

Sat Oct 18 21:16:59 [7] Removed IPC: SHMEM 1343489

Sat Oct 18 21:16:59 [7] Removed IPC: SHMEM 1376258

Sat Oct 18 21:16:59 [7] Removed IPC: SHMEM 1409027

Sat Oct 18 21:16:59 [7] Removed IPC: SHMEM 1441796

Sat Oct 18 21:16:59 [7] Removed IPC: SHMEM 1474565

Sat Oct 18 21:16:59 [7] Removed IPC: SHMEM 1507334

Sat Oct 18 21:16:59 [7] Removed IPC: SHMEM 1540103

Sat Oct 18 21:16:59 [7] Removed IPC: SHMEM 1572872

Sat Oct 18 21:16:59 [7] Removed IPC: SHMEM 1605641

Sat Oct 18 21:16:59 [7] Removed IPC: SEM 229377

Sat Oct 18 21:16:59 [7] Removed IPC: SEM 262146

Sat Oct 18 21:16:59 [7] Removed IPC: SHMEM 1638410

Sat Oct 18 21:16:59 [7] Removed IPC: SEM 294915

Sat Oct 18 21:16:59 [7] Removed IPC: SHMEM 1671179

Sat Oct 18 21:16:59 [12] ERROR: Bad message in queue: PID (PPID) = 3051 ( 2766)

Sat Oct 18 21:16:59 [20] TERMINATION COMPLETE

Sat Oct 18 21:19:21 [15] Running external script: /opt/lotus/bin/nsd

Sat Oct 18 21:19:42 [15] Running external script: /opt/lotus/notes/latest/linux/nsd.sh

Sat Oct 18 21:21:46 [1] STARTING FAULT RECOVERY by process: 3832

Sat Oct 18 21:21:46 [2] Terminating process: PID (PPID) = 4122 ( 3832)

Sat Oct 18 21:22:03 [4] Process hasn’t died, sending SIGKILL: PID (PPID) = 4122 ( 3832)

Sat Oct 18 21:22:08 [4] Process hasn’t died, sending SIGKILL: PID (PPID) = 4122 ( 3832)

Sat Oct 18 21:22:13 [4] Process hasn’t died, sending SIGKILL: PID (PPID) = 4122 ( 3832)

Sat Oct 18 21:22:18 [5] ERROR: Some processes won’t die

Sat Oct 18 21:22:18 [7] Removed IPC: SHMEM 1703936

Sat Oct 18 21:22:18 [7] Removed IPC: SEM 327680

Sat Oct 18 21:22:18 [7] Removed IPC: SHMEM 1736705

Sat Oct 18 21:22:18 [7] Removed IPC: SHMEM 1769474

Sat Oct 18 21:22:18 [7] Removed IPC: SHMEM 1802243

Sat Oct 18 21:22:18 [7] Removed IPC: SHMEM 1966088

Sat Oct 18 21:22:18 [7] Removed IPC: SHMEM 1998857

Sat Oct 18 21:22:18 [7] Removed IPC: SEM 360449

Sat Oct 18 21:22:18 [7] Removed IPC: SEM 393218

Sat Oct 18 21:22:18 [7] Removed IPC: SHMEM 2031626

Sat Oct 18 21:22:18 [7] Removed IPC: SEM 425987

Sat Oct 18 21:22:18 [12] ERROR: Bad message in queue: PID (PPID) = 4122 ( 3832)

Sat Oct 18 21:22:18 [20] TERMINATION COMPLETE

Sat Oct 18 21:28:38 [15] Running external script: /opt/lotus/bin/nsd

Sat Oct 18 21:29:01 [15] Running external script: /opt/lotus/notes/latest/linux/nsd.sh

Sat Oct 18 21:34:52 [15] Running external script: /opt/lotus/bin/nsd

Sat Oct 18 21:35:17 [15] Running external script: /opt/lotus/notes/latest/linux/nsd.sh

Sat Oct 18 21:41:41 [1] STARTING FAULT RECOVERY by process: 1704

Sat Oct 18 21:41:41 [2] Terminating process: PID (PPID) = 1849 ( 1704)

Sat Oct 18 21:41:58 [4] Process hasn’t died, sending SIGKILL: PID (PPID) = 1849 ( 1704)

Sat Oct 18 21:42:03 [4] Process hasn’t died, sending SIGKILL: PID (PPID) = 1849 ( 1704)

Sat Oct 18 21:42:08 [4] Process hasn’t died, sending SIGKILL: PID (PPID) = 1849 ( 1704)

Sat Oct 18 21:42:13 [5] ERROR: Some processes won’t die

Sat Oct 18 21:42:13 [7] Removed IPC: SHMEM 32769

Sat Oct 18 21:42:13 [7] Removed IPC: SHMEM 65538

Sat Oct 18 21:42:13 [7] Removed IPC: SHMEM 98307

Sat Oct 18 21:42:13 [7] Removed IPC: SHMEM 131076

Sat Oct 18 21:42:13 [7] Removed IPC: SHMEM 163845

Sat Oct 18 21:42:13 [7] Removed IPC: SHMEM 196614

Sat Oct 18 21:42:13 [7] Removed IPC: SHMEM 229383

Sat Oct 18 21:42:13 [7] Removed IPC: SHMEM 262152

Sat Oct 18 21:42:13 [7] Removed IPC: SHMEM 294921

Sat Oct 18 21:42:13 [7] Removed IPC: SEM 32769

Sat Oct 18 21:42:13 [7] Removed IPC: SEM 65538

Sat Oct 18 21:42:13 [7] Removed IPC: SEM 98307

Sat Oct 18 21:42:13 [7] Removed IPC: SHMEM 327690

Sat Oct 18 21:42:13 [12] ERROR: Bad message in queue: PID (PPID) = 1849 ( 1704)

Sat Oct 18 21:42:13 [20] TERMINATION COMPLETE

Sat Oct 18 21:52:41 [15] Running external script: /opt/lotus/bin/nsd

Sat Oct 18 21:53:06 [15] Running external script: /opt/lotus/notes/latest/linux/nsd.sh

Sat Oct 18 21:56:24 [1] STARTING FAULT RECOVERY by process: 2768

Sat Oct 18 21:56:24 [2] Terminating process: PID (PPID) = 2915 ( 2768)

Sat Oct 18 21:56:41 [4] Process hasn’t died, sending SIGKILL: PID (PPID) = 2915 ( 2768)

Sat Oct 18 21:56:46 [4] Process hasn’t died, sending SIGKILL: PID (PPID) = 2915 ( 2768)

Sat Oct 18 21:56:51 [4] Process hasn’t died, sending SIGKILL: PID (PPID) = 2915 ( 2768)

Sat Oct 18 21:56:56 [5] ERROR: Some processes won’t die

Sat Oct 18 21:56:56 [7] Removed IPC: SHMEM 360449

Sat Oct 18 21:56:56 [7] Removed IPC: SHMEM 393218

Sat Oct 18 21:56:56 [7] Removed IPC: SHMEM 491525

Sat Oct 18 21:56:56 [7] Removed IPC: SHMEM 524294

Sat Oct 18 21:56:56 [7] Removed IPC: SHMEM 557063

Sat Oct 18 21:56:56 [7] Removed IPC: SHMEM 589832

Sat Oct 18 21:56:56 [7] Removed IPC: SHMEM 622601

Sat Oct 18 21:56:56 [7] Removed IPC: SEM 131073

Sat Oct 18 21:56:56 [7] Removed IPC: SEM 163842

Sat Oct 18 21:56:56 [7] Removed IPC: SEM 196611

Sat Oct 18 21:56:56 [7] Removed IPC: SHMEM 655370

Sat Oct 18 21:56:56 [12] ERROR: Bad message in queue: PID (PPID) = 2915 ( 2768)

Sat Oct 18 21:56:56 [20] TERMINATION COMPLETE

hcl-bot · October 18, 2003, 9:08am

Subject: Domino 6.0.3 Server Crash during Startup

Im running RH 7.2, kernel version 2.4.7-10, and Im having exactly same problem.

I was running domino version 6.0.1 and when upgraded to version 6.0.3 the server crashes with Fatal Error signal = 0x0000000b.

Anyone?

hcl-bot · October 19, 2003, 12:49am

Subject: RE: Domino 6.0.3 Server Crash during Startup

I got this to work, but I had to install 6.5 then downgrade to 6.0.3

Then I found out that 6.0.3 and 6.5 didn’t work correctly with ODBC datasources so I reverted back to 6.0.2

I’ve heard a lot of bad things about 6.0.3, so it might be a good idea to wait on this one. I’ll bet you’ll see a 6.0.3 CF1 coming up soon.

hcl-bot · October 19, 2003, 3:13pm

Subject: RE: Domino 6.0.3 Server Crash during Startup

Can any of you both fwd me the NSD file created duirng the crash and last few lines of the console output ?

hcl-bot · October 20, 2003, 6:53am

Subject: RE: Domino 6.0.3 Server Crash during Startup

Sure, I can post it shortly.

I took a look at the NSD file earlier and noticed that server task is the one that is causing crash.

Trying to debug, I edit notes.ini and remove any additional tasks under servertasks= setting.

I got a warning during install stating that my kernel version is not the “right one”, but I got same message when updating it to version 6.0.1CF1 and it was running perfectly for months.

I cannot upgrade my kernel version properly, due to some drivers dependency on the kernel version I am using.

Do anyone think that might be some workaround for this?

Regards.

hcl-bot · October 20, 2003, 3:31am

Subject: Domino 6.0.3 Server Crash during Startup

Hello,

make a fixup over all your databases without domino running. Do the following in your bash:

change the directory to your /dominodata

type: /opt/lotus/bin/fixup

After this is done remove all servertasks in your notes.ini in the line servertasks and start domino without this tasks. Start any task separat.

This should be help to separat which tasks is chrashing.

Good luck.

Regards Rico

hcl-bot · October 21, 2003, 12:54am

Subject: RE: Domino 6.0.3 Server Crash during Startup

Hi ,

I have removed server tasks line from notes.ini still crashing. I could see that…

upgrading events4.nsf crash again…

Lotus Domino (r) Server, Release 6.0.3, September 18, 2003

Performing consistency check on log.nsf…

Completed consistency check on log.nsf

10/18/2003 21:52:36 Begin scan of databases to be consistency checked

10/18/2003 21:52:36 End scan of databases: 2 found

10/18/2003 21:52:38 Event Monitor started

10/18/2003 21:52:38 Performing consistency check on events4.nsf…

10/18/2003 21:52:38 An Adminp request has been submitted to update port information in the server document

10/18/2003 21:52:38 Server started on physical node Devlopment

10/18/2003 21:52:38 Completed consistency check on events4.nsf

10/18/2003 21:52:38 Upgrading the design and data of EVENTS4.NSF…

Stack base = 0xbedffc4c, Stack size = 8900 bytes

Fatal Error signal = 0x0000000b PID/TID = 2768/6151

Sat Oct 18 21:52:41 Running cleanup script

NSD is in progress …

Sat Oct 18 21:53:06 Running NSD

NSD is in progress …

–

hcl-bot · October 30, 2003, 4:39pm

Subject: RE: Domino 6.0.3 Server Crash during Startup

How about if you rename events4.nsf to events4.old , or somesuch, and start the server again?

hcl-bot · October 30, 2003, 11:43pm

Subject: RE: Domino 6.0.3 Server Crash during Startup

Still no luck , I tried that,. system creating new events4.nsf and database server thread started…immediately crash!!!

hcl-bot · October 31, 2003, 11:02am

Subject: RE: Domino 6.0.3 Server Crash during Startup

Not sure if you already did…

Can you post what Linux version and kernel release are you using at this box?

When I (tried) to get it running, install complained about my kernel version and my server crashed right after starting up.

I had not tried yet with a newer kernel release, but I will do it soon and post my experience.

Regards.

hcl-bot · November 2, 2003, 10:45am

Subject: RE: Domino 6.0.3 Server Crash during Startup

I’ve just installed Domino 6.0.3 in a Linux RH 8 - Kernel 2.4.20-7 and it is running smoothly.

It is a fresh install, but I can say that it is running with no issues…

hcl-bot · November 3, 2003, 5:51pm

Subject: RE: Domino 6.0.3 Server Crash during Startup

You get this error if you use a wrong version of glibc. I tried this on my box here.This is the reproducable error and it has nothing to do with events.nsf

Lotus Domino (r) Server, Release 6.5, September 18, 2003

Performing consistency check on log.nsf…

Completed consistency check on log.nsf

03.11.2003 23:50:57 Begin scan of databases to be consistency checked

03.11.2003 23:50:57 End scan of databases: 6 found

03.11.2003 23:50:58 Event Monitor started

Stack base = 0xbedffc4c, Stack size = 8900 bytes

Fatal Error signal = 0x0000000b PID/TID = 12972/6151

Mon Nov 3 23:50:59 Fault recovery is in progress

Using the correct glibc the error disappears. Symlinking old libraries to new ones does not always help, it depends on your installed versions.

Cheers

Heini

hcl-bot · November 4, 2003, 7:13am

Subject: RE: Domino 6.0.3 Server Crash during Startup

Heini,

can you post what version of glibc you were using and which one you installed to solve this error, please…

Regards.

hcl-bot · November 7, 2003, 6:00am

Subject: RE: Domino 6.0.3 Server Crash during Startup

Well,

I am mostly running Debian so I am of little help here but crash problems are mostly related to glibc errors.

I installed this one: libstdc++2.9-glibc2.1

This is not the best solution as it makes it basically work but crashes from time to time. I think I will have to recompile it.

The default install is libstdc++2.10-glibc2.2 1:2.95.4-11woody1 whích I was not able to get it working use various symlinking.

Cheers

Heini

hcl-bot · November 10, 2003, 7:27am

Subject: RE: Domino 6.0.3 Server Crash during Startup

I upgraded my Linux box to RH7.3 and prefer to run Domino 6.5 instead of R6.0.3

So far, so good… running perfectly for 3 days!

I am sure that issues I was getting are related with glibc provided by RH7.2, because I also tried a kernel upgrade with no luck.

hcl-bot · November 10, 2003, 7:45pm

Subject: RE: Domino 6.0.3 Server Crash during Startup

Can you give us the version of the glibc that is working for you.

Also, - which kernel are you running and what is the version number of the thread library?

I prefer running Domino on Debian Woody too, but have still severe difficulties to get 6.5 to run on a Dell SMP system with kernel 2.4.20.

I’stuck with Domino 6.0.2CF1 right now, but would love to upgrade to 6.5 ASAP.

The more details you could provide regarding the library versions installed on your RH7.3 box, the better for all of us here trying to get “our” Domino running as stable as you have your machine.

Thanks for helping us out

Uwe

hcl-bot · November 11, 2003, 7:19am

Subject: RE: Domino 6.0.3 Server Crash during Startup

Here it goes:

RH7.3 - Uniprocessor

Kernel 2.4.18-3 (original from CD)

glibc version:

ldconfig -v | grep glib

libglib.so.1 → libglib.so.1.0.6

libglib-2.0.so.0 → libglib-2.0.so.0.0.1

libglib-1.2.so.0 → libglib-1.2.so.0.0.10

(before upgrading my RH7.2 to RH7.3 I had only libglib-1.2.so.0)

thread lib version:

ldconfig -v | grep thread

libthread_db.so.1 → libthread_db-1.0.so

libpthread.so.0 → libpthread-0.9.so

libgthread-2.0.so.0 → libgthread-2.0.so.0.0.1

libgthread-1.2.so.0 → libgthread-1.2.so.0.0.10

libpthread.so.0 → libpthread-0.9.so

I am using a out-of-box RH7.3 install, so it should work of for you! (I hope).

I´d like to report that I upgraded a second Domino server to 6.5 under RH7.3 and everything is working ok.

Let me know, how it goes for you.

Regards.

hcl-bot · November 16, 2003, 11:32pm

Subject: RE: Domino 6.0.3 Server Crash during Startup

Hi,

I am also running Domino in Debian.

I tried enough times to upgrade or new install of 6.0.3 or 6.5 — Keeps crashing.

I know something wrong with OS, but I am not able to find.

I tried upgrading Kernel but still no help.

My glibc version is 2.2.5 .

Should I upgrade my glibc ???

–

Venkat.

hcl-bot · April 16, 2004, 8:53am

Subject: RE: Domino 6.0.3 Server Crash during Startup

Hi,

I used to have the same things when I try to upgrade a Linux 7.2 or 7.3 with Lotus Domino R6.0.3 or R6.5 or R6.5.1

I made a call to Lotus Support and after a month, I receive that Workaround that working fine:

Workaround:

install the compat-glibc-6.2-2.1.3.2 and compat-libstdc+±6.2-2.9.0.16.

Additional Technical Info:

After install the avove two RPMs, symbolic link state as follows in my test

environmnet…

$ ls -la libstdc++*

libstdc+±2-libc6.1-1-2.9.0.so

libstdc+±3-libc6.2-2-2.10.0.so

libstdc+±libc6.1-1.so.2 → libstdc+±2-libc6.1-1-2.9.0.so

libstdc+±libc6.2-2.so.3 → libstdc+±3-libc6.2-2-2.10.0.so

libstdc++.so.2.7.2 → libstdc++.so.2.7.2.8

libstdc++.so.2.7.2.8

libstdc++.so.2.8 → libstdc++.so.2.8.0

libstdc++.so.2.8.0

libstdc++.so.2.9 → libstdc++.so.2.9.dummy

libstdc++.so.2.9.dummy

Hope it help you.

Daniel.