Need (easy) help with 653 on Redhat AS 3.0

HCL Domino Domino Forum
1

I am having fits getting 653 to run to some reasonable performance level on Redhat AS 3.0. During loading testing, it will throw a “Panic:Insufficent Semaphores” when the user count gets up to about 2700 which isn’t much for the hardware. (Note, it’s the number of sessions, not transcations, that matter. This box can and has done 55000 tpm)

Anyway, got a ticket open with IBM (for 2 weeks now) and instead of directly addressing the semaphore issue, they are hung up on a quirk of “ps -ef”.

The PPID of the helper tasks (amgr, event, update, etc) does not match the PID of the server. In fact, the PPID of these helper tasks does not reflect any existing process, defunct or otherwise. So my question for whomever is willing to lend a hand is this, when you run ps -ef, does the PPID of the helper tasks match the main server process (/opt/lotus/notes/latest/linux/server [or whatever that path is]) or is it something else. If you could post your results, that’d be great.

I actually have two boxes doing this but they are completely identical (exact same hardware, exact same image) so it’s too much apples to apples to make any kind of real comparison.

Thanks in advance.

Zach

PS. If you have had performance problems with 653 on RHAS 3.0 related to “Panic:Insufficent Semaphores” and you got it fixed, I’d love to know how. I promise not to tell IBM support. Vital Stats: Egenera blade running RHAS 3.0 (patched to latest greatest that I know of) using an EMC DMX SAN. 2x3Ghz Xenon, 6GB RAM with different LUNs for bin, data and translog. No VMWare, no partitions, pretty straightforward install.

2

Subject: Need (easy) help with 653 on Redhat AS 3.0

Hi Zach,

If you are having semaphore problems, your “ps -ef” will NEVER show nothing.

Now any process in a *nix machine haves his own ID. There is a father and a child process, they can be connected on a specifc tree, but but each process haves his own ID. So if every process is having a different ID, your system is running normal.

Now the semaphore problem is more complicated. A semaphore is created every time more than one process needs the same resource.

I don’t know how to do this on a linux machine, I did it on a SUN, editing /etc/system file, changing a few parameters there. I believe people from RedHat will be able to help you.

HTH

Daniel

3

Subject: Check out the tweaks we performed for the RH AS 2.1

It was a few years ago but it’s still relevant.

http://www.notesbench.org/summary.nsf/0/caf3d729944c16f085256c1000139d48/$FILE/redhat_notesbench_080802.pdf

At the time, I couldn’t get to the notesbench site but I would try at least once a day until it is back. There’s good changes that dramatically improve performance (i.e., we got 7000 users with two partitions).

If you can’t get the report, send me an email and I’ll email it to you.

Mike Robinson

http://www.invcs.com

4

Subject: RE: Check out the tweaks we performed for the RH AS 2.1

Mike,I’ve already found this document (if this is the RHAS 2.1 Notesbench Disclosure) and have been using it. We already met or exceeded everything but the kernel.sem. I know, I know, if you have semaphore problems, up them…but, as our Linux admin says, that is for AS 2.1 and we are running 3.0, plus, Oracle doesn’t need this many sems so something must be wrong with Domino. . We’ve since got some feedback from IBM, to whom the Linux admin will listen so, hopefully, we will be upping that shortly.

What I am really curious about now is the fact the the server tasks have as a PPID a number which doesn’t reflect an existing process. Moreover, the server process only shows up once when it should be showing up serveral times.

Thanks for the response and that AS 2.1 document has been helpful.

Thanks

Zach Mabe

5

Subject: In regards to PPID

Hi Zach, sorry I didn’t really answer your original question. Remember on Linux threads are implemented as light weight processes so that could be what you are looking at…

Mike Robinson

http://www.invcs.com

6

Subject: Check out the tweaks we performed for the RH AS 2.1

It was a few years ago but it’s still relevant.

http://www.notesbench.org/summary.nsf/0/caf3d729944c16f085256c1000139d48/$FILE/redhat_notesbench_080802.pdf

At the time, I couldn’t get to the notesbench site but I would try at least once a day until it is back. There’s good changes that dramatically improve performance (i.e., we got 7000 users with two partitions).

If you can’t get the report, send me an email and I’ll email it to you.

Mike Robinson

http://www.invcs.com

7

Subject: Need (easy) help with 653 on Redhat AS 3.0

You can try some of these parameters set in the sysctl.conf in the /etc directory.

kernel.shmmni=8192

kernel.sem=250 32000 32 1024

vm.pagecache = 10 15 15

vm.max-readahead = 8

vm.min-readahead = 2

net.ipv4.tcp_fin_timeout 15

net.ipv4.tcp_max_syn_backlog 16384

net.ipv4.tcp_tw_reuse 1

net.ipv4.ip_local_port_range 1024 65535

I found the readahead cache can be a big performance impact, the default will have the system swapping way too much.

also in the notes.ini you might add

ConstrainedSHM=1

ConstrainedSHMSizeMB=1600

This will limit the memory you can access. We have seen problems as the machine gets past the 2 gig line.

and here is part of the output from a ps -ef .

notes 29723 1 0 Mar27 ? 00:00:00 /bin/sh /opt/lotus/local/notessrv startat

notes 29789 29723 0 Mar27 ? 00:00:03 /opt/lotus/notes/latest/linux/server

notes 29790 29789 0 Mar27 ? 00:00:03 /opt/lotus/notes/latest/linux/server

notes 29791 29790 0 Mar27 ? 00:04:11 /opt/lotus/notes/latest/linux/server

notes 29906 29790 0 Mar27 ? 00:00:08 /opt/lotus/notes/latest/linux/server

notes 29977 29789 0 Mar27 ? 00:04:20 /opt/lotus/notes/latest/linux/event

notes 29993 29790 0 Mar27 ? 00:04:38 /opt/lotus/notes/latest/linux/server

notes 29996 29977 0 Mar27 ? 00:00:01 /opt/lotus/notes/latest/linux/event

notes 29997 29996 0 Mar27 ? 00:02:49 /opt/lotus/notes/latest/linux/event

notes 30002 29996 0 Mar27 ? 00:00:06 /opt/lotus/notes/latest/linux/event

notes 30039 29996 0 Mar27 ? 00:03:20 /opt/lotus/notes/latest/linux/event

notes 30040 29996 0 Mar27 ? 00:01:26 /opt/lotus/notes/latest/linux/event

notes 30042 29996 0 Mar27 ? 00:01:15 /opt/lotus/notes/latest/linux/event

notes 30043 29996 0 Mar27 ? 00:01:03 /opt/lotus/notes/latest/linux/event

notes 30044 29996 0 Mar27 ? 00:00:05 /opt/lotus/notes/latest/linux/event

notes 30045 29996 0 Mar27 ? 00:00:27 /opt/lotus/notes/latest/linux/event

notes 30046 29996 0 Mar27 ? 00:00:51 /opt/lotus/notes/latest/linux/event

notes 30102 29790 0 Mar27 ? 00:04:49 /opt/lotus/notes/latest/linux/server

notes 30103 29790 0 Mar27 ? 00:06:32 /opt/lotus/notes/latest/linux/server

notes 30104 29790 0 Mar27 ? 00:02:23 /opt/lotus/notes/latest/linux/server

notes 30105 29790 0 Mar27 ? 00:00:08 /opt/lotus/notes/latest/linux/server

notes 30109 29790 0 Mar27 ? 00:00:21 /opt/lotus/notes/latest/linux/server

notes 30110 29790 0 Mar27 ? 00:00:03 /opt/lotus/notes/latest/linux/server

notes 30111 29790 0 Mar27 ? 00:00:00 /opt/lotus/notes/latest/linux/server

notes 30112 29790 0 Mar27 ? 00:00:38 /opt/lotus/notes/latest/linux/server

notes 30113 29790 0 Mar27 ? 00:00:29 /opt/lotus/notes/latest/linux/server

notes 30114 29790 0 Mar27 ? 00:00:03 /opt/lotus/notes/latest/linux/server

notes 30115 29790 0 Mar27 ? 00:11:32 /opt/lotus/notes/latest/linux/server

notes 30116 29790 0 Mar27 ? 00:00:06 /opt/lotus/notes/latest/linux/server

notes 30117 30104 27 Mar27 ? 07:35:30 /opt/lotus/notes/latest/linux/replica

notes 30118 30104 0 Mar27 ? 00:02:56 /opt/lotus/notes/latest/linux/router

notes 30119 30104 1 Mar27 ? 00:31:28 /opt/lotus/notes/latest/linux/update

notes 30120 30104 1 Mar27 ? 00:29:57 /opt/lotus/notes/latest/linux/update

notes 30122 30104 2 Mar27 ? 00:40:32 /opt/lotus/notes/latest/linux/update

notes 30125 30104 0 Mar27 ? 00:01:08 /opt/lotus/notes/latest/linux/amgr

notes 30129 30104 0 Mar27 ? 00:04:48 /opt/lotus/notes/latest/linux/adminp

notes 30130 30104 1 Mar27 ? 00:22:26 /opt/lotus/notes/latest/linux/sched

notes 30131 30104 0 Mar27 ? 00:00:04 /opt/lotus/notes/latest/linux/calconn

notes 30141 30104 0 Mar27 ? 00:00:13 /opt/lotus/notes/latest/linux/http

notes 30144 30117 0 Mar27 ? 00:00:01 /opt/lotus/notes/latest/linux/replica

notes 30146 30104 0 Mar27 ? 00:00:37 /opt/lotus/notes/latest/linux/imap

8

Subject: RE: Need (easy) help with 653 on Redhat AS 3.0

Odd. This is what our ps -ef looks like
pre>
ltsdom6 837 811 0 Mar25 ? 00:00:00 -bash -c /opt/lotus/bin/server
ltsdom6 993 837 2 Mar25 ? 01:58:03
/opt/lotus/notes/latest/linux/server
ltsdom6 1013 993 0 Mar25 ? 00:11:43
/opt/lotus/notes/latest/linux/logasio NOTESLOGGER reserved
ltsdom6 1176 993 0 Mar25 ? 00:13:30
/opt/lotus/notes/latest/linux/event
ltsdom6 1487 1480 0 Mar25 ? 00:24:55
/opt/lotus/notes/latest/linux/update
ltsdom6 1489 1480 0 Mar25 ? 00:03:37
/opt/lotus/notes/latest/linux/router
ltsdom6 1490 1480 0 Mar25 ? 00:03:45
/opt/lotus/notes/latest/linux/amgr
ltsdom6 1492 1480 0 Mar25 ? 00:06:13
/opt/lotus/notes/latest/linux/adminp
ltsdom6 1496 1480 0 Mar25 ? 00:03:08
/opt/lotus/notes/latest/linux/calconn
ltsdom6 1498 1480 0 Mar25 ? 00:07:51
/opt/lotus/notes/latest/linux/sched
ltsdom6 1509 1480 0 Mar25 ? 00:17:32
/opt/lotus/notes/latest/linux/http
ltsdom6 1518 1480 0 Mar25 ? 00:04:07
/opt/lotus/notes/latest/linux/imap
ltsdom6 1523 1480 0 Mar25 ? 00:04:19
/opt/lotus/notes/latest/linux/ldap
ltsdom6 1524 1480 0 Mar25 ? 00:03:58
/opt/lotus/notes/latest/linux/pop3
ltsdom6 1526 1480 0 Mar25 ? 00:03:07
/opt/lotus/notes/latest/linux/stats
ltsdom6 1593 1490 0 Mar25 ? 00:03:24
/opt/lotus/notes/latest/linux/amgr -e 1
ltsdom6 10332 1595 1 13:26 ? 00:00:00 /bin/sh
/tmp/notes9C7D74/Y3450473.sh
ltsdom6 10333 10332 0 13:26 ? 00:00:00 /usr/bin/iostat -d -x 60 2

]

The PPID of the server tasks don’t match anything and we have only one
server…hmmm.

Thanks for tuning stuff. I’ll try to get our Linux admin to take a look.

Thanks
Zach,