Massive problems with https connections to Domino Server over Proxy(squid)

hcl-bot · February 5, 2004, 3:47am

Hi,I have asked this question some time ago but we have never found a solution.

We are using squid 2.5 S4 and also tried v3, OS is Redhat EL ES3,

clients are always IE6 and IE5.5.

Squid is the gateway to a small transfer net to firewall and then to DMZ and internet.

Firewall has changed from Checkpoint FW1 to an iptables firewall, but no change in behaviour.

Domino is R6.51 on RH ES 3.

I can login to Domino server fine but after some views and klicking too fast in our web application IE comes to a standstill, the domino server is blocked, there is no http or https traffic to the domino server.

Nobody can work anymore!

Exactly if I close my IE all works normal, http and https runs fine.

This happens only if I use squid, when I go directly this never happens, all is fine.

Here is my observation:

There are many tcp connections from my client to squid in state

‘connected’ (around 20 to 30)

and there are many connections from squid to domino server in state

‘connected’ (again around 20 to 30)

Output of the domino http task:

05.02.2004 08:45:14 Http Worker Thread ID [44012]: Working session [4014]: Session State [SSL Handshake] :

05.02.2004 08:45:14 Http Worker Thread ID [48013]: Working session [3fed]: Session State [SSL Handshake] :

05.02.2004 08:45:14 Http Worker Thread ID [4c014]: Working session [3fee]: Session State [SSL Handshake] :

05.02.2004 08:45:14 Http Worker Thread ID [50015]: Working session [3fef]: Session State [SSL Handshake] :

05.02.2004 08:45:14 Http Worker Thread ID [54016]: Working session

… cut here

as many http worker threads I configure (around 20 to 30…).

The question is: why goes SSL Handshake wrong and connection is not getting terminated?

And why don’t I see this behaviour without squid?

Here is an excerpt from domino release notes that might go into this direction:

SSL Session Resumption

SSL now performs session resumption. This will greatly improve performance when the Notes HTTP Client or server is

using SSL, and may have a minor (positive) effect on other “Internet” protocols as well.

The default number of resumable sessions that will be cached on the server is 50. To modify the number of sessions

cached, set the SSL_RESUMABLE_SESSIONS notes.ini variable to the desired number. Setting

SSL_RESUMABLE_SESSIONS=1 will disable SSL session resumption on the server.

In Domino 6, the number of resumable sessions will not dynamically climb to match the server SSL load, and there is

currently no means of configuring sessions to time out and expire.

Benign Internet Explorer error message

Error messages that look like this

handshake failure, IP address [9.99.99.999], Keyring [R6keyfile.kyr], [SSL

Error: Network IO error], code [4165]

are produced in many benign circumstances, especially by Microsoft Internet Explorer. The most frequent

circumstance is when IE drops an SSL connection and then resumes it. The error is generated even though there is no

apparent interruption percieved by the client.

Any help is really appreciated!

Rainer

hcl-bot · February 5, 2004, 10:20am

Subject: Massive problems with https connections to Domino Server over Proxy(squid)

Rainer,

search for “code 4165” in this forum, it’s most likely an IE bug.

Good luck,

Uli

hcl-bot · February 6, 2004, 2:46am

Subject: RE: Massive problems with https connections to Domino Server over Proxy(squid)

Hi Uli,Error 4165 is an informational message but should not do any harm (see release notes of R6.51).

It is generated when IE tries to resume an SSL connection but something (domino?) fails.

Now it is a different issue, it seems domino gets stuck in the middle of the SSL handshake and does not drop the stuck connection.

Rainer

hcl-bot · February 6, 2004, 11:03am

Subject: RE: Massive problems with https connections to Domino Server over Proxy(squid)

Rainer,

Did you test your setup with a different browser, e.g. Mozilla or Opera? Same issue?
Do you have self signed certificates generated by the Domino Internet CA or certs signed by a commercial trust center?
Is the root cert installed and shown as trusted in Windows Internet Options?
Do you have the following checkbox ticked on the extended Internet Options tab: “use HTTP 1.1 over proxy connections” and at least for testing purposes in the security section only the checkboxes “use SSL 3.0” and “use TLS 1.0” (all other checkboxes deactivated)
Could you provide the relevant iptables lines (redirect to squid port and HTTPS section)?
Are the latest Service Packs installed on your Win clients? The first Win2k and WinXP releases are pretty buggy regarding SSL support.
Do you have similar security setings in your Domino web site configuration document:

Tab “security” on our web site document:

TCP Authentication

Anonymous: No

Name & password: Yes

Redirect TCP to SSL: No

SSL Authentication

Anonymous: No

Name & password: Yes

Client certificate: No

SSL Options

Key file name: key.kyr

Protocol version: Negotiated

Accept SSL site certificates: No

Accept expired SSL certificates: Yes

Check for CRLs: No

Trust expired CRLs: Yes

Allow CRL search to fail: Yes

SSL Security

SSL ciphers:

RC4 encryption with 128-bit key and MD5 MAC

RC4 encryption with 128-bit key and SHA-1 MAC

Triple DES encryption with 168-bit key and SHA-1 MAC

Enable SSL V2:

hcl-bot · February 7, 2004, 4:17am

Subject: RE: Massive problems with https connections to Domino Server over Proxy(squid)

Uli, thanks for your answer.

Did you test your setup with a different browser, e.g. Mozilla or Opera? Same issue?

Mozilla is considerably slower(!) in loading the pages and I could not see this behaviour.

Do you have self signed certificates generated by the Domino Internet CA or certs signed by a commercial trust center?

Commercial TC

Is the root cert installed and shown as trusted in Windows Internet Options?

Yes.

Do you have the following checkbox ticked on the extended Internet Options tab: “use HTTP 1.1 over proxy connections” and at least for testing purposes in the security section only the checkboxes “use SSL 3.0” and “use TLS 1.0” (all other checkboxes deactivated)

Yes. Tried with HTTP 1.1 and without and only using SSL and TLS.

Could you provide the relevant iptables lines (redirect to squid port and HTTPS section)?

No, it’s not a transparent proxy. → no iptables here.

Are the latest Service Packs installed on your Win clients? The first Win2k and WinXP releases are pretty buggy regarding SSL support.

Latest Windows Update on my XP test client.

Do you have similar security setings in your Domino web site configuration document:

…

The rest looks very similar.

SSL V2 is turned off.

Rainer

hcl-bot · April 20, 2004, 7:40am

Subject: RE: Massive problems with https connections to Domino Server over Proxy(squid)

Try something simple like renaming the key files to default names. IE, keyfile.kyr!

Renaming the keyfile to anything else breaks TLS.

Domino, Unique quirky and non standard but we still love it!

hcl-bot · February 8, 2007, 10:34am

Subject: RE: Massive problems with https connections to Domino Server over Proxy(squid)

I have a similar problem using plain http.

Essentially, when there are too many connections getting used between squid and a browser, domino http seems to get stuck and simply ignore the next request.

Anybody having a similar experience?

hcl-bot · April 30, 2008, 10:16am

Subject: Massive problems with https connections to Domino Server

We still have this problem with 6.5.6 FP3 currently, and before that with several other 6.x versions . Maybe it is an IE / proxy / combination of both -problem, but we cannot drop support for customers that use IE.

The bad thing is, ALL threads are blocked, sometimes several times a day several days a week. They go away after 5 minutes or so , but in this timeframe NO Requests are processed by the server. Very bad.

We tried to track down the problem with “tell http debug session on” etc. but no luck (massive output of htsess*-files in IBM_TECHNICAL_SUPPORT). With debug sessions on, The well-known error

“HTTP Server: SSL handshake failure, IP address [xxx.xxx.xxx.xxx], Keyring [pmskey.kyr], [SSL Error: Network IO error], code [4165]”

is logged when the blocked threads time out (with xxx.xxx.xxx.xxx as the outbound IP adress, that means not 127.0.0.1 or 0.0.0.0).

As soon as we have access to the ESR Tool, we will open a PMR for this.

But maybe someone here has a new idea on that topic?

tell http show thread state

30.04.2008 15:21:06 Http Worker Thread ID [12dc]: Working session [b94]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [13cc]: Working session [b50]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1298]: Working session [b51]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1140]: Working session [b52]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [f84]: Working session [b53]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [cb0]: Working session [b54]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [13e8]: Working session [b55]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [fe0]: Working session [b56]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [4b8]: Working session [b57]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [d20]: Working session [b58]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1190]: Working session [b59]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [db4]: Working session [b5a]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [ad0]: Working session [b5b]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1c8]: Working session [b5c]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [518]: Working session [b5d]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [9c0]: Working session [b5e]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [11a4]: Working session [b5f]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [13dc]: Working session [b60]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [ea0]: Working session [b61]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [4f0]: Working session [b62]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [ec8]: Working session [b63]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [7fc]: Working session [b64]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [13e0]: Working session [b65]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [ec4]: Working session [b66]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [ed4]: Working session [b67]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1248]: Working session [b68]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [b80]: Working session [b69]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [111c]: Working session [b6a]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [113c]: Working session [b6b]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1120]: Working session [b6c]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1558]: Working session [b6d]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1004]: Working session [b77]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [153c]: Working session [b6f]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1144]: Working session [b70]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1678]: Working session [b71]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [c80]: Working session [b72]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1288]: Working session [b73]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1110]: Working session [b74]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [248]: Working session [b75]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1530]: Working session [b76]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [ef0]: Working session [b78]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1730]: Working session [b79]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [11b4]: Working session [b7a]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [cf4]: Working session [b7b]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1758]: Working session [b7c]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1680]: Working session [b7d]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [e58]: Working session [b7e]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [b58]: Working session [b7f]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1154]: Working session [b80]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1d4]: Working session [b81]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [be0]: Working session [b82]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [125c]: Working session [b83]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1060]: Working session [b84]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1088]: Working session [b85]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [d80]: Working session [b86]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1620]: Working session [b87]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [16f4]: Working session [b88]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1284]: Working session [b89]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [15fc]: Working session [b8a]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1148]: Working session [b8b]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [fcc]: Working session [b8c]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [14fc]: Working session [b8d]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [9b4]: Working session [b8e]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [c0]: Working session [b8f]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [e7c]: Working session [b90]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [edc]: Working session [b91]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1dc]: Working session [b92]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [8a8]: Working session [b93]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [e88]: Working session [b95]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [9b8]: Working session [b96]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [15e0]: Working session [b97]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [c58]: Working session [b98]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [b78]: Working session [b99]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [aa0]: Working session [b9a]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [fd0]: Working session [b9b]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [139c]: Working session [b9c]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [1218]: Working session [b9d]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [87c]: Working session [b9e]: Session State [SSL Handshake] :

30.04.2008 15:21:06 Http Worker Thread ID [f3c]: Working session [b9f]: Session State [SSL Handshake] :

30.04.2008 15:21:24 Remote console command issued by Dev-01/Voessing-PMS/DE: tell http show users

tell http show users

30.04.2008 15:21:26 There are 13 current HTTP user sessions

…