See the following Technote for the latest information:
Title: How is IBM Domino impacted by the POODLE attack?
Doc #: 1687167
URL: http://www.ibm.com/support/docview.wss?uid=swg21687167
Please refer to the WIKI article cited in the Technote for more information:
IBM Domino Interim Fixes to support TLS 1.0 which can be used to prevent the POODLE attack
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/IBM_Domino_TLS_1.0
Subject: Re: There were no changes to self-signed certs in this IF
Dave,
We’ve had SSLv2 not enabled in Domino for many years (i.e. on Server Doc, Internet Ports tab “Enable SSL V2” has been unchecked for as long as I recall it being available). Does the change in the current IF remove functionality beyond this setting with regards to SSL v2?
Also, you refer to client browsers in some of your responses, but the problem that drove some of us to post SSL log snippets here is in regards to inbound SMTP only, not HTTP(S).
If this is what you are seeing, then I would recommend upgrading or reconfiguring the clients to stop using SSLv2. If you’re concerned about SSLv3, then you should be horrified by SSLv2.
If we consider the “client” to be inbound SMTP server connections, I have zero ability to get Twitter (as a real example) to change how it sends mail. I also have very limited leverage with our own clients who could no longer send us mail after we updated to 9.0.1 FP2 IF1. Especially when I’m not 100% sure what is going on. These clients (who are far larger than my firm) have said “we’re not having any issues delivering mail to anybody but you. It is your problem”.
Several of the clients who could no longer send us mail happen to use the same third-party service Messagelabs/messagescreen.com, so that these clients’ IT folks have directed me to their vendor – which has not led to anything productive so far. Again, the implication is that we’re misconfigured because they are not having issues with anybody else.
In order to keep my sales staff and our clients happy – in lieu of any conclusive evidence that it isn’t our problem – my only option at the moment is to set incoming SSL to “disabled”. We’ve had clients in the past who refused to send mail that way, but at the moment it is keeping more critical mail flowing than setting SSL inbound to Enabled and failing to accept connections from many sources for a yet to be determined reason.
Subject: What do you see for one of these incoming connections with DEBUG_SSL_ALL=1 set in the notes.ini? <>
Subject: Interim Fix also breaks directory assistance connection over LDAPS
We installed this hot fix to today. TLS worked fine when connecting to our Domino server. However, our Domino server uses directory assistance to connect to an external LDAP server and this connection no longer worked. IBM has said this issue will be fixed in 9.0.1 Fix Pack 3 which is due out in the first quarter of 2015.
Subject: Did you receive an SPR?
I’m not aware of any issues in this space, beyond the fact that all support for SSLv2 was dropped.
Unable to connect to patched Domino servers using SSLv2 backwards compatibility mode
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/SSLv2 http://www-10.lotus.com/ldd/dominowiki.nsf/dx/SSLv2
This issue could be present for outbound connections as well if your LDAP server only supports SSLv2, as the patched Domino server will no longer use SSLv2 regardless of how it is configured.
Subject: Thaks for your support but…
… it does not work as expected.
Still the same error.
Subject: Please set DEBUG_SSL_HANDSHAKE=2 and DEBUG_SSL_CIPHERS=2 in your notes.ini and post the output of a failed connection here <>
Subject: Thanks for the heads-up, and…
I’m glad this took less than two weeks (10/23 - 11/3), instead of “several weeks” as originally announced. Good job!
Subject: SMTP inbound SMTP problems after Domino 9.0.1 FP2 IF1
Hi Andres
We (recipient) also have a trusted certificate! The problems seems to be the self signed/untrusted certificate of the SENDER.
And btw…IBM remarked correctly it’s a IF not a hotfix…my bad.
Subject: fix released
The latest interim fixes released on Friday fix the POODLE/TLS problem now.
Subject: Update: Inbound SMTP TLS/SSL handshake problem
I’ve been working with IBM regarding this issue via a PMR for the past two weeks. There is no conclusive resolution as of this moment.
I’m not going to give all the details, but I’ve gone back and forth many times, sent multiple log files after trying various suggested notes.ini/configuration changes, etc, etc, etc. These have been analyzed, presumably, but no fix.
There was speculation that our certificate – which was SHA-1, because until just this recent release SHA-2 certs were not supported – might have been the issue. So, I updated the certificate to SHA-2, and still no joy with certain inbound SSL handshakes failing.
NOTE: for those of you who have GoDaddy for your SSL, switching to a SHA-2 cert was relatively painless and cost free. I first updated our cert from SHA-1 to SHA-2, which is an option in the GoDaddy SSL management screen. Then, realizing that this “new” cert would be based on the original CSR and old private keys, which were SHA-1, I used the option in GoDaddy to rekey the certificate as if the keys were lost. I generated a new CSR using the arcane and obtuse command-line method outlined by IBM support (here: http://www-10.lotus.com/ldd/dominowiki.nsf/dx/3rd_Party_SHA-2_with_OpenSSL_and_kyrtool?open) http://www-10.lotus.com/ldd/dominowiki.nsf/dx/3rd_Party_SHA-2_with_OpenSSL_and_kyrtool?open and got a full SHA-2 certificate with new keys from GoDaddy. Again, no additional cost; same expiration date for cert as the old one. The cert works fine and tests fine using the Qualys SSL HTTP test AND the CheckTLS test for SMTP (CheckTLS: The Email Encryption Testing Authority) http://www.checktls.com/index.html. BTW, I have yet to find a way to get the SHA-2 cert into the “IBM HTTP Server” transparent proxy I have configured to sit in front of Domino, only into Domino itself. This whole thing is a convoluted PITA.
As mentioned, this did not fix the SMTP handshake issue with certain inbound senders, which still error with:
Mapping SSL error -6996 to 4166 [SSLProtocolErr]
I think the issue might be related to this from Symantec: “Symantec Messaging Gateway may fail to connect to mail servers utilizing strict TLS version 1 transport encryption”
Please select your identity provider. - Support Portal http://www.symantec.com/business/support/index?page=content&id=TECH215003
Regardless, IBM needs to fix this so that Domino can reliably accept mail, encrypted, from as many sources as possible. If gmail, Yahoo mail, and hotmail accept connections from the sources in question, then so must our company. If Domino can’t do it, then it is Domino that needs fixing, regardless of any “strict” reading of RFCs or whatever. I can’t not accept mail from the sources in question (some of whom are clients), and I have no power to get these organizations to change what they are doing. Their response is “we’re not having any problems sending mail to anybody except to you; the problem is on your end.” And I agree with them.
As for “Nad Surf”'s http://www-10.lotus.com/ldd/ndseforum.nsf/xpAuthor.xsp?Author=Nad%20Surf question regarding why connections don’t fall back to unencrypted after the encryption handshake fails, it is my understanding that if the sending server is configured to send unencrypted, it will typically only do so if the receiving server indicates it doesn’t support SSL/TLS at all (i.e. it doesn’t offer STARTTLS during the connection negotiation). If the receiving server indicates it DOES support encryption and gives a STARTTLS, and if the two servers cannot agree on a way to encrypt the connection, then the failure does not trigger a plain text send as an alternative. The connection simply fails, and the sender probably tries it over and over until it times out and returns the mail.
It would be nice if encryption handshake failures did fall back to plain (if configured on the server), because I’ve yet to find an inbound connection that fails the encryption handshake yet is unwilling to forgo encryption completely and deliver mail in the clear if I turn off inbound encryption on Domino. No, I don’t have an RFC reference. This is my experience.
The root problem I’m assuming is that the “fix” given to us by IBM strictly supports only TLS 1.0, not 1.1 or 1.2. This is not acceptable. TLS 1.1 was defined by RFC in 2006, going on nine years ago. TLS 1.2 was defined nearly seven years ago, in 2008. (Transport Layer Security - Wikipedia). https://en.wikipedia.org/wiki/Transport_Layer_Security#TLS_1.1 These are NOT new standards.
Our choices are:
A) don’t patch/update Domino and remain vulnerable to POODLE, etc, You’ll have increasing issues from visitors of all kinds due to supporting SSL 3. This isn’t a real option.
B) update to 9.0.1 FP2 IF1 and incompletely support the current TLS standards – which have been available for going on a decade
C) update Domino but disable SMTP inbound SSL/TLS encryption so that you don’t miss critical mail from certain senders (twitter is one off them, BTW)
My “solution” to reliably receiving inbound mail – which is all users and management is really interested in – is option “C”. Obviously, this isn’t really a “solution”.
SMTP inbound TLS on Domino is incomplete/broken as currently offered. It only satisfies a lawyer’s interpretation of “we have given the clients a solution to the problem”, and cannot be used for inbound SMTP by organizations in the real world without the risk of rejecting significant legitimate mail.
Subject: Option D
FWIW
We decided to go for yet another option. I.e., we put Domino behind a reverse proxy (nginx), which gives us state-of-the-art SSL/TLS support. Of course that approach has its caveats, too. Some have been discussed in this forum and elsewhere. Another big downside we stumbled across was the fact that SPAM control (previously Domino-based) was no longer functioning as desired. We had to set up Linux-based SPAM control from scratch – a steep learning curve if you’ve never done that before. Bottom line: This whole mess (my apologies for the harsh word) cost us a LOT of time and resources.
Subject: See this - by Darren Duke
Darren Duke Blog Zone http://blog.darrenduke.net/Darren/DDBZ.nsf/dx/poodle-tls-the-poodle-strikes-back.htm
Subject: notes.ini parameter for SPR LMES9QRUZY
I got right now following notes.ini parameter concerning the SSLv2 ClientHello problem and SPR LMES9QRUZY from IBM support.
Please use the notes.ini parameter: SSL_ENABLE_INSECURE_SSLV2_HELLO=1
Have to wait for a service time slot to try it, will provide some feedback later on …
Subject: FP3IF1?
Did you install IF1 for FP3? It appears this is where SPR LMES9QRUZY was addressed → http://www-01.ibm.com/support/docview.wss?uid=swg21657963 http://www-01.ibm.com/support/docview.wss?uid=swg21657963
edit: This was supposed to be a reply to Mark Borg - I don’t know why this forum doesn’t sort entries properly.
Subject: TLS 1.2
I’m having this problem with a client that is using TLS 1.2. Our Domino is set to use TLS 1.0 (v9.0.1 FP3IF1), of course. This is the extract of the log
[0B4C:000B-0F50] 02/23/2015 07:09:48.30 PM SSLCheckCertChain> Valid certificate chain received
[0B4C:000B-0F50] 02/23/2015 07:09:48.30 PM int_MapSSLError> Mapping SSL error 0 to 0 [SSLNoErr]
[0B4C:000B-0F50] 02/23/2015 07:09:48.30 PM SSL_Handshake> Enter
[0B4C:000B-0F50] 02/23/2015 07:09:48.30 PM SSL_Handshake> Current Cipher 0x0000 (Unknown Cipher)
[0B4C:000B-0F50] 02/23/2015 07:09:48.44 PM SSLReadRecord> Rejecting connection - record contentType not in range for SSLv3 or TLS
[0B4C:000B-0F50] 02/23/2015 07:09:48.44 PM SSLReadRecord> First 4 bytes of SSL/TLS record: 0x80 0xB3 0x01 0x03
[0B4C:000B-0F50] 02/23/2015 07:09:48.44 PM SSLReadRecord> This is probably an SSLv2 ClientHello record which is not supported by default to improve “out of the box” security
[0B4C:000B-0F50] 02/23/2015 07:09:48.44 PM SSLReadRecord> See the SSLv2 page on the Notes/Domino wiki for more information.
I’m going to try that notes.ini parameter this afternoon, but I was wondering if it won’t represent any future security issue, since the initial idea was to not support SSLv2 and v3 anymore.
EDIT 02/25/2015: It worked using SSL_ENABLE_INSECURE_SSLV2_HELLO=1
I still don’t know if this parameter may represent any kind of vulnerability. If anyone can shed some light on this matter, I’ll be most grateful
Subject: SMTP inbound SMTP problems after Domino 9.0.1 FP2 IF1
I too am seeing inbound connection problems from some sending servers after updating to 9.0.1 FP2 IF1. Senders who could send successfully before update immediately could not send after the update.
Enabling DEBUG_SSL_HANDSHAKE=2 and DEBUG_SSL_CIPHERS=2 resulted in log items such as Ronald Hoppe noted.:
[1990:0008-1A68] 11/10/2014 03:38:15 PM SMTP Server: Remote host spring-chicken-ba.twitter.com (199.16.156.166) found in whitelist at PrivateWhitelist
[1990:0008-1A68] 11/10/2014 03:38:15 PM SMTP Server: spring-chicken-ba.twitter.com (199.16.156.166) connected
[1990:0008-1A68] 11/10/2014 03:38:15.85 PM int_MapSSLError> Mapping SSL error 0 to 0 [SSLNoErr]
[1990:0008-1A68] 11/10/2014 03:38:15.85 PM SSL_Handshake> Enter
[1990:0008-1A68] 11/10/2014 03:38:15.85 PM SSL_Handshake> Current Cipher 0x0000 (Unknown Cipher)
[1990:0008-1A68] 11/10/2014 03:38:15.92 PM SSL_Handshake> After handshake state= 3 Status= -6996
[1990:0008-1A68] 11/10/2014 03:38:15.92 PM SSL_Handshake> Exit Status = -6996
[1990:0008-1A68] 11/10/2014 03:38:15.92 PM int_MapSSLError> Mapping SSL error -6996 to 4166 [SSLProtocolErr]
[1990:0008-1A68] 11/10/2014 03:38:16 PM SMTP Server: spring-chicken-ba.twitter.com (199.16.156.166) disconnected. 0 message[s] received
This sequence occurs over and over as the sending server tries to repeatedly send email until it ultimately exceeds the fail time limit on their end.
If I disable the requirement for inbound SSL connections by changing the Server Configuration Setting “SSL negotiated over TCP/IP port” to “Disabled”, then all inbound mail arrives successfully (albeit, unencrypted). I’ve had to leave inbound SSL for SMTP disabled for now to keep mail flowing and customers and my employees from screaming.
BTW, the Domino server passes both inbound and outbound TLS SMTP tests from CheckTLS.com here: CheckTLS: The Email Encryption Testing Authority http://www.checktls.com/index.html as well as the HTTP server tests from Qualys (different protocol, but just to be thorough).
My cert is from GoDaddy, is still SHA-1, and is functioning and unchanged for the past 18 months. I’m going to update to SHA-2, but lets address one thing at a time – unless that is known to be the root problem.
I’ve also opened a Service Request from IBM, but have not yet received more than the initial auto-responder.
Subject: The IF removed all support for SSLv2
The IF removed all support for SSLv2, as described in the Notes/Domino wiki:
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/SSLv2 http://www-10.lotus.com/ldd/dominowiki.nsf/dx/SSLv2
If setting DEBUG_SSL_ALL=1 on your Domino server results in a line like the following…
[30406:00011-233826048] 11/05/2014 06:36:49.27 PM SSL_RCV> 00000000: 80 65 01 03 01 ‘.e…’
[30406:00011-233826048] 11/05/2014 06:36:49.27 PM S_Read> Exit, read 5 bytes
[30406:00011-233826048] 11/05/2014 06:36:49.27 PM SSL_Handshake> After handshake state= 3 Status= -6996
[30406:00011-233826048] 11/05/2014 06:36:49.27 PM SSL_Handshake> Exit Status = -6996
[30406:00011-233826048] 11/05/2014 06:36:49.27 PM int_MapSSLError> Mapping SSL error -6996 to 4166 [SSLProtocolErr]
… then the incoming connection is attempting to use SSLv2 compatibility mode and being rejected.
A “good” incoming SSLv3 or TLS connection will look like this instead:
[10846:00012-930277120] 11/11/2014 12:34:26.18 PM SSL_RCV> 00000000: 16 03 01 00 53 ‘…S’
[10846:00012-930277120] 11/11/2014 12:34:26.18 PM S_Read> Exit, read 5 bytes
[10846:00012-930277120] 11/11/2014 12:34:26.18 PM S_Read> Enter len = 83
[10846:00012-930277120] 11/11/2014 12:34:26.18 PM S_Read> Switching Endpoint to sync
[10846:00012-930277120] 11/11/2014 12:34:26.18 PM S_Read> Posting a nti_rcv for 83 bytes
If this is what you are seeing, then I would recommend upgrading or reconfiguring the clients to stop using SSLv2. If you’re concerned about SSLv3, then you should be horrified by SSLv2.
Subject: Re: SMTP inbound SMTP problems after Domino 9.0.1 FP2 IF1
Christian, thanks for posting your experience. Regarding IBM’s response to you:
It seems that the issue connecting via port 465 secured SSL only happens to users with self signed certificates. Tests wherein the client application has trusted certificate were successfully connected via secured SSL port-- sending and receiving email, proving that the hotfix is working fine.
Problems occur when the application (client) does not have all certs imported and trusted in the client certificate store.
So I would recommend checking which clients are affected by the issue and ensure that all the certs are imported and trusted.
Okay, maybe. But if you look at the log snippet I posted you’ll see that the inbound sender is Twitter. Obviously, it’s possible that Twitter uses self signed certificates, but something makes me think the $27-billion company probably has real certs with a certificate authority on their SMTP servers. Maybe not. Regardless, Domino should accept self-signed certs to encrypt the stream anyway, since that is perfectly allowable. IBM’s own instructions for setting up a certificate for Domino servers even includes a section on using a self-signed cert.
With regards to your comment #2 about falling back to non-encrypted: there is a difference between being the sender and being the receiver. The setting “SSL negotiated over TCP/IP port = enabled” tells Domino to try to negotiate inbound SMTP TLS/SSL, or will allow Domino to accept non-encrypted SMTP connections (setting it to “Required” disallows inbound non-encrypted connections, as I’m sure you know). However, it is up to the sending SMTP server to initiate the retry using a non-encrypted connection if the SSL one fails. It appears to me that some of the sending systems are not configured to do this, so they beat on the Domino server over and over, failing each time to negotiate an encrypted connection and not ever trying to send it unencrypted after failing. And yet, if we set “SSL negotiated over TCP/IP” to “disabled” they are perfectly happy sending mail unencrypted and never even trying to negotiate a TLs connection.
Note: Domino has a NOTES.INI setting “RouterFallbackNonTLS=1” which applies to outbound mail connections and causes Domino to try sending non-encrypted if it is unable to send mail over encrypted connection. If this is set to “0” then Domino will be one of those systems that won’t retry a non-encrypted connection should the TLS connection fail or not be supported. IBM description for this setting here: http://www-10.lotus.com/ldd/dominowiki.nsf/dx/routerfallbacknontls http://www-10.lotus.com/ldd/dominowiki.nsf/dx/routerfallbacknontls
I agree with you that the recent fix is not “working fine”. There is no internet requirement that the certs be from an authority and not self-signed to send encrypted SMTP (the process is not being used to verify identity, only to create an encrypted SMTP stream). If that is something IBM broke via the recent IF1 update, then they need to add a notes.ini setting to turn the requirement off that it won’t accept self-signed certs.
Subject: There were no changes to self-signed certs in this IF
Please see my earlier responses regarding SSLv2.
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/SSLv2 http://www-10.lotus.com/ldd/dominowiki.nsf/dx/SSLv2