SSL TLS SMTP setup and configuration issues

HCL Domino Domino Forum
1

Could I firstly ask for a straw poll to see who has actually got this working? Not forgetting that if a STARTTLS session fails the SMTP server will retry in clear text, so if you are not running the debug commands and checking the logs it may not be working.

There seem to be several variables here that are difficult to pin down. In essence the SSL certificate is a 128Bit Certificate issued by a Domino CA using the request database. There are several options in the Key when signing by the CA that is not well documented.

Is there anybody with some real world experience in TLS? I am happy to share my findings but I must admit I cannot get it to work!

The debug commands are:

SMTPDebug=3

SMTPDebugIO=3

SMTPCLIENTDEBUG=3

TraceSSLHandshake=1

ReportSSLHandshakeErrors=1

SmtpSaveOutboundToFile=1

debug_outfile=c:\trace\ssloutfile.txt

A typical client session would be:

250-TLS

250-HELP

250-AUTH LOGIN

250-STARTTLS

250-SIZE

250 PIPELINING

STARTTLS

220 Ready to start TLS

Server key (1024 bits) too strong for EXPORT ciphers. Disabling cipher RSA_EXPORT_WITH_RC4_40_MD5

SSL handshake error: 1C7Ah

Attempting to Disconnect:

CommandQUIT:

Connection terminated with status: 2562

And the server would see

State change from Greeting to Greeting

Processing in Greeting state

State change from Greeting to Connected

Processing in Connected state

EHLO command received

Processing in Connected state

STARTTLS command received

Processing in Connected state

STARTTLS command (cont.)

SSL Error: Keyring File access error

State change from Greeting to Greeting

2

Subject: Add DEBUG_SSL_ALL=3 and SSL_TRACE_KEYFILEREAD=1 to your notes.ini

The SSL debug parameters that you indicate were useful in R5, but won’t have any effect in R6. Try setting DEBUG_SSL_ALL=3 and SSL_TRACE_KEYFILEREAD=1 in your notes.ini and post that trace here – that will provide enough information for people to try to analyze the problem that you are having.

That said, the only source of configuration advice that I can point you to is the documentation itself:

MAIL

Securing SMTP sessions using the STARTTLS extension

SMTP sessions conducted over a standard TCP/IP channel are vulnerable to eavesdropping because the unencoded transmission can be easily intercepted. To protect SMTP communications, servers can use transport-layer security (TLS), more commonly known as SSL encryption, to provide privacy and authentication.

Some servers support SSL for SMTP communications by sending and receiving SMTP traffic through the SSL port (port 465 by default) only. However, because this requires that both the sending and receiving servers support SMTP over SSL, this solution isn’t always practical.

To provide SSL security for SMTP transfers over TCP/IP, Domino supports the use of negotiated SSL. In a negotiated SSL scheme, the sending and receiving hosts each use the SMTP STARTTLS extension, defined in RFC 2487, to signal their readiness to negotiate an SSL connection. The receiving server displays the STARTTLS keyword in response to the sending server’s EHLO command. The sending server issues the STARTTLS command to request the creation of a secure connection. After the initial TLS handshake completes successfully, the two parties proceed to set up an SSL channel between them. Both the sending and receiving server must possess SSL certificates.

For more information on obtaining server certificates, see the topic “Setting up SSL on a Domino server.”

Supporting STARTTLS for outbound SMTP sessions

A Domino server configured to use negotiated SSL for outbound mail connects to the receiving server’s SMTP TCP/IP port (port 25 by default). If the initial SMTP response from the receiving server indicates that it supports the STARTTLS extension, Domino issues the STARTTLS command to request the use of SSL to encrypt the rest of the session.

If the receiving server did not advertise support for STARTTLS in response to the Domino server’s EHLO command, the sending Domino server continues with an unencrypted SMTP TCP/IP session.

To enable outbound STARTTLS support, “set the SMTP outbound TCP/IP port status to: Negotiated SSL.”

Supporting STARTTLS for inbound SMTP sessions

You can configure Domino to support the STARTTLS command for inbound SMTP transactions. When a Domino SMTP server is set to use negotiated SSL for inbound sessions, the server advertises support for STARTTLS in response to EHLO commands the TCP/IP port receives from connecting hosts. A connecting host can then issue the STARTTLS command to request an encrypted session.

If Domino is configured to require STARTTLS for SMTP sessions over TCP/IP and a connecting host cannot meet this demand, no mail is sent over the connection.

To enable inbound STARTTLS support:

“Enable the SMTP listener task.”

“Enable the SMTP inbound TCP/IP port.”

“Enable the STARTTLS ESMTP extension.” This causes Domino to advertise STARTTLS as one of its supported extensions in the ESMTP EHLO greeting response.

(Optional) Enable name-and-password authentication for the SSL port. Although SMTP sessions that use negotiated SSL are conducted over the Domino TCP/IP port, Domino uses the authentication options you set for the server’s SSL port to determine how to handle name-and-password arguments.

Requiring name and password authentication for SMTP STARTTLS sessions

Enabling ESMTP support for negotiated SSL allows a server to accept requests to use SSL over TCP/IP from remote servers that connect anonymously. However, not all inbound connections are anonymous. A connecting SMTP server may be configured to send Domino a name and password by means of the ESMTP AUTH command.

To support connections from SMTP clients that send a name and password during a negotiated SSL session, set the value of the Name & password field for the SMTP inbound SSL port to Yes. You do not have to enable the SSL port. If the SSL port does not support name-and-password authentication, the Domino SMTP server rejects the AUTH command from the remote server and returns an error indicating that the command is not implemented.

Even though Domino receives the AUTH command over the TCP/IP port, Domino uses the SSL name-and-password authentication settings to determine whether to accept the AUTH request because it receives the command in the context of an SSL session. The Name & password authentication setting for the TCP/IP port is ignored.

3

Subject: Wow - usefull information - Thanks Dave

You are my hero when Batman dies, I used to say Superman but then he died!

I am glad I did not mention how much money!

The problem for the receiving server was that the SSL key was called TTkeyfile.kyr. It seems to look for keyfile.kyr by default!

TLS worked - SSL session established!!! Warwickshire County Council will be happy and so will the guys doing the CPS secure mail system including Microsoft and tomorrow A huge Mobile phone company will be happy when this could actually work. Need the info got the info fixed the problem, thanks again Dave.

The originating server still has an SSL error but I will redo the keys. It did initiate the session correctly though. It is getting late now so more work later today.

Here is how I tested and what I saw.

Few problems spotted. The server was using keyfile.kyr and I had the smtp inbound set to ATNkeyfile.kyr. Easily fixed, copied and renamed. Where is the outbound certificate set?

Here is the trace.

The first part is from my server Domino 6.0.1 and the second to a domino 6.0.1 server (The system time is not the same on the servers!)

20/03/2003 00:16:22 [0550:0008-0A7C] SMTPClient: CommandSTARTTLS: STARTTLS

20/03/2003 00:16:22 [0550:0008-0A7C] SMTPClient: ReceiveResponse: 220 Ready to start TLS

20/03/2003 00:16:22.46 [0550:0008-0A7C] ReadKeyfile> Recovering password from stash file

20/03/2003 00:16:22.50 [0550:0008-0A7C] ReadKeyfile> Password is password

20/03/2003 00:16:22.50 [0550:0008-0A7C] ReadKeyfile> Reading keyfile d:\notesrv\data\keyfile.kyr

20/03/2003 00:16:22.83 [0550:0008-0A7C] ReadKeyfile> Looking for trusted roots

20/03/2003 00:16:22.85 [0550:0008-0A7C] ReadKeyfile> Found trusted roots

20/03/2003 00:16:22.85 [0550:0008-0A7C] ReadKeyfile> Exit status = 0

20/03/2003 00:16:22.88 [0550:0008-0A7C] ReadKeyfile> Recovering password from stash file

20/03/2003 00:16:22.88 [0550:0008-0A7C] ReadKeyfile> Password is password

20/03/2003 00:16:22.88 [0550:0008-0A7C] ReadKeyfile> Reading keyfile d:\notesrv\data\keyfile.kyr

20/03/2003 00:16:22.88 [0550:0008-0A7C] ReadKeyfile> Looking for cert chain

20/03/2003 00:16:22.88 [0550:0008-0A7C] ReadKeyfile> Got cert chain

20/03/2003 00:16:22.88 [0550:0008-0A7C] ReadKeyfile> Exit status = 0

20/03/2003 00:16:22.89 [0550:0008-0A7C] ReadKeyfile> Recovering password from stash file

20/03/2003 00:16:22.89 [0550:0008-0A7C] ReadKeyfile> Password is password

20/03/2003 00:16:22.89 [0550:0008-0A7C] ReadKeyfile> Reading keyfile d:\notesrv\data\keyfile.kyr

20/03/2003 00:16:22.89 [0550:0008-0A7C] ReadKeyfile> Looking for private key

20/03/2003 00:16:22.89 [0550:0008-0A7C] ReadKeyfile> Decoding keys

20/03/2003 00:16:22.89 [0550:0008-0A7C] ReadKeyfile> Keys decoded

20/03/2003 00:16:22.89 [0550:0008-0A7C] ReadKeyfile> Exit status = 0

Checking keyfile certificates:

20/03/2003 00:16:25.33 [0550:0008-0A7C] SSLCheckCertChain> Valid certificate chain received

20/03/2003 00:16:25.33 [0550:0008-0A7C] int_MapSSLError> Mapping SSL error 0 to 0

20/03/2003 00:16:25.33 [0550:0008-0A7C] SSL_Handshake> Enter

20/03/2003 00:16:25.35 [0550:0008-0A7C] SSL_Handshake> Current Cipher 0x0000 (Unknown Cipher)

20/03/2003 00:16:25.35 [0550:0008-0A7C] SSL_Handshake> SSL Undetermined attempt

20/03/2003 00:16:25.46 [0550:0008-0A7C] S_Write> Enter len = 60

Xmt buffer:

00000000: 3A80 0301 0000 0021 0000 0010 0400 0000 ‘.:…!..’

00000010: 0005 0A00 0000 0009 6200 0000 0003 0200 ‘…b…’

00000020: 0000 0001 0100 0001 0280 8000 C7B7 4DC9 ‘…7GIM’

00000030: B46B 914E D76B 9D24 B7D4 9815 ‘k4N.kW$.T7…’

20/03/2003 00:16:25.47 [0550:0008-0A7C] S_Write> Switching Endpoint to sync

20/03/2003 00:16:25.47 [0550:0008-0A7C] S_Write> Posting a nti_snd for 60 bytes

20/03/2003 00:16:25.47 [0550:0008-0A7C] SSL_EncryptData> SSL not init exit

20/03/2003 00:16:25.55 [0550:0008-0A7C] S_Write> Switching Endpoint to async

20/03/2003 00:16:25.55 [0550:0008-0A7C] SSL_EncryptDataCleanup> SSL not init exit

20/03/2003 00:16:25.58 [0550:0008-0A7C] S_Write> nti_done return 60 bytes rc = 0

20/03/2003 00:16:25.58 [0550:0008-0A7C] S_Write> Exit, wrote 60 bytes

20/03/2003 00:16:25.58 [0550:0008-0A7C] S_Read> Enter len = 1

20/03/2003 00:16:25.58 [0550:0008-0A7C] S_Read> Switching Endpoint to sync

20/03/2003 00:16:25.58 [0550:0008-0A7C] S_Read> Posting a nti_rcv for 1 bytes

20/03/2003 00:16:25.58 [0550:0008-0A7C] SSL_RcvSetup> SSL not init exit

20/03/2003 00:16:25.58 [0550:0008-0A7C] S_Read> Switching Endpoint to async

20/03/2003 00:16:25.60 [0550:0008-0A7C] S_Read> nti_done return 0 bytes rc = 9

20/03/2003 00:16:25.60 [0550:0008-0A7C] S_Read> nti_done return 0 bytes rc = 9 Event = 0x400

20/03/2003 00:16:25.60 [0550:0008-0A7C] SSL_Handshake> After handshake state= 2 Status= -6989

20/03/2003 00:16:25.60 [0550:0008-0A7C] SSL_Handshake> Exit Status = -6989

20/03/2003 00:16:25.60 [0550:0008-0A7C] int_MapSSLError> Mapping SSL error -6989 to 4165

20/03/2003 00:16:25.61 [0550:0008-0A7C] SSL_EncryptData> SSL not init exit

20/03/2003 00:16:25.61 [0550:0008-0A7C] SSL_EncryptDataCleanup> SSL not init exit

20/03/2003 00:16:25.61 [0550:0008-0A7C] SSL_RcvSetup> SSL not init exit

20/03/2003 00:16:25 [0550:0008-0A7C] SMTPClient: SSL handshake error: 1C7Ah

20/03/2003 00:16:25 [0550:0008-0A7C] SMTPClient: Attempting to Disconnect:

20/03/2003 00:16:25 [0550:0008-0A7C] SMTPClient: CommandQUIT:

20/03/2003 00:16:25 [0550:0008-0A7C] SMTPClient: Connection terminated with status: 2562

Uhmm, Bad password? I am renaming the keyfile now… No keyfile… It was caled TTkeyfile.kyr? Copied and renamed.

0A80:0009-0AC8] R: STARTTLS

20/03/2003 00:21:08.08 [0A80:0009-0AC8] SMTP CITask StateMachine> Sent 24 bytes

to 80.192.219.41

[0A80:0009-0AC8] S: 220 Ready to start TLS

20/03/2003 00:21:08 SMTP Server: pc-80-192-219-41-nm.blueyonder.co.uk (80.192.

219.41) connected

20/03/2003 00:21:08.15 [0A80:0009-0AC8] ReadKeyfile> Recovering password from st

ash file

20/03/2003 00:21:08 SMTP Server [0A80:0009-0AC8] Processing in Connected state

20/03/2003 00:21:08 SMTP Server [0A80:0009-0AC8] EHLO command received

20/03/2003 00:21:08 SMTP Server [0A80:0009-0AC8] Processing in Connected state

20/03/2003 00:21:08.19 [0A80:0009-0AC8] ReadKeyfile> Password is

20/03/2003 00:21:08.19 [0A80:0009-0AC8] ReadKeyfile> Reading keyfile d:\notesrv\

data\keyfile.kyr

20/03/2003 00:21:08.19 [0A80:0009-0AC8] ReadKeyfile> Read failed: bad password

20/03/2003 00:21:08.19 [0A80:0009-0AC8] ReadKeyfile> Exit status = 272

20/03/2003 00:21:08 SMTP Server [0A80:0009-0AC8] STARTTLS command received

20/03/2003 00:21:08 SMTP Server [0A80:0009-0AC8] Processing in Connected state

20/03/2003 00:21:08 SMTP Server [0A80:0009-0AC8] STARTTLS command (cont.)

20/03/2003 00:21:08.24 [0A80:0009-0AC8] int_MapSSLError> Mapping SSL error -6982

to 4162

20/03/2003 00:21:08 SSL Error: Keyring File access error

4

Subject: RE: Wow - usefull information - Thanks Dave

The first server seems to be behaving correctly from your trace:

20/03/2003 00:16:25.60 [0550:0008-0A7C] S_Read> nti_done return 0 bytes rc = 9 Event = 0x400

20/03/2003 00:16:25.60 [0550:0008-0A7C] SSL_Handshake> After handshake state= 2 Status= -6989

20/03/2003 00:16:25.60 [0550:0008-0A7C] SSL_Handshake> Exit Status = -6989

20/03/2003 00:16:25.60 [0550:0008-0A7C] int_MapSSLError> Mapping SSL error -6989 to 4165

20/03/2003 00:16:25.61 [0550:0008-0A7C] SSL_EncryptData> SSL not init exit

20/03/2003 00:16:25.61 [0550:0008-0A7C] SSL_EncryptDataCleanup> SSL not init exit

20/03/2003 00:16:25.61 [0550:0008-0A7C] SSL_RcvSetup> SSL not init exit

20/03/2003 00:16:25 [0550:0008-0A7C] SMTPClient: SSL handshake error: 1C7Ah

That segment indicates that the first server (initiating the SSL connection) is sending the ClientHello packet and is then receiving an orderly disconnect from the other server, probably in the nature of a TCP FIN. Unfortunately, it can’t tell what is wrong, only that the other end quit.

As for the second trace, well… did you rename the stash file when you renamed your keyring file? They should have the same name, just different extensions. [.kyr vs .sth]

20/03/2003 00:21:08.19 [0A80:0009-0AC8] ReadKeyfile> Read failed: bad password

20/03/2003 00:21:08.19 [0A80:0009-0AC8] ReadKeyfile> Exit status = 272

Good luck,

dave

5

Subject: RE: Wow - usefull information - Thanks Dave

From the looks at that file negotiated or TSL SSl wasn’t even working in R5 I’ve never seen the option to enable ESMTP. I will give it a godd with R6. and see what happens.

6

Subject: RE: Add DEBUG_SSL_ALL=3 and SSL_TRACE_KEYFILEREAD=1 to your notes.ini

This issue has apparently been resolved in release 6.5.3. Please see release notes for details.

7

Subject: SSL TLS SMTP setup and configuration issues

HiI tried for over 8 months to get SSL TSL to work with R5. This was with an open ticket severity 1 to Lotus the entire time. First problem was in 5.08 it was completely broken, had to wait for 5.10. as SSL e mail did not resend on port 25 if no SSL connection as found, it would just hang in the mail.box. Once 5.0.10 was installed I got everything configured and kept getting cert errors where the other servers said I did not have permission to use SSL TSL.

Another problem was even if you selected negotiated it would still try to connect over port 465, which is the old and out of date Netscape standard of SSL e mail transmission. Lotus could never figure out what was wrong with my cert or config so I gave up after 8 months and had to route certain e mail to an Exchange server which we had to set up to send TSL SSL e mail out which we got working within a couple of days.

In my experience I came to the conclusion no one in the Domino world uses SSL e mail and Lotus themselves had no real idea how to troubleshoot it or get it working. You can do a search on my name in the R5 forum and see my many months of attempts to get some help and how little response I had.

My advice is if it doesn’t work right off the bat give up and relay to an Exchange server setup to send out SSL e mail. I don’t like Exchange very much but in this case it worked without a flaw.

8

Subject: SSL TLS SMTP setup and configuration issues - Need input

Hi Mark,

Thanks for the response. I have read your responses and posts and I seem to have as many, although some has been archived.

I agree totally with you on IBM’s stance here. I have had a ticket out for months as well. I think this issue needs to be escalated.

I will pay good money to anybody that claims they have TLS working. Let me be more specific, TLS working on Lotus Domino 6 encrypting sessions to servers other than Domino. Ah heck, it does not even work Domino to Domino.

I agree that using Exchange is the only answer. Why does nobody raise this as an issue? Does nobody use TLS?

IBM dogmatically clings to standards that just suit themselves while we now have to go and justify damn Exchange servers to perform the simplest of tasks.

Please forum, any assistance on TLS would be appreciated. We are not talking noddy bits of information as Mark and I have both spent enormous amounts of time on this. Our failures are specific to the stage where STARTTLS is negotiated and keys are exchanged. We need to trace that information or find out why that is crashing out. What are the damn error messages and what do they really mean? Need input!

Ray

9

Subject: RE: SSL TLS SMTP setup and configuration issues - Need input

Yea I’d put up 100$ or more to anyone that can help get this working as advertised. Most frustrating experience in my Admin life. No one uses it no one knows how to fix it and even Lotus doesn’t have a clue. At this point they might as well yank the feature. I was hopefull R6 woulld fix it but now I guess not.

10

Subject: RE: SSL TLS SMTP setup and configuration issues - Need input

I have had the misfortune of dancing the same dance, but I think I have some updated information.

We use SSL over SMTP for secure e-mail between systems that use this protocol.

As we know, in the initial iteration of secure email over SSL used Port 465 and is enabled as part of the Domino implementation.

Two issues arise with this implementation that I have seen. First, if a recipient domain has SSL configured to use authentication.

Outbound E-mail delivery will fail with an “authentication required” error and the Domino server will not retry using port 25 SMTP, so no e-mail will be transferred to the recipient domain.

The second issue occurs if a company has a firewall that looks for dynamic attacks or port scanning.

Since Domino try’s port 465 first, then falls back to port 25 if this fails. This behavior often appears as an attack of sorts, so the recipient firewall will block all subsequent e-mail attempts from our servers. To resolve the issue, the receiving e-mail administrator will then need to put our servers on their white lists.

To resolve this issue, we disabled port 465 and changed SMTP port 25 outgoing from Enabled to “negotiated SSL” as per Lotus support. This would force SSL to be negotiated at the SMTP level on port 25 using STARTTLS.

This configuration fails because Domino refuses to fall back to port 25 if SSL negotiation fails. The e-mail would time out and no e-mail will pass to the recipient domain.

This was fixed in 6.5.4 with the addition of an INI setting that will allow the server to retry after failure of SSL negotiation.

RouterFallbackNonTLS=1

Using this setting however caused any outgoing SMTP session whose recipient Primary MX record was un available to send the router into a loop, never delivering the e-mail and negatively effecting e-mail.

Lotus determined this was an issue and have since delivered a hotfix to resolve the issue. This hotfox was applied on Thursday April 14, 2005 to version 6.5.4 of my SMTP servers and the problem seems to have been resolved.

Now I am still vexed with two issues that Lotus has not found solutions for.

The first is a method to verify who sending to us and who we are send to using TLS. When people ask if we use TLS with Xdomain.com all I can say is, uuuuhhhhh I have the box checked.

Secondly, how do I notify the recipient of an e-mail that the message he\she is reading was delivered securely? Outlook and other e-mail clients make it quite obvious that an e-mail was encrypted. I hate to suggest using Outlook.

While on the flip side, how can we force a message to be sent securely via TLS if the content needs to be secure?

Having nifty SMTP e-mail encryption is cool, but somewhat useless if nobody knows and I can not verify the security of any given email (or domain).

11

Subject: RE: SSL TLS SMTP setup and configuration issues - Need input

Hi Mattew,

How can I use server certificate verification in Lotus Domino?