SSL Handshake problem - Error 4163

Hi!

We have an application that provides the function for smartcard users to register their personal certificates (contained in the card) on Domino directory.

The application runs well in R5, but when we tested it under R6 (6.02CF1), the following problem shown at the console:

“HTTP Server: SSL handshake failure, IP address [xxx], Keyring [xxx], [SSL Error: Bad or missing remote certificate], code [4163]”

Following is a brief explaination on the working of the application.

  • The digital IDs are stored in smartcard, which is read by USB reader connected to the PC.

  • The smartcard software (on local PC) contains a verifier and a proxy. In IE, it was setup that all secured connection will be routed to the proxy.

  • The verifier will be launched by the proxy whenever IE hits a server requesting client authentication over SSL.

  • User then enter the PIN and the personal certificate is passed to Domino server for authentication.

In R5, the process shown above run smoothly without any problem. However in R6, the process got error during the handshaking with server, and proxy records a “Web server handshake failed” error.

However, if I used the normal digital IDs (PKCS12 installed in certificate stores), no problem happened. Besides that, one thing to note is that the digital IDs has some extra fields compared to normal one (however the vendor explained that they follow the x509 stadard that accepts some custom fields)

As conclusion, we can say that the problem comes from the smartcard application or digital ID format. My problem now is, I cannot extract the digital ID out from the card, to test it without using the smartcard software. That’s why I have no idea which is the real cause of the problem.

I’m lost here. Anyone can provide some help?