Hi,I have a PKCS#11 middlware that generates an error when called by Lotus Notes. The call to C_Sign function using CKM_RSA_PKCS (PKCS1 RSA) mechanism with ulDataLen = 90 (<= RSA_MODULUS_LENGTH - 11) returns CKR_DATA_LEN_RANGE - Data length is out of range. This is caused by a card limitation (36 bytes max).
Can someone tell me if there is way to change this call from Lotus and have it calling c_sign with a different length or at least have it performing the hash before the call to c_sign.
That approach to lock the ID file deliberately uses the full range of N-11 instead of the S/MIME oid+hash to make it less likely that an attacker with a rootkit could trick a user into "accidentally"generating a matching resultant value. Starting in 8.0, if that approach fails with CKR_DATA_LEN_RANGE due to somebody encoding an arbitrary limit into their library violating the PKCS#11 spec, Notes will try again using CKM_RSA_X_509 (raw RSA crypto) and perform the PKCS#1 padding before passing the buffer to the PKCS#11 library.