WinCrypt RSA vs Java org.bouncycastle

So here is a question. I have an old code on a Windows system that just takes a short string and makes CryptEncrypt on it with a public RSA key. The minimum working example is here (avoiding any checks to make it shorter. avoiding freeing as well)

std::string testData = "12345678";
HCRYPTPROV context;
CryptAcquireContext(&context, nullptr, nullptr, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT | CRYPT_SILENT);

auto pubK = FromBase64(PublicKey);
CERT_PUBLIC_KEY_INFO *pubKeyInfo = nullptr;
DWORD keyLength = 0;
CryptDecodeObjectEx(X509_ASN_ENCODING, X509_PUBLIC_KEY_INFO, 
    pubK.data(), pubK.size(), 
    CRYPT_ENCODE_ALLOC_FLAG, nullptr, 
    &pubKeyInfo, &keyLength);

HCRYPTKEY key;
CryptImportPublicKeyInfo(context, X509_ASN_ENCODING, pubKeyInfo, &key);

std::vector<std::uint8_t> result(testData.begin(), testData.end());
DWORD size = testData.size();

CryptEncrypt(key, NULL, true, 0, result.data(), &size, result.size());
result.resize(size);
size = testData.size();
CryptEncrypt(key, NULL, true, 0, result.data(), &size, result.size());

std::cout << ToBase64(result) << "n";

The code works and returns a base64 encoded string. like lTq01sOcgDcgwtDaFFoHH/Qb6xLw0KU+/n/+3t0eEb8l4N69QGcaEWf1qso3a4qn7Y8StlXcfMe8uspNF/KDj6qQOMvCuM+uUl+tkLd5NXiESsjycgjyxAqdCIO71iTSmsYVcsS3fY/gtIbO4UAFnCRPOXoSyqWqpXW7IRtFzL2N3MxgIBlIMErNvNWs5HPA7xAY/XO6UpSMWsQO4ppccdeNLSZDPwOxohKD/BX5oDin81nFn7fvIZgghfH5knF1nezK8IGKl+vtbgrwlUUULp/wJ4POceyIn0HaZoVsaCu6xFJcUJGfBqSvm4GZqkp2MlGxBODku0OSgEfIDEGMTg==.

Then I have to decrypt this string with the private key on another side running java. I use bouncycastle:

    try {
        Security.addProvider(new BouncyCastleProvider()); 
        String value = "";
        AsymmetricKeyParameter privateKey = 
            (AsymmetricKeyParameter) PrivateKeyFactory.createKey(Base64.getDecoder().decode(privateKeyValue));
        AsymmetricBlockCipher e = new RSAEngine();
        e = new org.bouncycastle.crypto.encodings.PKCS1Encoding(e);
        e.init(false, privateKey);

        byte[] messageBytes = Base64.getDecoder().decode(inputdata);
        byte[] hexEncodedCipher = e.processBlock(messageBytes, 0, messageBytes.length);
        value = new String(hexEncodedCipher);
        System.out.println(value);
        return value;
    }
    catch (Exception e) {
        System.out.println(e);
    }

And this code shows me the next error:

org.bouncycastle.crypto.DataLengthException: input too large for RSA cipher.

And I believe I’m missing something on the windows side, because if I use the same keys on the Java side and crypt the same data with the same public key, the decryption works as expected.

Here the keys, I generated with openssl for this question (RSA 2048):
Private:

MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCeL7K++3+vl52Q
WFC6ThgvNPlhHLlP5ylIMi/7Gy60wcCHtx8R5Hzi2j7Kx/uBXyr7SCbePS7NtqHx
meVK3VhEvYHz2uYaUNf6GqJgNNjfRymnL5ll8K8lq1wym6A7KZ+L3qLHH1oI6GfW
+uf32nUfy1PQvsatPN7aiJ4ymiaJj2LcI6Bp78CINsu56Cx9QBN9uoXqKO/7NjOz
y2GRdYdckiqCkcmGDzA4/5tJROxj21gUbxwoUR+yz2sVFGlcJycsDDJiIXiPjbVN
XgM/YfmRjdAPQQwXXj9AK5w/dp9TPq621WqYES74rIY05cba4v1kihmFbs0DHLrV
fQZB0Pu9AgMBAAECggEAGuD//nO9vpiErYJUNVQPx/W4akf3NRySZzIf9QspZI2H
qYf0P5YTonhzMwHIOrNxGkGoWRsMWOgvnF4KGC6EUSniaw1HDDGwgU8FSFOyhj4R
VddAuZGsMTps8CyBjYwFED9EaZFqOxlCi8UWpYb5X+2s0EuadtVhCMEuIGsRIU53
mfW11182YtbAI7Zqi7wg/w0yRz5rVj4ph8nbSFPgi6qtVWA9bxVxNeRTXyQDWjUU
NFQzGcRG6D/SYTnRzTndVM5TmTwEVt0hvJNA2/pMWF5XMu6B2exOpJIVVVdQ40l/
XHh33SU8+gQYY56rHzFebCa0Nuxwdk0paluv+x3CAQKBgQDKwe8QXfNlMDqrM06n
iq/NwKE7tB+1gs1nKr4qZdbI71IzgQJgIJcxbJwbuE6z6Yk8PoancEELH9mSY7EN
YGeSO3QsO1LF1vyJWgWQU1G3pfC9sh8sY6f9WYg8+JogIeJsgjwf7PXrGaBq8++Q
GBMUATzPAh/ERIqUip0nEQ2IJwKBgQDHuYgPvv77qhLc+ZNOdhaZtibeVsFx50V+
a+qR+INSTJ/CChNkoVtMg803ZQWckJBOYL/TS2Mw+ctNomUg+ImjULGT9GzqMeme
HkPpAj5pNyNVpRfVNZfmS2cwwuRyE1QbFGs3C4p7D5MKKCfl4/8MBQXtJGWrQZFr
owh2NNyHewKBgHsMmSYorlcBnwlZOOnK7AiFWBRgq0G/4SI0OXaHmYMWYp+pMqTe
AoPXMyJLh0/+ce/izlt9b6vtp2AFKmVA1XpUpJtXYVN5tocw3+GH/zbh+SlWmT6a
OFAz7s953CeWCNDrdMu3RkNoqQdfhUrAoYtpeNr0ogy9wBCH0vnrinfPAoGBALNy
U/hpz+lH1qjqKFsPqKC001ljM21msL60sU4zrbHNHKEXsnLwsvodVc3Wm2MfVDjH
nrJ2gomnde2r4hbsl6W/w70+mHkXHWKuqK97D54zJzE1IyOygmctCmr6QIzqJuAp
yWbsnKCSzrcKe0aHQkmHXdrCoAJt5/2AvwKN3jJvAoGAIaLts1F7shIZ365FXyWD
poOFtXsNRXESkcYpJ2Pn+K0fUNtdH4BtR/5Z5bsEWXBf2LbL7s15UZFZbck0KH3K
22BRgPQze7wEhMVFlIMNhF17WrOh2NTGUAVz2CYSthbYh0QSI+5XJk0AEfiQcqYk
+E4bZw/RUTY94V+ITEK6F8g=

public

MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAni+yvvt/r5edkFhQuk4Y
LzT5YRy5T+cpSDIv+xsutMHAh7cfEeR84to+ysf7gV8q+0gm3j0uzbah8ZnlSt1Y
RL2B89rmGlDX+hqiYDTY30cppy+ZZfCvJatcMpugOymfi96ixx9aCOhn1vrn99p1
H8tT0L7GrTze2oieMpomiY9i3COgae/AiDbLuegsfUATfbqF6ijv+zYzs8thkXWH
XJIqgpHJhg8wOP+bSUTsY9tYFG8cKFEfss9rFRRpXCcnLAwyYiF4j421TV4DP2H5
kY3QD0EMF14/QCucP3afUz6uttVqmBEu+KyGNOXG2uL9ZIoZhW7NAxy61X0GQdD7
vQIDAQAB

What do I wrong?

Answer

CryptEncrypt returns the ciphertext in little-endian format, see CryptEncrypt( Remarks, last section). For the decryption in Java the byte array messageBytes must therefore be inverted, e.g. here.