The private key could not be processed. Key should be unencrypted and in PEM format

Andrey · January 26, 2021, 1:39pm

Trying to add custom private key for certificate.
Extension of the private key is *.key,
file content starts from -----BEGIN PRIVATE KEY-----

Program shows that private key file format didn’t match, and unencrypted PEM is needed.

How can I fix this?

Thanks!

webprofusion · January 26, 2021, 2:34pm

Hi, can you confirm what type of key this is (RSA or EC)? Can it be parsed OK using any PEM decoder tool? How was it generated?

Our code simply tries to parse the key file as PEM format, the main limitation is that the key file can’t be password protected. This uses the BouncyCastle OpenSssl implementation.

If you can generate keys on demand, could you supply an example key that doesn’t work? i.e. one that’s not actually private.

Andrey · January 26, 2021, 3:17pm

Key is generated by mail server, thats all i know.
Thanks for the tip about PEM decoder tool,
i’ll try to generate new private key, that matches this tool.

Example key, that doesn’t work is here

(this key is not actualy private)

webprofusion · January 27, 2021, 3:12am

Thanks, so if I run this through openssl it seems to be valid but that the original is not-quite correct base64 encoding (or rather, BouncyCastle can’t/doesn’t want to parse it):

openssl rsa -in C:\temp\key.pem -check

Which then gives me an output which works with Certify:

webprofusion · January 27, 2021, 3:13am

Which email server product is this, maybe I can investigate it a little with them?

Andrey · January 27, 2021, 12:23pm

This is Zimbra Collaboration server.
Thanks for help,
private key for mail server i’v regenerate whith mathched parameters and replace it in console of mail server.

stausend · March 26, 2025, 3:21pm

I am going to necro this post because i ran into the same issue. None of the private keys i generate with openssl work. I would expect that a standard tool like openssl can be used for this.

openssl genpkey -algorithm RSA -out private_key.pem -pkeyopt rsa_keygen_bits:4096

creates a PKCS#8 key. more modern and recommended by openssl. unfortunately does not import.

openssl genrsa -out private_key.pem 4096

creates the classic PKCS#1 RSA key. Does not import into the app either.

Please advise how i can create a private key that is working with the application. This is a requirement. We are licensed users.

webprofusion · March 27, 2025, 2:41am

Thanks, licensed users should contact support at certifytheweb.com when they have specified requirements to discuss.

As you can see from the age of this post custom private keys are not frequently used with the app (currently) but we should definitely should extend that support.

webprofusion · March 27, 2025, 2:51am

In the meantime, to convert your key to one with an RSA header (as expected by the app currently):

openssl rsa -in private_key.pem -out rsa_privatekey.pem -traditional

[A fix will be included in the next major release, the beta for which will be available in the next few weeks]

stausend · March 27, 2025, 9:43am

Thank you for the information. This worked.

stausend · March 28, 2025, 12:19pm

i was giving praise too fast. there are now problems with the created pfx using that key.

openssl pkcs12 -in star.mysite.com.pfx 
Enter Import Password:
Bag Attributes
    localKeyID: 02 B5 08 14 F4 05 2D 89 E8 72 93 6C 12 ED B6 28 0B B9 BF 48 
    friendlyName: *.mysite.com [Certify] 28.03.2025 10:04:39 to 26.06.2025 11:04:38
Error outputting keys and certificates
4057DAA863720000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:../crypto/evp/evp_fetch.c:386:Global default library context, Algorithm (RC2-40-CBC : 0), Properties ()

This does not look like a valid pfx and other tools and appliances like cisco Firewalls seem to have problems with this.

Will you modernize your application to use the current standards soon?

stausend · March 28, 2025, 12:51pm

I don’t know what your app does, but when i manually request a certificate with certbot with my key and password, i get back a working pfx. So your app must somehow change the pfx returned by let’s encrypt.

Hope that helps. If you need any more information feel free to ask.

webprofusion · March 29, 2025, 3:53am

Defaults you need to be aware of:

The default PFX algorithms retain compatibility with older versions of Windows. These conflict with the defaults for openssl 3.0+. You can opt into modern PFX algorithms using Settings > General Settings > Enable Modern PFX Algs
The default PFX has a blank password, you can set this under Certificate > Advanced > Signing & Security, scroll down to password.
In both cases you then need to request your certificate again to rebuild the pfx.

Regarding the openssl error, this is a feature of openssl 3.0+ where they changed the default algorithms understood by pkcs12 and it differs from the default expected on Windows (where the app is typically used).

E.g. to work with an example pfx using default settings:
openssl pkcs12 -in example.pfx -legacy -passout pass: -passin pass:

Note that if your version of Windows Server is an in-place upgrade from an older version you may find PFX using Modern Algs do not function on that server. That’s why we don’t use it as the default.

Let’s Encrypt themselves return a PEM encoded cert (not a PFX), which we convert into a PFX as that’s what’s commonly used on Windows.

I’m intrigued as to why you need to specify a custom private key at all? Is there some sort of legacy hardware that doesn’t let you provide your own private key?