ERR_SSL_PROTOCOL_ERROR - Renewed Let's Encrypt certificate in IIS

Question
1

I had a Let’s Encrypt certificate installed on my IIS web site that just expired today. The Certify application completed the renewal process, but I have my own semi-automated deployment application, so the PFX file was just sitting there. I tried to install it with my deployment app (.NET), but it had some issues and I couldn’t apply the new certificate to the https binding.

I deleted the certificate from the WebHosting store through the certlm management console and then re-imported it manually from there using the PFX. This time I was able to apply the new certificate to the https binding without error in IIS, but when I try to visit the web site, I’m getting the following error codes:

Google Chrome/Microsoft Edge (Chromium): ERR_SSL_PROTOCOL_ERROR
Mozilla Firefox: SEC_ERROR_BAD_SIGNATURE
IE11: Can’t connect securely to this page (no error code)

I’m getting the same results on Windows 10, Windows 11 and Android OS on different networks. I ran an SSL test through SSL Labs and SSL Shopper and everything seems to indicate the certificate is valid.

I’ve fully rebooted the Windows Server 2018 server where IIS is running, but I’m still getting the error. I’ve deleted and re-imported the PFX file multiple times without change. I even deleted the https binding completely, restarted the site, then recreated the binding to see if that would help, but it didn’t.

I realize that IIS can be a little “finicky” when it comes to CSR’s and certificates that weren’t generated through their own process, but I was able to use the previous Let’s encrypt certificate without a problem. I also have another Let’s Encrypt certificate deployed for my network security appliance which seems to be working without a problem, so the issue is clearly either with the certificate PFX or within IIS. I’m just not sure which at this point. In the meantime, my site is effectively shut down until I can get this resolved.

I looked at another, similar issue here in the community: Certificate is valid however, sites using it receiving ERR_SSL_PROTOCOL_ERROR - Question - Certify The Web - Support Community. It mentioned that changing the RSA key to a 4096-bit key fixed their issue, but my key is already 4096-bit (as seen in the SSL Labs test). I’m probably just overlooking something, but I don’t have a clue what that might be.

2

Hmm, I’d suspect private key permissions maybe? What’s the reason you don’t use the Certify IIS deployment, is it because you’re deploying to some other machine?

I’ve seen issues before where you sometimes have to import a PFX twice (in .net) to fix private key issues, if your import process is running as SYSTEM (Certify actually does this).

I’d also try changing your IIS https binding to something other cert, save it, then set it back.

3

Thanks for the reply, @webprofusion

Hmm, I’d suspect private key permissions maybe? What’s the reason you don’t use the Certify IIS deployment, is it because you’re deploying to some other machine?

Yes. I have the Certify application installed on a “central” server. It renews four different certificates (subdomains), then I use my application to deploy them to each of the endpoints - two separate IIS servers (I haven’t tried the second one yet), one SonicWALL firewall (I successfully deployed to this device without any problem), and one WS_FTP server (I haven’t tried that one yet either).

Because I know that private key issues can come up at times, I followe a post on TechNet and ran certutil -store webhosting <thumbprint> to see about deleting the private key. Here’s the result (redacted for brevity):

================ Certificate 0 ================
Serial Number: <SERIAL>
Issuer: CN=R3, O=Let's Encrypt, C=US
 NotBefore: 5/17/2022 12:22 PM
 NotAfter: 8/15/2022 12:22 PM
Subject: CN=portal.ciaokc.com
Non-root Certificate
Cert Hash(sha1): <HASH>
  Key Container = <GUID>
  Unique container name: <NAME>
  Provider = Microsoft Enhanced Cryptographic Provider v1.0

Certificate Public Key:
Version: 3
Public Key Algorithm:
    Algorithm ObjectId: 1.2.840.113549.1.1.1 RSA
    Algorithm Parameters:
    05 00
Public Key Length: 4096 bits
Public Key: UnusedBits = 0
<HEX>
Key Id Hash(rfc-sha1): <HASH>
Key Id Hash(sha1): <HASH>
Key Id Hash(bcrypt-sha1): <HASH>
Key Id Hash(bcrypt-sha256): <HASH>

Container Public Key:
Public Key Algorithm:
    Algorithm ObjectId: 1.2.840.113549.1.1.1 RSA
    Algorithm Parameters: NULL
Public Key Length: 2048 bits
Public Key: UnusedBits = 0
<HEX>
Key Id Hash(rfc-sha1): <HASH>
Key Id Hash(sha1): <HASH>
Key Id Hash(bcrypt-sha1): <HASH>
Key Id Hash(bcrypt-sha256): <HASH>

ERROR: Certificate public key does NOT match stored keyset
Encryption test FAILED
CertUtil: -store command completed successfully.

I’m not exactly sure what all that means (LOL), but the ERROR seems to be a pretty good indicator that the cert is not getting imported properly for some reason.

4

Sorry I’ve not seen that error before but how are you performing the import exactly? There is the concept of ephemeral and stored keys etc and sometime the flag you use in .net when storing the cert will matter. I assume your app runs on the servers, or does it do stuff remotely?

5

I understand. Right now, I’m working with the cert I imported manually through the certlm management console, so my application isn’t involved in the current process (unless it messed something up in the store, or something).

Interestingly, I ran the same certutil command on the previous certificate (the one that just expired) and I got the following, very different response:

webhosting "Web Hosting"
================ Certificate 3 ================
Serial Number: <SERIAL>
Issuer: CN=R3, O=Let's Encrypt, C=US
 NotBefore: 3/3/2022 12:33 PM
 NotAfter: 6/1/2022 12:33 PM
Subject: CN=portal.ciaokc.com
Non-root Certificate
Cert Hash(sha1): <HASH>
  Key Container = portal.ciaokc.com [Certify] 3/3/2022 11:33:31 AM to 6/1/2022 12:33:30 PM
  Unique container name: <NAME>
  Provider = Microsoft Enhanced Cryptographic Provider v1.0
Private key is NOT exportable
Encryption test passed
CertUtil: -store command completed successfully.
6

Try your import using Powershell instead of the UI, that will make your process repeatable without having to remember which options you picked:

7

An alternative for deploying to multiple IIS servers is to use the Centralized Certificate Store feature (CCS) which is where you copy the PFX file with a specific naming scheme to a share all the machines can see (or a share on each machine) and you configure the IIS binding to use CCS (pickup from the path you give it). Certify has a Deploy to CCS task which handles things like the naming and can copy to a network location etc.

8

Try your import using Powershell instead of the UI, that will make your process repeatable without having to remember which options you picked:

Well, I must’ve done something wrong with the PS command b/c it put it in the wrong store, but it did import the PFX file without an error. Even so, everything looks the same - the browsers are giving errors and the certutil command on the new certificate is still reporting an error as well.

I’ll definitely look into the CCS option. In the meantime, I still have no idea what’s causing the problem. I don’t think that re-requesting the cert from Let’s Encrypt would solve the problem, but that’s kinda where my mind is leading me at this point. While the certificate appears to be valid, there’s something strange going on with it so perhaps a “fresh” version would give IIS a chance to start over.

9

Does the cert work on the original machine that ordered the cert? You can test this by editing your hosts file to add a fake entry that points to your machine IP then using the same machines browser to browse to that website (you also need a webserver running on that machine with an https binding for the given name).

Also, when you get the original PFX are you just copying it C:\ProgramData\certify\assets or are you exporting it manually from windows? I’d just use the original file.

10

Also, when you get the original PFX are you just copying it C:\ProgramData\certify\assets or are you exporting it manually from windows? I’d just use the original file.

I’m using the original file from the certify\assets directory - no export or manipulation of that file.

Does the cert work on the original machine that ordered the cert?

I don’t have a web server running on that machine, so it’ll take some work to test that.

The strangest thing to me is that I was able to get the last Let’s Encrypt certificate (which was also requested through the Certify application with the same configuration) to work without a problem, even when it was deployed through my .NET application. It took me a few tries to get it right, but it still worked. This time, I’m bypassing my application (after the failed attempt).

11

Try also using certlm.msc on the misbehaving machine and right click the certificate > All Tasks > Manage Private Keys… - if you get an error then there’s a permission problem on the private key (if not it’ll just show the current permissions). When you import the PFX try making it exportable.

12

I should say I assume you’re logged in as Administrator and that the cert is importing into the Local Machine certificate store and not the User certificate store.

1 Like
13

I assume you’re logged in as Administrator and that the cert is importing into the Local Machine certificate store and not the User certificate store.

Yes, I’m logged in with my administrator account. Just to be on the safe side, though, I logged out and back in with my “main” administrator account and tried the whole process:

  1. Deleted the certificate from the store/Refresh
  2. Checked the bindings in IIS (nothing listed) - Stop/Start the web site
  3. Re-imported the certificate from the original PFX, marking it as exportable (no error)
  4. Check the https binding in IIS - It already had the new certificate listed, so I switched it to the expired one and stopped/restarted the site again. (The site was back to showing the warning for the expired certificate)
  5. Changed the https binding back to the new certificate and stopped/restarted the site again.

This leaves me exactly where I was before with the same errors.

Try also using certlm.msc on the misbehaving machine and right click the certificate > All Tasks > Manage Private Keys…

Interesting… I don’t get the option to “Manage Private Keys” when I right-click the certificate. The only choices I have are to “Open” or “Export”. This is true for all of the certificates in the certlm management console (Run as Administrator). I even tried going through the steps I found on StackOverflow very carefully (twice) and that option still didn’t show up.

But, reading a bit further, there was a suggestion that Windows 10 1809 (this is Windows Server 2018, so, close enough) removed that option for any certificate not in the Personal store (mine is in the Web Hosting store), so I dragged that cert into the Personal store and tried again. This time, the “Manage Private Keys” option does show up and opens the property window without any error. SYSTEM, NT AUTHORITY, and local administrators have permission.

Just for testing, I left the certificate in the Personal store and used that to reset the https binding on the site. Unfortunately that didn’t work either.

When you import the PFX try making it exportable.

I usually do, but I’ve tried with both exportable and non-exportable. Sorry to be such a pain, but this is really frustrating. I can’t see any reason for it not to work exactly as expected like it did last time.

14

Yes it’s a weird one, btw I don’t think Server 2018 is a real thing, so I’m guessing its Server 2019 :slight_smile:

Does your IIS Application pool run as a special user or just some default?

15

Yep. You caught me. It’s 2019. :blush:

The Application Pool doesn’t show any special user. The Identity is just set to NetworkService.

16

OK, so you may want to double check the private key permissions allow networkservice read (because it’s neither SYSTEM or Administrator).

If that does nothing you could also try certutil --repairstore <serial> as per this article to see if the private key storage can be fixed:
https://www.entrust.com/knowledgebase/ssl/what-are-the-steps-to-recover-the-private-key-of-an-ssl-certificate-in-an-iis-environment

17

Well, I went ahead and re-issued the cert from Certify, just in case. All the same steps with the new PFX resulted in exactly the same issue.

I added NetworkService to the permission list on the private key (I checked and it wasn’t on the old one, but I’m happy to try anything at this point), but still no luck.

When I run the certutil -repairstore webhosting <serial> (elevated command prompt), it pops up asking me for a smart card. When I cancel that (b/c I don’t have one), it gives me the same errors.

ERROR: Certificate public key does NOT match stored keyset
Encryption test FAILED
CertUtil: -repairstore command FAILED: 0x80090010 (-2146893808 NTE_PERM)
CertUtil: Access denied.
18

Haven’t seen that before! Ok, it’s definitely still having trouble with the private key. I’d bet if you manually export the PFX from the certificate store of the original machine and import to the other machine it will work ok, which doesn’t really solve the problem but is something you could script.

19

Well, that’s certainly something to try. I’ll see what happens.

Trying to think of anything, I’m just going to leave this here in case it sparks any inspiration:

image

image

image

image

20

And, that’s a big, fat “NOPE!”. I imported the latest PFX into the Personal certificate store and checked it with certutil -store my <serial>. It gives me the exact same error as before on this machine.

Just for further testing, I went ahead and imported the PFX for the one cert that I’ve successfully installed so far. Same machine, same store, same settings… I ran certutil -store my <serial> on that certificate and it passed without error.

================ Certificate 1 ================
Serial Number: <SERIAL>
Issuer: CN=R3, O=Let's Encrypt, C=US
 NotBefore: 5/25/2022 10:22 AM
 NotAfter: 8/23/2022 10:22 AM
Subject: CN=firewall.ciaokc.com
Non-root Certificate
Cert Hash(sha1): <HASH>
  Key Container = <GUID>
  Unique container name: <NAME>
  Provider = Microsoft Enhanced Cryptographic Provider v1.0
Encryption test passed
CertUtil: -store command completed successfully.

Now I’m comparing the settings between the two to see if I can find the culprit.

The only difference I see (besides the fact that the “broken” one is for IIS and the “working” one is for a SonicWALL appliance) is that the “working” one doesn’t have a CSR specified in the Managed Certificates configuration settings in Certify whereas this one for IIS does. IIRC, I set that up with a generic CSR, so maybe that has something to do with it? Again, though, it worked last time, so I can’t imagine why it wouldn’t work this time, but still…

WORKING
image

Of course, that means that I’m going to have to re-RE-request the certificate, but I’m definitely willing to give it a go.