Upcoming expiry of DST Root CA X3 and R3 intermediate for Let's Encrypt

Announcements
1

Summary: If you experience issues with certificates being untrusted by browsers after the 30th of September, reboot your server. For more details read on.

Our constantly updated knowledge base article is here: Let's Encrypt DST Root CA X3 expiry Sept 30th 2021 | Certify The Web Docs

The following information may pre-date our knowledge base article content:

Update: Certify The Web v5.5.4 has an automatic method to disable the expiring R3
In addition to the above, you may find that some servers do not serve the default R3 > ISRG X1 > DST Root CA X3 chain, resulting in incompatibility with old versions of Android. To resolve this, manually install the ISRG Root X1 cross signed to DST Root from (Chain of Trust - Let's Encrypt) https://letsencrypt.org/certs/isrg-root-x1-cross-signed.der:

  • Download the .der file
  • Open certlm.msc, browse to intermediate certification authorities, right-click > All Tasks > Import.., select .der file types and browse to your downloaded file, then complete the import. The updated (default) chain will now be used.

Let’s Encrypt’s DST Root CA X3 root certificate and one version of it’s R3 intermediate will be expiring on the 30th of Sept 2021. The R3 intermediate chained to DST Root CA X3 is replaced by the R3 chained to ISRG Root X1.

When serving your websites, Windows builds a valid certificate chain and uses it for the https (TLS) connection. Windows currently favors the wrong (expiring) R3 because it has a newer start date than the correct one, this is a built-in windows behavior.

Windows is expected to switch to the correct R3 intermediate automatically but if you experience any issues with certificates suddenly being considered invalid, try a server reboot first. If a chain validation issue persists try deleting the old R3 after it expires (the one issued by DST Root CA X3, not the one issued by ISRG Root X1) in the windows certificate manager: Manage Computer Certificates > Intermediate Certification Authorities, again a reboot may be required.

After the DST Root CA X3 expiry, default certificate chains will be Your Certificate > R3 > ISRG Root X1 > DST Root CA X3 (expired). This is because Let’s Encrypt are preserving the use of the expired root for older OS compatibility (Android 7.0 and lower, Windows XP etc). If you experience client issues with this chain you can opt to use the ISRG Root X1 chain without the DST Root CA X3 root. See Frequently Asked Questions | Certify The Web Docs

In the event that the above does not resolve a compatibility issue with clients you need to support, you could consider migrating to an alternative CA with a root that’s trusted by your clients. You will need to investigate which trusted roots are present in your clients trust stores and migrate accordingly. To read more about adding new CA accounts and migrating see: Certificate Authorities | Certify The Web Docs - note that rate limits do apply with all Certificate Authorities, so if you intend to migrate you should consider starting to test and migrating now.

5

I am little confused on this, is there anything that needs to be done on my server? Will this affect our customers sites? This issue will only really affect anyone using older versions of android?

6

Our overall expectation based on the community advice of Let’s Encrypt and Microsoft is that there will be no issue, but we do not provide a guarantee or assurance.

The caution in the notice above is due to the fact we cannot test the actual real world root/intermediate certificate expiry ourselves. All we have been able to do is set a test client and server date/time after the expiration and observe if the certificates still work. In our artificial testing the windows server didn’t always serve the correct chain until it was rebooted but assurances from Let’s Encrypt and Microsoft engineers (past and present) have been that in the real world it will all just work as expected.

I realise this ‘wait and see’ approach is unsatisfactory (it is for us too), but we are neither Let’s Encrypt nor Microsoft, our software just renews certificates and takes what it’s given by the certificate authority, we do not warranty the validity of certificates or certificate chains, we are not a certificate authority and we do not control how OS trust mechanisms work. We just help you manage and renew leaf/end-entity certificates and that’s all we can control.

Our advice for any users who are nervous about the expiry changeover is to test a change of CA to either BuyPass Go or ZeroSSL (both are supported by Certify The Web). Let’s Encrypt are the most commonly used ACME CA, but there’s no harm in testing another CA to see if it’s right for your organisation, this will vary depending on the type of service you provide, the certificate you need to get and the most commonly trusted roots in your clients trust stores.

7

Further to this, we expect to release an update to Certify next week which includes a built in clean up method for the old expiring R3.

To expedite the process users can perform one of the following methods of cleanup:

Move the old R3 to Untrusted

  • Open certlm.msc, expand Untrusted Certificates > Certificates, and Intermediate Certification Authorities > Certificates
  • Locate “R3 Issued by DST Root CA X3” (not the one issued by ISRG Root X1). Drag it from intermediate to untrusted.
  • optionally, reboot.

image

You may then need to reboot to serve the updated chain.

Disable Certificate Purposes
In windows, run certlm.msc (or open Manage computer certificates), browse to Intermediate Certification Authorities, right click the R3 issued by DST Root CA X3 (not ISRG Root X1), choose Properties. Check ‘Disable all purposes for this certificate’. A reboot may be required for the change to take effect for your served chain. This disables the use of the old R3 certificate without deleting it, so that way it can’t be inadvertently restored. In our testing however the R3 may still be preset in the serve chain, so deletion is also useful.

Alternatively, you can use any of the following methods to delete the old R3 certificate, however in some cases the certificate can be re-introduced automatically by the OS.

Registry Cleanup Method (does not require psexec):

Run cmd as Administrator and delete the registry entry for the R3 intermediate in the Local System CA store:

reg delete HKEY_USERS\S-1-5-18\Software\Microsoft\SystemCertificates\CA\Certificates\48504E974C0DAC5B5CD476C8202274B24C8C7172 /f

or

Download the current version of psexec (if you don’t have a copy) PsExec - Windows Sysinternals | Microsoft Docs

Manual Cleanup Method

  • From an administrator command prompt cd to the path where psexec is.

  • then use the following to launch cmd as Local System: psexec -i -s cmd.exe

  • then launch certmgr, delete R3 issued by DST Root CA X3 from Intermediate Certification Authorities (if present, not the R3 issued by ISRG Root X1)

or

Powershell method:

  • Launch powershell as local system: psexec.exe -i -s powershell.exe

  • Delete the expiring R3:

Get-ChildItem cert:CurrentUser\CA\48504E974C0DAC5B5CD476C8202274B24C8C7172 | Remove-Item

Check the served chain
Use SSL Server Test (Powered by Qualys SSL Labs) or openssl to check the served chain:

openssl s_client -servername isrg.projectbids.co.uk -connect isrg.projectbids.co.uk:443

If the correct chain is still not being served, edit an https binding in IIS (set to different cert then set back) or request a new certificate to force a binding update, alternatively you may need to reboot server to force the new chain to be served.

9

It’s being tested currently, expected release by the end of this week or start of next week [Update: version 5.5.4 has now been released]. After upgrading the old R3 certificate will be marked as not for use and moved to the disallowed certificate store (so even if it’s reinstalled it will still be disabled).

Note that you don’t need to upgrade at all, this will just be a helpful automation to achieve much the same as the above techniques. Our Certify app updates never specifically require a reboot.

You may however need to reboot to get the updated chain. In some cases it works without a reboot but in others it does need a reboot. To check the served chain you can use the Qualsys SSL checker or openssl as mentioned above.