We get about a 20 second hang on our site with the Certify SSL certificate renews. I don’t think it restarted the application pool, but my monitoring system acted like it did. Is there anyway to adjust this to do it autmotically at night? This was at a peek time.
HTTPS binding updates in IIS are transactional and happen pretty much instantly if you only have a few sites, unless you have many sites in which case it can take a few seconds for IIS to save the updated applicationhost.config (it’s one file). We don’t update web.config so your app won’t be restarting because of Certify.
Are you using IIS or are you using another server like Kestrel and if so are you custom loading your certificates?
If something is causing your application to restart you should review what’s happening in application start and avoid massive startup workloads - 20 seconds is a long time.
I realise I sound a bit dismissive - clearly something is happening, so it would be good to know more about this system/application. Do you do any custom tasks on renewal? I assume you are using http validation and that the log shows the http challenge service is starting ok (the temporary http challenge response service that sits in front of IIS - if this isn’t working then we fallback to filesystem based challenge response via IIS).
If the web.config doesn’t get touched that’s great. However I do have 13 sites that use that certificate and the CPU did go to 100% during that short period. The logs in event viewer said the new certificate was added and the time stamp on the SSL cert is exactly during the hang. If anything the site didn’t restart otherwise it takes around a minute on my high traffic site but it was definitely a hang.
Thanks, I’ll investigate this. Weirdly it’s not an issue that’s been reported by anyone else over the last 5 years. We have over 100k daily users so I’m surprised it hasn’t come up before.
I do agree there is clearly something happening here that we don’t expect/understand.
Are your certificates one single certificate applied to one site (with multiple domains or a wildcard) or a set of different certificates applied to one site across different bindings?
You could send details through to support at certifytheweb.com if you don’t want to share here.
It’s one certificate wildcard. I can follow up when the renewal date comes up and watch when it happens next time to maybe verify it happens again before you waste any time.
Note that you could click ‘Request Certificate’ during a maintenance window (e.g. every month) to control the renewal more (if you do it, then the app will never reach it’s auto renewal), and you can optionally set the auto renewal interval under Settings to (up to) 60 days or even disable auto renewal for that cert (but that risks expiry if you forget to manually renew).
I’m still interested in seeing if I can reproduce it so I can understand if this affects all users etc, so I’ll still have a look.
If a domain was renewed 6/1/2021 11:17:23 AM and expires 8/30/2021 should I expect it to renew on 8/30/2021 at 11:17:23 AM?
No, the renewals are checked hourly (starting from when the background service starts, so if the service started at 08:34 then the next check would be 09:34). Renewal times are not intended to be predictable as they are intended to be automatic and allow for failures. If a failure occurs (e.g. your website or the Let’s Encrypt service is unavailable) the renewal will be attempted again later until eventually it will only check every 2 days (e.g. if your website has been moved it a different server then http validation will continually fail).
In addition our default renewal interval is 30 days, this means 30 days after your certificate was last renewed we will try again, we do not wait for expiry to approach as it’s extremely common for people to modify webserver configuration after getting it working, leading to http validation failures and therefore renewal failures. The app then notifies them of the failure so they can resolve whatever they’ve broken before they actually do need a new cert. Certs don’t get deployed/applied until they are successfully renewed.
In the case of any ACME certificate tool on any operating system, specific renewal date/time cannot be guaranteed because you don’t control all the moving parts.
If you really want to try to control renewal time you could schedule a task (e.g. monthly) with certify renew --force-renew-all (see https://docs.certifytheweb.com/docs/commandline) which will attempt to renew all certificates regardless of when they were last renewed.
Out of interest, do you have any rules defined in IIS that restart the app pool if CPU gets to 100% or memory gets low? I wondered if there was something triggering an automated app pool restart. The reason should be logged in Event Viewer under System > (Filter by Event Source “WAS”)
No there isnt and it would of said it in the event viewer.
As a follow up to this, I am able to reproduce this problem on my development machine (windows 10) by setting a breakpoint in Application_Start then simply adding or removing an IIS binding in IIS manager (which is effectively the same as Certify is currently doing via the IIS admin API). I can confirm it happens when you run the cert renewal in Certify as well. The application restart is not logged by IIS/asp.net in Event Viewer and the only clue is a System > HttpEvent warning about the binding being added or removed.
As far as I was aware adding/removing IIS bindings should have no effect on the running IIS app, so this is news to me. I’m wondering if there is some other factors which affects this.
Thanks very much for raising this issue, I’ll investigate if we can change either the default binding update process or provide an alternative strategy via the settings UI. In the meantime I would suggest settings your renewal interval to the max 60 days to avoid sooner renewals and consider running the renewal manually during a regular maintenance window (or scheduled task).
Great, glad I was of some help.
1 Like