We use pfSense as a firewall on our servers facing the web. When we try to renew our Certificates with Certify The Web and Let’s Encrypt, it fails saying it couldn’t verify us. We host our DNS on Cloudfare and tried verifying the DNS as well as doing http verifications. Everything failed. We hired a guy who has hundreds of Certify The Web systems running. He couldn’t figure it out.
What we did find is that if we bypass pfSense it works so we know it’s something related to pfSense but what? Port 80 is open to those IPs. I need help.
I tried to find a list of Certify the Web IPs to help hunt down the problem but was told that there is no list and they don’t their list of IPs available.
Obviously if it would verify with Cloudflare that would be the best. It passes the test but then fails on the renewal.
Hi Jerry,
I can’t comment on how pfSense works but usually failures are straightforward connectivity problems and the log will tell you more or less where the problem is, so a log would really help diagnose.
HTTP validation requires that you allow incoming TCP port 80 requests - that sounds obvious but can often be surprisingly tricky especially when third party security products try to help. Certify will attempt to starts its own HTTP listener via http.sys that sits in front of IIS etc and for most users that will work to answer http challenges. Port 80 needs to be open on the windows firewall, any AV/Security exclusions that are applicable to your security products need to be applied to allow that listener (certify.exe) to listen.
DNS Validation is available if HTTP validation can’t work (because their is no HTTP route to the server). DNS validation works by creating an _acme-challenge TXT record in your DNS zone for each domain/subdomain so any conflicting records with that name need to be removed first.
If you have specific error messages we can help diagnose further.
What would a conflicting record in DNS look like? And if it conflicted with one domain would it just prevent the renewal on that certificate or would it abort all the renewals?
On the domain listed in the error, I don’t see any abnormal entries in DNS.
I do see a list of about 2 dozen DNS entries listed under one of my other domain names however. There are a couple dozen “_acme-challenge.mail.insertdomainhere” entries in it’s DNS. Do those all need to be deleted?
We found a work-around. We had tried doing DNS verification but it kept failing. It turns out that Certify the web had created a DNS entry but then just left it there. So when it came back to renew it was creating a new entry but reading the old one? We deleted all their DNS entries (37 of them) and it passed. So we won’t need to do the http verification which means we don’t need it to find it’s way past pfSense.
While this doesn’t solve the pfSense question, it does solve our problem so I’m going to move on.
Thanks for raising that issue, our cloudflare DNS integration is pretty mature so I’m surprised it left some records. Glad you got it sorted.
Well it appears that the solution didn’t last for long. I’m back to getting errors saying that validation failed. I’ve tried totally disabling the firewall to rule out anything from pfSense and it still fails so it’s clearly a Certify The Web and/or DNS issue.
It still passes when I run the test, but fails when it tries to actually do the update. I’ve deleted all the DNS entries that Certifies leaves laying around in the DNS and re-run the renewal. It creates two new files in the DNS for the primary domain, nothing in the other domains (should there be???). I’m using a global account credential. Do I need one for each domain instead?
Response from Certificate Authority: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.mail.domain2.com - check that a DNS record exists for this domain [BadRequest :: urn:ietf:params:acme:error:dns]
2024-10-16 08:13:22.003 -04:00 [INF] DNS: Deleting TXT Record '_acme-challenge.mail.domain2.com' :'dPmeJUmw4KnCntrOinlOMBQB0Ef6L1EGDdiC9KSYPis', [mail.domain2.com] in ZoneId '0681f443c9dc2cdca5621c3b425b51d3' using API provider 'Cloudflare DNS API'
2024-10-16 08:13:22.257 -04:00 [INF] Identifier already has current authorization, skipping verification: images.domain1.com
2024-10-16 08:13:22.257 -04:00 [INF] Identifier already has current authorization, skipping verification: www.images.domain1.com
2024-10-16 08:13:22.257 -04:00 [ERR] Validation of the required challenges did not complete successfully. Validation failed: mail.domain2.com [dns]
Response from Certificate Authority: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.mail.domain2.com - check that a DNS record exists for this domain [BadRequest :: urn:ietf:params:acme:error:dns]
Thanks, the problem is that different DNS zones need an authorization configuration each.
Your cloudflare credential itself can either be global (for all zones on your account) or you can configure it to be restricted to specific zones/domains, that’s up to you. I assume your credential token is global. Cloudflare DNS | Certify The Web Docs
However, when you setup DNS authorization you are asked to select a ZoneID to target and that’s basically one per domain. So to target multiple zones with one certificate you need to add one authorization config per cloudflare DNS zone/domain (scroll down on the Authorization tab and click Add Configuration) and each will have a different ZoneID selected.
You will need to set the Domain Match field on your configurations (e.g. *.domain2.com) so that the app knows which config to match to which domains. You can then review which authorization config will match which domains on the Preview tab. If you then click Test again TXT records will be created in each DNS zone separately. DNS Validation (dns-01) | Certify The Web Docs
The instructions on the first link appear to be for the old Cloudflare GUI so I had to look for things a bit but I found them.
You lost me however on the DNS authorization asking for a ZoneID. Is this still in Cloudflare (where?) or in the Certify Manager?(where?) I don’t see where I’m supposed to enter these authorizations.
When you click on your managed certificate in Certify (from the left side list of managed certificates) you will see an Authorisation tab.
That’s where you can configure how domain validation will be performed.
Ah. I wasn’t seeing the tabs down the right hand side. Thank you.
I was still failing, and pulling my hair out. And then I realized that instead of *.domain.com I had just entered domain.com on the Domain Match line so everything was failing. I changed it to include the * and SUCCESS!
Still not sure why I’d had success back in August but then it failed in October. But… it’s done so I’m good for another couple of months at least.
THANK YOU again.
New problem. Now that the certificates all updated we’ve lost access to the site. According to cloudflare the certificates from here are not compatible any longer. Indeed my site is nothing but 525 errors now.
While I may have solve the update problem, I’ve gone from the frying pan into the fire.