Unable to validate - 400 - Connection reset by peer

Question
1

I’m running a site on Windows Server 2019, IIS 10, and have been successfully using a different LetsEncrypt client until recently it wouldn’t renew anymore. I spent all day troublshooting without luck, including downloading and attempting to use CertifyTheWeb (which I find much more pleasant than the command line alternative I used before). I get the same error using both clients.

Anyway, I’m not sure what information I need to provide here other than the the error log. Oh, although I didn’t have to do any of this before, I have verified that “.well-known/acme-challenge/configcheck” was created on my site and after manually creating a web.config it’s also accessible on the public web. Running Let's Debug also verifies the error (“Connection reset by peer”). Before going crazy, can anyone help? Here’s the log file:

2022-03-28 20:37:16.593 +02:00 [INF] All Tests Completed OK
2022-03-28 20:37:42.543 +02:00 [INF] [Preview Mode] Completed certificate request and automated bindings update (IIS)
2022-03-28 20:38:45.983 +02:00 [INF] ---- Beginning Request [linkdev.langlo.no] ----
2022-03-28 20:38:45.995 +02:00 [INF] Certify/5.6.7.0 (Windows; Microsoft Windows NT 10.0.17763.0) 
2022-03-28 20:38:46.002 +02:00 [INF] Beginning Certificate Request Process: linkdev.langlo.no using ACME Provider:Certes
2022-03-28 20:38:46.003 +02:00 [INF] Requested identifiers to include on certificate: linkdev.langlo.no
2022-03-28 20:38:46.004 +02:00 [INF] Beginning certificate order for requested domains
2022-03-28 20:38:46.026 +02:00 [INF] BeginCertificateOrder: creating/retrieving order. Retries remaining:2 
2022-03-28 20:38:47.133 +02:00 [INF] Created ACME Order: https://acme-v02.api.letsencrypt.org/acme/order/472550320/75352548880
2022-03-28 20:38:47.438 +02:00 [INF] Fetching Authorizations.
2022-03-28 20:38:48.361 +02:00 [INF] Got http-01 challenge https://acme-v02.api.letsencrypt.org/acme/chall-v3/92399747070/ymmrsg
2022-03-28 20:38:48.660 +02:00 [INF] Got dns-01 challenge https://acme-v02.api.letsencrypt.org/acme/chall-v3/92399747070/XYTNFQ
2022-03-28 20:38:49.701 +02:00 [INF] Http Challenge Server process available.
2022-03-28 20:38:49.702 +02:00 [INF] Attempting Domain Validation: linkdev.langlo.no
2022-03-28 20:38:49.702 +02:00 [INF] Registering and Validating linkdev.langlo.no 
2022-03-28 20:38:49.702 +02:00 [INF] Preparing automated challenge responses (linkdev.langlo.no)
2022-03-28 20:38:49.705 +02:00 [INF] Preparing challenge response for the issuing Certificate Authority to check at: http://linkdev.langlo.no/.well-known/acme-challenge/ZN-QdPg78DCf9J7wKDamm-aeQJNhU2HrfPXe9KslFK0 with content ZN-QdPg78DCf9J7wKDamm-aeQJNhU2HrfPXe9KslFK0.6hDWJlHlkX8zKc5iUwM-ATSDghhtCo8D86FFHAk2Qes
2022-03-28 20:38:49.705 +02:00 [INF] If the challenge response file is not accessible at this exact URL the validation will fail and a certificate will not be issued.
2022-03-28 20:38:49.720 +02:00 [INF] Using website path D:\WebSites\linkdev.langlo.no
2022-03-28 20:38:49.721 +02:00 [INF] Checking URL is accessible: http://linkdev.langlo.no/.well-known/acme-challenge/ZN-QdPg78DCf9J7wKDamm-aeQJNhU2HrfPXe9KslFK0 [proxyAPI: True, timeout: 5000ms]
2022-03-28 20:38:49.823 +02:00 [WRN] Problem checking URL is accessible : http://linkdev.langlo.no/.well-known/acme-challenge/ZN-QdPg78DCf9J7wKDamm-aeQJNhU2HrfPXe9KslFK0 The remote server returned an error: (404) Not Found.
2022-03-28 20:38:49.823 +02:00 [INF] Checking URL is accessible: http://linkdev.langlo.no/.well-known/acme-challenge/ZN-QdPg78DCf9J7wKDamm-aeQJNhU2HrfPXe9KslFK0 [proxyAPI: False, timeout: 5000ms]
2022-03-28 20:38:49.948 +02:00 [INF] (local check) URL is accessible. Check passed. HTTP OK
2022-03-28 20:38:49.948 +02:00 [INF] Requesting Validation: linkdev.langlo.no
2022-03-28 20:38:49.971 +02:00 [INF] Attempting Challenge Response Validation for Domain: linkdev.langlo.no
2022-03-28 20:38:49.971 +02:00 [INF] Registering and Validating linkdev.langlo.no 
2022-03-28 20:38:49.971 +02:00 [INF] Checking automated challenge response for Domain: linkdev.langlo.no
2022-03-28 20:38:51.180 +02:00 [INF] Domain validation failed: linkdev.langlo.no 
Fetching http://linkdev.langlo.no/.well-known/acme-challenge/ZN-QdPg78DCf9J7wKDamm-aeQJNhU2HrfPXe9KslFK0: Connection reset by peer BadRequest urn:ietf:params:acme:error:connection
2022-03-28 20:38:51.888 +02:00 [INF] Validation of the required challenges did not complete successfully. Domain validation failed: linkdev.langlo.no 
Fetching http://linkdev.langlo.no/.well-known/acme-challenge/ZN-QdPg78DCf9J7wKDamm-aeQJNhU2HrfPXe9KslFK0: Connection reset by peer BadRequest urn:ietf:params:acme:error:connection
2022-03-28 20:38:51.888 +02:00 [INF] Validation of the required challenges did not complete successfully. Domain validation failed: linkdev.langlo.no 
Fetching http://linkdev.langlo.no/.well-known/acme-challenge/ZN-QdPg78DCf9J7wKDamm-aeQJNhU2HrfPXe9KslFK0: Connection reset by peer BadRequest urn:ietf:params:acme:error:connection
2022-03-28 20:38:51.889 +02:00 [INF] Validation of the required challenges did not complete successfully. Domain validation failed: linkdev.langlo.no 
Fetching http://linkdev.langlo.no/.well-known/acme-challenge/ZN-QdPg78DCf9J7wKDamm-aeQJNhU2HrfPXe9KslFK0: Connection reset by peer BadRequest urn:ietf:params:acme:error:connection
2022-03-28 21:08:54.595 +02:00 [INF] [Preview Mode] Completed certificate request and automated bindings update (IIS)
2022-03-28 21:09:14.522 +02:00 [INF] All Tests Completed OK

Many Thanks,
Tor

2

Hi Tor, the root cause of the error you are seeing is this:

Your webserver is terminating the connection on purpose - so you have a firewall (or malware protection etc.) that’s filtering specific IP addresses, IP ranges or geographic ranges.

Your website works fine for me (I can connect to TCP port 80, as would be required for http validation to work) but if I try Let’s Debug it fails: Let's Debug

If you do not control this filtering/firewall config (and therefore can’t fix it yourself) you could use DNS validation instead of http validation (see the Authorization tab in the app to configure that and see also DNS Validation (dns-01) | Certify The Web Docs). You could also just try a different Certificate Authority (who’s validation attempts will come from different IP range) Certificate Authorities | Certify The Web Docs

Note that if you do subscribe to some sort of IP address-range spam filtering software, such tools are occasionally vulnerable to denial of service by simply blacklisting IP addresses for services you actually want to use (such as Let’s Encrypt).

3

Hi, and thanks for the reply! I’ve done further testing today and so far it seems like there’s “someone” blocking access to the “.well-known/acme-challenge” path. After disconnecting my VPN into the server I was able to reproduce the connection reset problem from my machine as well. The interesting thing was if I renamed the folder I was able to connect and see the file’s contents. And if I created a similar folder path (".also-known/acme-challenge") that also worked.

I’ve reached out to my host server’s support guys and will hopefully have a response from them soon.

4

So, to block connections based on /.well-known/ etc that would need to be something which can inspect your http traffic (i.e. a content aware firewall), so that would imply your host has some pretty fancy stuff going on - or their is a proxied connection to your server somewhere in the pipeline.

5

Actually, thinking about it, it’s more likely that one of the apps which registered a listener for http://+/.well-known/acme-challenge has simply left a broken listener registered. If you have tried any other acme apps other than win-acme or Certify then make sure to uninstall those (there are only a few other possible options), also make sure there are no win-acme or Certify.exe processes running in task manager.

If that doesn’t help, try a reboot.

6

I’m very curious to find out what your response on this. We are having the exact same issue from a configuration that worked right up until about the 8th of March. This is a system that has been running on certbot for years. I have gone through all the same steps and exact same behavior. I was sure that I had diagnosed this to our Palo Alto firewall, but it could be at the ISP level. When coming in over VPN it works just fine, updating and overriding hostname in hosts file also shows it working just fine. But coming in over the internet is a problem. What could have happened since March 8th?

7

On certbot or Certify The Web, or something else? First check your website can answer standard http requests (tcp port 80, not just https port 443). Hosts file tricks just prove your site works locally, test your site from a mobile phone data connection instead of over wifi, ensure that your firewall forwards port 80 requests to the server.

8

Yes. For me all that is true. I can access over the web externally on port 80 I can access all the files unless it has /.well-known/acme-challenge/ in the path. I have multiple web servers for different environments:dev,stage,test,upgrade,prod that all developed the same problem on the same date. No changes had occurred from when it worked and then didn’t. I even went and created a whole new web server from scratch to test and it had the problem from the inception. Tonight we are planning to get the Palo Alto out of the way and put the web server directly on the edge. That should tell us if it’s ISP or Palo Alto that’s the problem. I was just mostly curious to see what torminator found as a resolution, if he found a resolution.

9

Hi, sorry for not following up earlier. In our case the response I got from our server partner was that this was a configuration change done my Palo Alto where they had “changed the application from web-browsing to acme” (translated from Norwegian). Sorry for the inaccurate description, but that’s what I got. I could push harder for a better explanation if needed. I also got this screenshot:

image
image817×25 2.48 KB

10

No. Thank you for the response. I won’t be able to make the change until later tonight but this is for sure my issue. I have looked at the rule and will be adding the “acme-protocol” to the application list. I haven’t dug into the docs on Panorama, but I’m sure this is a content database update. I would expect there to be more hits on the web when trying to find this issue. I’m sure we aren’t the only ones experiencing this so super happy to have it on the web for other poor souls.

Thank you very much!

11

Confirmed. That was the issue. Add “acme-protocol” to the applications in your web server rule on Palo Alto.

1 Like