Invalid response, but files are there and accessible

PaulReid · June 25, 2020, 6:35pm

I seem to be running into an unexpected issue. I keep getting a failure with an invalid response error, yet the test process passes, files are being created, and they’re accessible from the outside. At first I had issues with the common 403 error, but I resolved that (extensionless files issue with IIS 10, which I solved by modifying the web.config file the software uses - the one it comes with doesn’t work for IIS 10). This is a Windows Server 2016 machine with Exchange 2016 installed. Here is the log file I get after trying to request a certificate:

2020-06-25 11:14:15.639 -07:00 [INF] ---- Beginning Request [Default Web Site] ----
2020-06-25 11:14:15.639 -07:00 [INF] Certify/5.0.12.0 (Windows; Microsoft Windows NT 10.0.14393.0) 
2020-06-25 11:14:15.646 -07:00 [INF] Beginning Certificate Request Process: Default Web Site using ACME Provider:Certes
2020-06-25 11:14:15.646 -07:00 [INF] Requested domains to include on certificate: exchange.pcmlawco.com;autodiscover.pcmlawco.com
2020-06-25 11:14:15.646 -07:00 [INF] Beginning certificate order for requested domains
2020-06-25 11:14:15.646 -07:00 [INF] BeginCertificateOrder: creating/retrieving order. Retries remaining:2 
2020-06-25 11:14:15.922 -07:00 [ERR] Certes.AcmeRequestException: Fail to load resource from 'https://acme-v02.api.letsencrypt.org/acme/new-order'.
urn:ietf:params:acme:error:badNonce: JWS has an invalid anti-replay nonce: "0001txDy-TgNyJdTgbmCAJR6NptcEmtfTlpUCgv49fR39aA"
   at Certes.Acme.IAcmeHttpClientExtensions.<Post>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Certes.AcmeContext.<NewOrder>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Certify.Providers.ACME.Certes.CertesACMEProvider.<BeginCertificateOrder>d__30.MoveNext() in C:\Work\GIT\certify_5.0.x\certify\src\Certify.Providers\ACME\Certes\CertesACMEProvider.cs:line 566
2020-06-25 11:14:15.922 -07:00 [ERR] BeginCertificateOrder: error creating order. Retries remaining:1 :: JWS has an invalid anti-replay nonce: "0001txDy-TgNyJdTgbmCAJR6NptcEmtfTlpUCgv49fR39aA" 
2020-06-25 11:14:16.934 -07:00 [INF] BeginCertificateOrder: creating/retrieving order. Retries remaining:0 
2020-06-25 11:14:17.069 -07:00 [INF] Created ACME Order: https://acme-v02.api.letsencrypt.org/acme/order/89696326/3923462715
2020-06-25 11:14:17.172 -07:00 [INF] Fetching Authorizations.
2020-06-25 11:14:17.704 -07:00 [INF] Got http-01 challenge https://acme-v02.api.letsencrypt.org/acme/chall-v3/5463096406/Xt90bg
2020-06-25 11:14:17.916 -07:00 [INF] Got dns-01 challenge https://acme-v02.api.letsencrypt.org/acme/chall-v3/5463096406/nUMwlw
2020-06-25 11:14:18.334 -07:00 [INF] Got http-01 challenge https://acme-v02.api.letsencrypt.org/acme/chall-v3/5474039454/0H9gHw
2020-06-25 11:14:18.534 -07:00 [INF] Got dns-01 challenge https://acme-v02.api.letsencrypt.org/acme/chall-v3/5474039454/5zm48A
2020-06-25 11:14:19.556 -07:00 [INF] Http Challenge Server process available.
2020-06-25 11:14:19.557 -07:00 [INF] Attempting Domain Validation: exchange.pcmlawco.com
2020-06-25 11:14:19.557 -07:00 [INF] Registering and Validating exchange.pcmlawco.com 
2020-06-25 11:14:19.557 -07:00 [INF] Performing automated challenge responses (exchange.pcmlawco.com)
2020-06-25 11:14:19.557 -07:00 [INF] Preparing challenge response for the issuing Certificate Authority to check at: http://exchange.pcmlawco.com/.well-known/acme-challenge/PBk16H9mYyDJhJ5FKuHPpQjl-1tU9Nc39oRtCa6bt8U with content PBk16H9mYyDJhJ5FKuHPpQjl-1tU9Nc39oRtCa6bt8U.gUciPCYus201IdA0yQyKdd99i22tsdGyDDB3k9cMxTw
2020-06-25 11:14:19.557 -07:00 [INF] If the challenge response file is not accessible at this exact URL the validation will fail and a certificate will not be issued.
2020-06-25 11:14:19.577 -07:00 [INF] Using website path C:\inetpub\wwwroot
2020-06-25 11:14:19.578 -07:00 [INF] Checking URL is accessible: http://exchange.pcmlawco.com/.well-known/acme-challenge/PBk16H9mYyDJhJ5FKuHPpQjl-1tU9Nc39oRtCa6bt8U [proxyAPI: True, timeout: 5000ms]
2020-06-25 11:14:20.498 -07:00 [INF] (proxy api) URL is not accessible. Result: [403] Resource not accessible, Timeout or Redirected
2020-06-25 11:14:20.498 -07:00 [INF] Checking URL is accessible: http://exchange.pcmlawco.com/.well-known/acme-challenge/PBk16H9mYyDJhJ5FKuHPpQjl-1tU9Nc39oRtCa6bt8U [proxyAPI: False, timeout: 5000ms]
2020-06-25 11:14:20.628 -07:00 [INF] (local check) URL is accessible. Check passed. HTTP OK
2020-06-25 11:14:20.628 -07:00 [INF] Requesting Validation: exchange.pcmlawco.com
2020-06-25 11:14:20.629 -07:00 [INF] Http Challenge Server process available.
2020-06-25 11:14:20.629 -07:00 [INF] Attempting Domain Validation: autodiscover.pcmlawco.com
2020-06-25 11:14:20.630 -07:00 [INF] Registering and Validating autodiscover.pcmlawco.com 
2020-06-25 11:14:20.630 -07:00 [INF] Performing automated challenge responses (autodiscover.pcmlawco.com)
2020-06-25 11:14:20.630 -07:00 [INF] Preparing challenge response for the issuing Certificate Authority to check at: http://autodiscover.pcmlawco.com/.well-known/acme-challenge/U8ZtvcSJHZfB1uax7zbRyCEhk8nZXFbkcDMIaxjU4fE with content U8ZtvcSJHZfB1uax7zbRyCEhk8nZXFbkcDMIaxjU4fE.gUciPCYus201IdA0yQyKdd99i22tsdGyDDB3k9cMxTw
2020-06-25 11:14:20.630 -07:00 [INF] If the challenge response file is not accessible at this exact URL the validation will fail and a certificate will not be issued.
2020-06-25 11:14:20.648 -07:00 [INF] Using website path C:\inetpub\wwwroot
2020-06-25 11:14:20.650 -07:00 [INF] Checking URL is accessible: http://autodiscover.pcmlawco.com/.well-known/acme-challenge/U8ZtvcSJHZfB1uax7zbRyCEhk8nZXFbkcDMIaxjU4fE [proxyAPI: True, timeout: 5000ms]
2020-06-25 11:14:21.184 -07:00 [INF] (proxy api) URL is not accessible. Result: [403] Resource not accessible, Timeout or Redirected
2020-06-25 11:14:21.184 -07:00 [INF] Checking URL is accessible: http://autodiscover.pcmlawco.com/.well-known/acme-challenge/U8ZtvcSJHZfB1uax7zbRyCEhk8nZXFbkcDMIaxjU4fE [proxyAPI: False, timeout: 5000ms]
2020-06-25 11:14:21.186 -07:00 [INF] (local check) URL is accessible. Check passed. HTTP OK
2020-06-25 11:14:21.186 -07:00 [INF] Requesting Validation: autodiscover.pcmlawco.com
2020-06-25 11:14:21.193 -07:00 [INF] Attempting Challenge Response Validation for Domain: exchange.pcmlawco.com
2020-06-25 11:14:21.193 -07:00 [INF] Registering and Validating exchange.pcmlawco.com 
2020-06-25 11:14:21.193 -07:00 [INF] Checking automated challenge response for Domain: exchange.pcmlawco.com
2020-06-25 11:14:21.356 -07:00 [WRN] Challenge response validation still pending. Re-checking [10]..
2020-06-25 11:14:22.962 -07:00 [INF] Invalid response from http://exchange.pcmlawco.com/.well-known/acme-challenge/PBk16H9mYyDJhJ5FKuHPpQjl-1tU9Nc39oRtCa6bt8U [184.68.12.218]: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\"><html><head><meta http-equiv=\"Expires\" content=\"-1\"><meta http-equ"
2020-06-25 11:14:24.066 -07:00 [INF] Validation of the required challenges did not complete successfully. Invalid response from http://exchange.pcmlawco.com/.well-known/acme-challenge/PBk16H9mYyDJhJ5FKuHPpQjl-1tU9Nc39oRtCa6bt8U [184.68.12.218]: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\"><html><head><meta http-equiv=\"Expires\" content=\"-1\"><meta http-equ"
2020-06-25 11:14:24.067 -07:00 [INF] Validation of the required challenges did not complete successfully. Invalid response from http://exchange.pcmlawco.com/.well-known/acme-challenge/PBk16H9mYyDJhJ5FKuHPpQjl-1tU9Nc39oRtCa6bt8U [184.68.12.218]: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\"><html><head><meta http-equiv=\"Expires\" content=\"-1\"><meta http-equ"
2020-06-25 11:14:24.067 -07:00 [INF] Validation of the required challenges did not complete successfully. Invalid response from http://exchange.pcmlawco.com/.well-known/acme-challenge/PBk16H9mYyDJhJ5FKuHPpQjl-1tU9Nc39oRtCa6bt8U [184.68.12.218]: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\"><html><head><meta http-equiv=\"Expires\" content=\"-1\"><meta http-equ"

There’s also a challenge request file left behind (it seems to delete one, but not both for my request), which is here:

http://exchange.pcmlawco.com/.well-known/acme-challenge/U8ZtvcSJHZfB1uax7zbRyCEhk8nZXFbkcDMIaxjU4fE

Any ideas? I’m completely stumped.

Thanks!

PaulReid · June 25, 2020, 7:06pm

I fixed it! I had GeoIP enabled on the firewall for port 443 (trying to protect the Exchange Server from password attacks from other countries), which prevented the incoming validation check from connecting. Yay! It all works now!

I did notice it didn’t automatically bind the new certificate to the default website, like I figured it would. I had to go bind it myself - but once I did, it worked. I wonder what will happen when it renews? Will I have to go rebind it again?

jljtgr · June 25, 2020, 9:49pm

If you did not have a domain name listed in your http/80 binding, Certify may have been confused by the default name-less one. Now that you have a https/443 binding with a domain name, it should be able to figure it out more easily.

PaulReid · June 25, 2020, 11:09pm

Ah, that could be. It didn’t have anything in there previously. There were 2 port 443 bindings, but no domain name entered. Hopefully it’ll be happy come renewal time. I made a journal entry in Outlook to check it on the day it’s supposed to renew, just to make sure it goes smoothly.

Thanks for the input.

jljtgr · June 26, 2020, 12:14am

I forgot that you can have https/443 without a domain name. This is for browsers that don’t support Server Name Indication(SNI). Just make sure that the binding(s) that are using your certificate have the domain name listed. Certify should look at each binding and match with the primary/alternate names in the certificate. If the binding has a blank domain name or a domain not listed in the certificate, it probably won’t update that binding.

webprofusion · June 26, 2020, 1:24am

Additionally, if you look at the Preview tab for your managed certificate scroll down to the bottom and you will see a complete summary of the planned bindings updates for the next renewal. If any are missing then you need to fix that before your cert expires.

By default, auto-binding works by matching your certificate domains to a matching IIS hostname binding as it can’t make any assumptions about what your IIS website is based on IP only unless you have already manually setup a binding with the previous version of the cert (then it assumes you want to include that in renewals).

You can change the Deployment mode to Single Site in order to target a specific IIS site, then you can also configure the auto binding behavior if required.

Important Note: Windows can only bind one ssl cert to one network adaptor IP address. If you manually create an IP specific binding this takes priority over any binding that might then share that IP address (e.g. if you only actually have one IP address on the machine), which leads to invalid certs being served for other domains. This is why we use SNI and unassigned IP bindings by default, so be careful when setting up non-SNI bindings (especially IP specific ones) and avoid them unless you know you specifically need them. Old fashioned (pre-SNI) SSL instructions often talk about assigning IPs, don’t do it.

webprofusion · June 26, 2020, 1:27am

p.s. you can ignore challenge files being written to disk and web.configs etc. This is only relevant when the built in http challenge server doesn’t work (or you block it at the firewall!) and the process falls back to trying to validate via IIS.

PaulReid · June 26, 2020, 7:16pm

That’s all awesome information! Thank you so much. I did check the preview, and it looks like it’s going to target the right spots, now that it can see what is needed after I got the certificate set up in IIS. It looks like it’ll be fine. Thanks again!