Using Certify the web on more than one server

Hi

I’m new to the Certify web app and was wanting to know whether it was possible to use it to generate and renew certificates supporting the same domains on multiple servers (2-4 servers) (One certify the web app running on each server.)

or whether I should be looking at using a central certificate store (CCS) and running some custom scripts/ actions to add the generated/renewed certificates to the central certificate store and get IIS on each server to point to the CCS and setup/renew site bindings for certificate accordingly.

Could you confirm the best way in which the Certify web app could be used to solve this issue.

Thanks

Hi,
The app doesn’t currently have native CCS support but it is planned for the future. You could theoretically use a script of your own (a post-request powershell script) to deploy the certificate to your CCS store with the correct naming but I’m not sure if anyone is currently doing it that way.

The potentially larger issue is shared validation, if you are using http validation (where a challenge response file is created in the website) as this needs to be included in the response of all webservers unless you can direct all /.well-known/acme-challenge requests to a single server.

@smithr18 well it looks like you’re not alone! SNI on a pool of webservers

I’ve managed to get this working (at least initially) by creating a Network Share to contain the “.well-known” directory, and then configuring that as a virtual directory under each of the IIS sites, on each of the servers that need access to it. You also need to use the share when configuring “Certify the Web”, in the “Website Root Directory” field.

This config meant that whichever server Certify the Web was running on, it would put the verification file in a single, shared location, and then whichever server Let’s Encrypt hits over HTTP, that will also hit the same single, shared location. A big pain that we had to do this, but glad to get it working. I wasn’t able to use the DNS verification approach as each server needed it’s own unique TXT file configuring, with the same name but different value. Using that approach we would need to modify the TXT file every time each server tried to renew their certificate. That was obviously not a good option for an automated renewal process.

Hopefully the app will handle all of this natively in the future.

Thanks.

@j.strugnell Hello , I am trying to implement this for 2 load balanced servers that I have using HTTP verification. Could you please explain me the steps on how you got this working ?
Appreciate your help regarding this.

Hi, it’s been a while ;0) I have pretty much outlined the steps above?

Do you know how to give the CertifyTheWeb executable/process the access to write to the network share ?

I think it’s untested to change Certify’s service’s account.

If you’re using a collection of AD linked computers, the Certify service that runs as SYSTEM will access network objects using the machine account in AD. So if you connect from system1 to system2, \\system2\sharename will need to give permission to system1$, which is the machine account.

1 Like

Depending on what you’re trying to do you may also be able to impersonate another account, so if you’re scripting in powershell you can invoke your commands as another user by providing the credentials. Future versions of the app will allow user impersonation with credential storage.

For reference by others - the below may not be the ideal way to get this working, but it helped us:

  • the IIS virtual directory’s “physical path” needs to point to the .well-known directory inside the share, but inside CertifyTheWeb the Website Root needs to point to the root of the share (parent directory of .well-known)
  • The virtual directory’s Handler Mappings need to place the StaticFile mapping at the top (in the ordered list)
  • The virtual directory’s MIME Types need to include an entry for Extension . (a bare period) and MIME Type text/html
1 Like

Hi, I’ve been trying to get this to work too, for 2 days now, but can’t… any help will be appreciated.

I have two Azure VMs running IIS behind an Azure Load Balancer (standard). I can get certificates for wildcard domains via a DNS challenge, but can’t get http-01 challenge to work. Since the sites we’ll be hosting for customers will be a subdomain they create and route to our public Azure IP, we can’t use DNS challenges.

I’m using an Azure FileShare, mapped as a network drive to each VM. When I put the IIS site on there (using drive letter or UNC), I get this in the Certify log when I request a certificate:

Using website path E:\Sites\sitename
The website root path for sitename could not be determined. Request cannot continue.

I’ve turned off the ‘Enable HTTP Challenge Server’ setting.

I’ve tried the virtual directory approach, creating a virtual directory under the site called ‘.well-known’, with a physical path of E:.well-known. It gets past the point of the website root path, but then gives me this:

Failed to confirm URL is accessible : http://doman.name/.well-known/acme-challenge/S39AGPI-4eD-4M56iO8wWuT0Sp1BIbYDp3Nq-vfnvgQ
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Certify.Management.NetworkUtils.d__4.MoveNext() in C:\Work\GIT\certify_dev\certify\src\Certify.Shared\Management\NetworkUtils.cs:line 153

I’ve tried setting the Site root directory to e:\ (as suggested in previous posts) and get this again:

[INF] Using website path e:
[ERR] The website root path for sitename could not be determined. Request cannot continue.

i.e. the same as above.

As mentioned, E: is an Azure FileShare mapped to this drive, identically on both VMs. I have also tried using a local drive (C:), using Azure FileSync, I can see the well-known folder/files being created, but they don’t sync fast enough between the servers for the authentication to work (is there a setting/way to delay the http challenge for 90 secs, like for the DNS challenge?).

If I run the certify on both servers, one will succeed (i.e. the one the load balancer is currently failing) and the other gives the issues above - i.e. I’ve no reason to believe there’s an issue with DNS or site config, etc.

I’ll consider any other solution - Using an Azure App Gateway, using IIS Centralized Certificates, or anything else that may work. Ideally I’d like to consider using Certify (we’ve used it for a while and really value the automated renewal), but at the moment I’m open to any solution that will work.

At the moment, this is a new hosting build, i.e. I can change anything on the Azure config, but I really need to get this live before the end of the week.

Any help appreciated, I’m tearing my hair out with this one and really stuck for what I can try next!

Hi Richard,

In the first instance I’d advise anyone trying to do cert validation with load balancing to use DNS validation instead of http validation (either a supported DNS provider, your own scripting or using acme-dns). That way the servers can validate independently and manage their own certificates.

If you use http validation you need to work out how to ensure that requests to `/.well-known./acme-challenge/an_example’ resolve to the latest http challenge file generated by the app (from whichever server started the validation request). If you proceed with this technique you need to create a test file and make sure it resolves first. Your second problem is that Certify runs as Local System by default and so that process needs to be able to write to the share, which is sometimes not possible. Http challenges written out to a network share are currently not configurable, so for instance you can’t use impersonation etc.

A third option is to configure your load balancer to direct all /.well-known/acme-challenge requests (which are http, not https) to one specific server and only perform cert requests from that server, then use either the CCS deployment task or your own scripting to copy the cert to somewhere the other server can see, then configure CCS on the other server so it always looks for it’s certs there. All servers can optionally use CCS if that makes the config more consistent for administration.

Thank you for your help. We’ve now got a fix, but firstly re the DNS validation and path routing via the Load Balancer:

  • We’re hosting multiple instances of our web app for customers on these servers. So, though the domain names that belong to us can use DNS validation and that works fine, where a customer creates a subdomain and routes it to our IP, we don’t have access to the DNS and don’t want any more hassle (or to have to explain things unnecessarily) to customers.

  • Path-based routing. We’re using the Azure Standard Load Balancer for this which doesn’t support path-based routing. We could upgrade to the Azure App Gateway, but thats about 6x the cost (for the same SLA), so we want to get it working without Path-base routing if we can.

The fix though - our issue turned out to be all about file permissions. I changed the Certify system to run as the local admin on each server and it worked fine. But, since I didn’t want to do this permanently I needed to give LocalSystem access to the Azure FileShare, this was done by:

  • Using psexec to run a cmd window as LocalSystem (psexec -sid cmd.exe)
  • using this command (replacing the {…} values with the specific Azure info for our file storage resource): cmdkey /add:{Azure filestore name}.file.core.windows.net /user:Azure{fileshare name} /pass:{Azure Storage key}

I then switched the logon as account for the Certify Windows service back to LocalSystem and it all works perfectly.

This worked for the Virtual Directory approach (i.e. setting a .well-known virtual directory to point to a fileshare folder), but I’ve decided to run the whole IIS site from the fileshare and it works for that too.

I think thats the full set of info of what got it to work, I hope it helps others, and thanks for the help on this forum and also via support emails.

1 Like