Cert deployment into Azure Key Vault

stevo · November 8, 2021, 2:43pm

I’m having problems getting CtW pushing a staging cert into an AZ KeyVault. it’s been permissioned as a Key Vault Certificates Officer and is being accessed from a CtW instance running on a VM in the same Subcription as the Keystore.

PS C:\Users\steved> $azurePassword = ConvertTo-SecureString "<secret>" -AsPlainText -Force

PS C:\Users\steved> $credentials = New-Object Microsoft.Azure.Commands.ActiveDirectory.PSADPasswordCredential -Property @{ StartDate=Get-Date; EndDate=Get-Date -Year 2024; Password=$azurePassword}

PS C:\Users\steved> $MyServicePrincipal = New-AzADServicePrincipal -DisplayName "Certify The Web" -PasswordCredential $credentials
WARNING: Upcoming breaking changes in the cmdlet 'New-AzADServicePrincipal' :
- The output type is changing from the existing type :'Microsoft.Azure.Commands.ActiveDirectory.PSADServicePrincipal' to the new type :'Microsoft.Azure.PowerShell.Cmdlets.Resources.MSGraph.Models.ApiV10.IMicrosoftGraphServicePrincipal'
- The output type is changing from the existing type :'Microsoft.Azure.Commands.Resources.Models.Authorization.PSADServicePrincipalWrapper' to the new type :'Microsoft.Azure.PowerShell.Cmdlets.Resources.MSGraph.Models.ApiV10.IMicrosoftGraphServicePrincipal'
Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.
WARNING: 12:46:00 - New-AzAdServicePrincipal will no longer assign role 'Contributor' to new created service principal by default

PS C:\Users\steved> Get-AzADApplication | Select-Object displayname, objectid, applicationid|findstr Cert
WARNING: Upcoming breaking changes in the cmdlet 'Get-AzADApplication' :
- The output type is changing from the existing type :'Microsoft.Azure.Commands.ActiveDirectory.PSADApplication' to the new type :'Microsoft.Azure.PowerShell.Cmdlets.Resources.MSGraph.Models.ApiV10.IMicrosoftGraphApplication'
Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.
Certify The Web                                                         <ob1-id> <app-id>

PS C:\Users\steved> Get-AzSubscription -SubscriptionName "INFRA CORP"

Name       Id                                   TenantId                             State
----       --                                   --------                             -----
INFRA CORP <ob2-id>                             <tenant ID>                          Enabled

PS C:\Users\steved>

The Task is setup as

with the credential as

I can now see the application in my az console and have permissioned it with all permission bar purge (we’re using access policies rather than RBAC, even though we assigned the “Key Vault Certificates Officer” role above)

The worrying thing is I don’t see any attempt to establish connectivity to Azure at the time the script kicks off according to the logs, all I see logged is

2021-11-08 14:01:45.819 +00:00 [INF] ---- Beginning Request [infra-test2.***.com] ----
2021-11-08 14:01:45.819 +00:00 [INF] Certify/5.5.7.0 (Windows; Microsoft Windows NT 10.0.14393.0) 
2021-11-08 14:01:45.824 +00:00 [INF] Beginning Certificate Request Process: infra-test2.***.com using ACME Provider:Certes
2021-11-08 14:01:45.824 +00:00 [INF] Requested identifiers to include on certificate: infra-test2.***.com
2021-11-08 14:01:45.824 +00:00 [INF] Beginning certificate order for requested domains
2021-11-08 14:01:45.824 +00:00 [INF] BeginCertificateOrder: creating/retrieving order. Retries remaining:2 
2021-11-08 14:01:46.125 +00:00 [INF] Created ACME Order: https://acme-staging-v02.api.letsencrypt.org/acme/order/****************
2021-11-08 14:01:46.404 +00:00 [INF] Fetching Authorizations.
2021-11-08 14:01:47.531 +00:00 [INF] Got dns-01 challenge https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/*************************
2021-11-08 14:01:47.532 +00:00 [INF] Order authorizations already completed.
2021-11-08 14:01:47.543 +00:00 [INF] Requesting Certificate via Certificate Authority
2021-11-08 14:01:51.391 +00:00 [INF] Completed Certificate Request.
2021-11-08 14:01:51.419 +00:00 [INF] Performing Automated Certificate Binding
2021-11-08 14:01:52.472 +00:00 [INF] Completed certificate request and automated bindings update (IIS)
2021-11-08 14:01:52.899 +00:00 [INF] Request completed
2021-11-08 14:01:52.900 +00:00 [INF] Performing Post-Request (Deployment) Tasks..
2021-11-08 14:01:52.910 +00:00 [INF] Task [Export Certificate] :: Task is enabled and primary request was successful.
2021-11-08 14:01:52.910 +00:00 [INF] Export Certificate: Copying file (Windows file copy) to c:\users\dunnea\Documents\le-infra-test2.pfx
2021-11-08 14:01:52.911 +00:00 [INF] File Copy completed: c:\users\dunnea\Documents\le-infra-test2.pfx
2021-11-08 14:01:52.911 +00:00 [INF] Task [Deploy to Azure Key Vault] :: Task is enabled and primary request was successful.
2021-11-08 14:03:22.618 +00:00 [ERR] Failed to deploy certificate [InfraTest2] to Azure Key Vault :System.AggregateException: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry. ---> Azure.RequestFailedException: Unable to connect to the remote server
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessInternal>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.ResponseBodyPolicy.<ProcessAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.<ProcessAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RedirectPolicy.<ProcessAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()
   --- End of inner exception stack trace ---
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipeline.<SendRequestAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Security.KeyVault.KeyVaultPipeline.<SendRequestAsync>d__29.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Security.KeyVault.KeyVaultPipeline.<SendRequestAsync>d__19`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Security.KeyVault.Certificates.CertificateClient.<ImportCertificateAsync>d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Plugin.DeploymentTasks.Azure.AzureKeyVault.<Execute>d__6.MoveNext()
---> (Inner Exception #0) Azure.RequestFailedException: Unable to connect to the remote server
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessInternal>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.ResponseBodyPolicy.<ProcessAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.<ProcessAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RedirectPolicy.<ProcessAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()<---

---> (Inner Exception #1) Azure.RequestFailedException: Unable to connect to the remote server
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessInternal>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.ResponseBodyPolicy.<ProcessAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.<ProcessAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RedirectPolicy.<ProcessAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()<---

---> (Inner Exception #2) Azure.RequestFailedException: Unable to connect to the remote server
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessInternal>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.ResponseBodyPolicy.<ProcessAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.<ProcessAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RedirectPolicy.<ProcessAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()<---

---> (Inner Exception #3) Azure.RequestFailedException: Unable to connect to the remote server
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessInternal>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpWebRequestTransport.<ProcessAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.ResponseBodyPolicy.<ProcessAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.<ProcessAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RedirectPolicy.<ProcessAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()<---

2021-11-08 14:03:22.618 +00:00 [ERR] Key Vault Deployment Failed
2021-11-08 14:03:22.618 +00:00 [INF] Export Certificate :: Task Completed OK
2021-11-08 14:03:22.618 +00:00 [INF] Deploy to Azure Key Vault :: Key Vault Deployment Failed
2021-11-08 14:03:22.618 +00:00 [INF] Deployment Tasks did not complete successfully.

stevo · November 8, 2021, 2:48pm

updated to add,

I can access the Key Vault url from a browser and get an error message back, which implies raw connectivity to the KV is okay (it resolves to an azure public IP). No proxy is configured on the server

webprofusion · November 9, 2021, 5:00am

Thanks, I can confirm that keyvault is working in my own test using a keyvault setup a couple of years ago and I’ve tried it again using a newly setup keyvault and a current permissions model.

The ‘unable to connect to the remote server’ error doesn’t sound like a permissions issue but it’s possible. You mentioned that you’d setup the access without using RBAC, but the azure UI says that role “Only works for vaults that use the ‘Azure role-based access control’”

I note your powershell proves you can query azure but it doesn’t appear to perform a keyvault action.

Is your access setup for a standard Azure AD user or an App Registration?
You can normally access keyvault for any location, did you restrict the networking to only allow access from specific networks?

My own test with a new vault was setup as:

a new App Registration was added to Azure AD. I then added a new Client Secret under client credentials. (The secret “Value” is important, the secret ID is unused.)
Created new keyvault, then under Access Control (IAM) granted “Key Vault Certificates Officer” to my new “CertifyTestAppUser” app registration (it looks like you can use “Managed identity” or “User, group or service principal”, either way you have to search for the user. I used “Managed Identity”.

In Certify The Web I configured the keyvault task and added a new stored credential with the tenant id for the directory, application id (for the registered app, this is not the client secret id), then the client secret value.

stevo · November 9, 2021, 1:33pm

okay, to avoid any doubt I shall create a new RBAC based vault to see if that’s the issue (it might be an issue, but logically it can’t be the sole reason as until CtW is talking to the KV it can’t know what access control mechanism is in place)

I’ll also delete and rerun the app creation PS on the actual server, I ran it previously on my own laptop)

As an aside, I don’t seem to be able to use the “Browse Documentation” help link in the Add/Update stored credentials for Azure AD app client secret. This fails both from the server in question and a test community install on my own laptop

webprofusion · November 10, 2021, 12:09am

Thanks, sorry I think that documentation just doesn’t exist yet.

The only way to currently confirm that the KeyVault API is or isn’t being communicated with is to use something like Fiddler (on a test machine) to inspect the https conversation. Do you get the same or different results if you use a made up vault URI or wrong tenant/application id etc. I’m curious if the problem is connectivity or permissions.

stevo · January 6, 2022, 6:32pm

FYI this is working now with 5.6.2.0 using an RBAC based keyvault. It wasn’t working correctly until I upgraded, at which stage it all sprang into life.