@ArmouredGenius I don’t actually know much about the RDS Web Client myself (is that a new thing), this looks to be the .cer step you were referring to:
You will find that the basic setup works but when you get into multi-server setups then things get a bit more exotic. I’m pretty sure you can always use a PFX file though.