VMCA 6.5 as an Intermediate CA

Certificates have always been an interesting topic to me. I have seen complex PKI infrastructures on several occasions but more often than not most environments have utilized certificates in a limited capacity. In regards to vSphere, a majority of the time the default certificates provided by vSphere were sufficient for most deployments.

Security is certainly a hot topic these days and I believe we will see more and more organizations begin implementing additional security measures to ensure integrity and confidentiality of data transmission. The VMware Certificate Authority (VMCA) was released with vSphere 6.0. By default the VMCA issues a certificate to each vCenter Server and ESXi host. vSphere Admins (and Security Admins) can replace the default certificates with newer VMCA signed certificates, replace ALL certificates with customized certificates or configure the VMCA as an Intermediate (Subordinate) Certificate Authority (CA). The VMCA simplifies certificate management compared to previous releases of vSphere. It is much less cumbersome thanks to vSphere 6.

In this “how to” blog article I am going to walk you through the process of configuring the VMCA on an external Platform Services Controller (PSC) as an Intermediate CA. The VMCA root certificate can be replaced by a certificate signed by a third-party CA or an enterprise CA. I am going to demonstrate this procedure with a Microsoft Enterprise CA.

If you are not 100% familiar with the PSC deployment process, there are two possible deployment scenarios. There is an embedded deployment whereby the PSC and vCenter Server are deployed on the same virtual appliance or an external PSC where the PSC and vCenter Server roles are deployed on separate virtual appliances.

When looking at the various procedures for replacing the default certificates with a custom certificate, I stand beside VMware’s recommendation of deploying the PSC first followed by replacing the VMCA root certificate. Once you have accomplished this you can proceed with deploying vCenter and adding the ESXi hosts to the environment and everything else will seamlessly fall into place. The VMCA will assign these certificates as the vSphere infrastructure is deployed.
The other alternative? If the PSC, vCenter and ESXi hosts are deployed and a vSphere HA/DRS cluster is present (operational) then you will have to generate and replace the certificates for each component separately. Sounds like a lot more work to me.

There are three (3) steps (high-level) to configure the VMCA as an Intermediate CA and they must be done in order.

1. Replace the VMCA Root Certificate.
2. Replace the Machine SSL Certificates.
3. Replace the Solution User Certificates.

The Certificate Manager tool will manage all three steps for us. Once we have replaced the root certificate the tool will automatically replace the Machine SSL certificates and lastly replace the Solution User Certificates. As I stated previously, I am utilizing an Enterprise Microsoft PKI Infrastructure in this demonstration.

Environment Overview

Before we proceed with configuring the VMCA as an Intermediate CA let’s review a logical layout of the environment starting with the Microsoft PKI Infrastructure. I deployed only two (2) systems in my Initech.local lab environment.

• Offline Root CA (rootCA01)
• Active Directory Domain Controller (dc01.initech.local)
  • DNS Server
  • Subordinate CA
  • Web Server (IIS)

I could have deployed each onto a separate server but I have limited resources in my lab. In a real-world environment I would expect these roles to be separated (and likely redundant). I used the following Microsoft Technet article by Yung Chou to configure my PKI. Click HERE to see his awesome article which is very detailed.

NOTE:  The Microsoft Technet article by Yung Chou is sometimes unavailable. If the link does not work above please use this alternate link for configuring your MS PKI by Arthur Remy (3-part series starting w/ Part 1)

INSTALLING A TWO TIER PKI HIERARCHY IN WINDOWS SERVER 2016 – PART 1

Once the various roles were installed on my Windows Servers above I had to perform a few additional configuration steps to create the Certificate Templates needed for my Root VMCA Certificate and for my Machine/User Certificates. Reference VMware KB 2112009 for that procedure.

PSC Deployment & Default Certificates

I am deploying a fresh (pristine) vSphere 6.5 environment for my Initech.local lab and utilizing an External PSC and vCenter Server (separate appliance-based VMs). As I mentioned earlier, if the PSC and vCenter Server are both deployed and the ESXi vSphere HA/DRS cluster is configured then you will have some additional work to do.

I have deployed a few key components up so far. First, I have a standalone ESXi 6.5 host ready to go that I will be using to deploy my two VCSA appliances. I deployed an External PSC to this standalone ESXi host and now I am ready to proceed with replacing the default VMCA certificate. I have not deployed vCenter up to this point. Only the PSC. Before doing anything let’s take a look at the default certificates on the newly deployed PSC. My PSC FQDN is psc-01a.initech.local.

1. Type the following URL into your web browser and log in using the SSO Administrator account that was configured during the PSC deployment.

   https://<psc-fqdn>/psc

   

2. Under Certificates select the Certificate Store option and then select the TRUSTED_ROOTS store from the drop-down menu. Click Show Details for the certificate and review the information. Make note of some of this information as it is going to change very soon. Browse a few other stores and certificates and document some of that information as well because we will change those certificates later after the root certificate is changed.
   

   This slideshow requires JavaScript.
3. Next select the Certificate Authority option. Here you can review the default certificates on the PSC. There will be four (4) tabs across the top. If you have a freshly installed PSC like I do then you should only find certificates on the Active Certificates tab and the Root Certificate tab.
   

   This slideshow requires JavaScript.
4. The certificates that we are most interested in are the Trusted Root Certificates, Machine Certificates and the Solution User Certificates as we are going to replace these in order. Select the Certificate Management option and then enter the SSO admin credentials and click Submit. You will then see three (3) tabs. Document the information for each certificate (or take a screenshot). Select a tab, choose a certificate and then select Show Details.
   

   This slideshow requires JavaScript.
5. Logout of the PSC homepage and proceed to the next step.

Certificate Manager

Next we are going to replace the root certificate on the VMCA. One thing I recommend before starting is perform a quick snapshot of the PSC. If things go awry you can quickly revert back and start over. When we are finished with everything we will remove the snapshot.

Another thing to make note of is the local CSR template file (certool.cfg) found in the ‘usr/lib/vmware-vmca/share/config/’ directory. Open this CFG file and take a look at the contents. The CSR we are going to custom generate will look similar to this.

So we are going to be accessing the PSC appliance via Putty (SSH) and using WinSCP to copy our CSR and Certificates to and from the appliance. We also need to configure the default Shell to BASH for the root account. Open a Putty (SSH) session to the PSC and execute the following command to access the BASH shell.

shell.set --enabled true

Once you have accessed the temp BASH shell run the next command to configure the BASH shell for root access.

chsh -s /bin/bash root

Log out of the BASH shell and then log back in for the changes to take affect. Leave your Putty (SSH) open and then launch WinSCP. Open a session with the PSC using WinSCP and keep it open.

We are now ready to go! We will start everything by using the Certificate Manager tool on the PSC.

1. First thing we want to do is launch the Certificate Manager on the appliance. Execute the following command and then select Option 2.

   /usr/lib/vmware-vmca/bin/certificate-manager

   

2. Once you’ve selected Option 2 and pressed ENTER you will then be prompted to generate all certificates based on a configuration file. We want to press Y and then ENTER to continue.
3. Next we are going to be prompted for some information but first we need to provide the ‘Administrator@vSphere.local’ credentials.
4. The first configuration file is for the MACHINE_SSL_CERT.cfg. Enter the proper values for the certificate request. Some parameters are optional or you can simply use the default value.
5. The second configuration file is for the machine.cfg. Follow the prompts as you did above for the previous CFG.
6. The third configuration file is for the vsphere-webclient.cfg. Again, follow the prompts and continue.
7. Once the process is complete you will then be presented with two (2) new options. One to Genearte Certificate Signing Request(s) and Key(s) and the second option is for Importing custom certificates. We are going to use Option 1 and then enter a directory location for the CSR(s) and Private Key(s). I created my own /certs directory on my PSC which I will use for this step.
   
8. Next we will configure the certool.cfg file. Follow the prompts to configure this file that will be used for the CSR request. A new CSR file and KEY file will be generated and placed into my /certs directory. Minimize your Putty session with the PSC as we will come back to it later. I have a WinSCP session open with my PSC and browse to my /certs directory where I see my new CSR and key. I copy these two files over to a folder on my workstation.
   
9. Now we submit the CSR request to my Microsoft Subordinate CA that is running on my domain controller (dc01.initech.local). I then type the following URL into my web browser to access my AD Certificate Services certificate request page and then select the Request a certifcate link.

   http://dc01.initech.local/certsrv

   

10. On the very next page select the advanced certificate request link.
    
11. I then locate the vmca_issued_csr.csr file that I copied to my local workstation and open it with a text editor (Notepad, Notepad++, other). I then copy the contents of my CSR files and paste them into the Saved Request (Base-64-encoded certifcate request) box. From the Certificate Template drop-down menu I select my vSphere 6.5 VMCA template. Then click Submit.
    

    This slideshow requires JavaScript.
12. On the Certificate Issued page I then choose the Base 64 Encoded option and then select both links to Download certificate and Download certificate chain.
    
13. I now have two new certificates on my workstation. The ‘certnew.cer’ is the newly signed custom certificate. The ‘certnew.p7b’ is the certificate chain from my Microsoft Subordinate CA. We have to open the Certificate Chain (p7b) and export those two certificates. Simply right-click the .p7b file. A ‘certmgr’ window will open. Expand the certificate and select the Certificates folder. Here I will see my certificates in my chain. The two certificates I am working with are highlighted. The other certificate is from my offline Root CA that I used to configure my Microsoft PKI earlier. We will need this certificate as well.
    
14. Right-click each certificate and select Export. You will have to export each certificate one at a time. During the Export Wizard select the Based-64 Encoded X.509 (.CER) option and maintain the same certificate name as they appeared in my certmgr console in Step 13. Once I am finished I will see all three (3) certificates.
    
15. I have to merge these three certificates into ONE certificate. If you reference Pages 112-113 in the Platform Services Controller Guide you will see my NEW certificate must include all three (3) certificates in the following order.

    -----BEGIN CERTIFICATE-----
    <Certificate of VMCA>
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    <Certificate of Subordinate CA>
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    <Certificate of Root CA>
    -----END CERTIFICATE-----

    I repeat….THEY MUST BE IN THAT ORDER! Starting with my new VMCA cert followed by the Subordinate CA certificate and lastly the Root CA certificate. I accomplish this from a Command Prompt on my Windows-based workstation and browse to the folder containing my new certs. I execute the following command to merge the three (3) into ONE master certificate that I will name NEW-VMCA-CERT.crt.

    type vCenter.cer subordinate-DC01.cer ROOTCA01.cer > NEW-VMCA-CERT.crt

    

16. Next we have to copy this new certificate to the PSC. Before doing so, I first open the certificate with notepad to verify that my certificates have been merged into one.I then COPY that new certificate from my workstation back to my /certs directory on my PSC.
    
17. Next I return to my Putty (SSH) session with my PSC. I have two options available…Continue to import Custom Certificate(s) and to Exit the Certificate Manager. Select Option 1 to continue with the import.
    
18. First I am prompted to supply my NEW certificate. This is my NEW-VMCA-CERT.crt certificate that I just merged and copied onto my PSC. I will then be prompted for the custom key for root which was generated back in Step #8. I enter my two paths to my certificate and my key and then press Y when I am ready.
    
19. Here is the great thing about the Certificate Manager tool. It is going to do EVERYTHING for me. It is going to replace the ROOT CERTIFICATE on my PSC and generate the new Machine SSL Certificates and Solution User Certificates for me. This final step will take some time to complete.
    
    NOTE: If any issues occur during this procedure and it fails it will automatically ROLL BACK to the original certificates. You should also reference the ‘certificate-manager.log’ file located in the ‘/var/log/vmware/vmcad/’ directory on the PSC.
20. I am now going to stop and start all services on the PSC as instructed in the output from the previous process. (Or you can just simply reboot the appliance.)

    service-control --stop --all
    service-control --start --all

 

Certificate Verification

Next we are going to log back in the PSC SSO webpage and verify our new certificates. Type the following URL into your web browser using the following format.

https://<psc-fqdn>/psc

Type in the SSO administrator credentials and then click Login.

Once you are logged into the web page select Certificate Management and then provide the SSO administrator credentials. Review the NEW certificates one by one. Below I have provide screenshots of my new ROOT CERTIFICATE.

This slideshow requires JavaScript.

Conclusion

So that is the simple procedure of configuring the VMware Certificate Authority (VMCA) as an Intermediate CA running on an external PSC 6.5 appliance. The procedure can be a bit frustrating at times but follow the steps very carefully and you can easily configure the VMCA to act as an Intermediate CA for your vSphere 6.5 infrastructure.

Going forward from this point you can simply proceed with deploying the vCenter Server 6.5 (appliance) and integrate it with the PSC and lastly build and configure the vSphere HA/DRS cluster(s) as you would normally.

Keep in mind there are risks involved with this model. Having a Subordinate CA, a ‘rogue admin’ with full access/rights to the PSC could potentially ‘mint’ new and fully trusted certificates. And this trust goes up the ladder to the organization’s Root CA. My recommendation is for you to consult your Security Team before making the final decision to configure the VMCA as a Subordinate CA.

UPDATE: May 2nd, 2018

If you are looking to accomplish this but you are using an Embedded PSC and not an external PSC please take a look at my other blog article specific to that deployment. Click the link below to go there now!

VMCA 6.5 with Embedded PSC as an Intermediate CA

Useful Links

Platform Services Controller Administration (ESXi 6.5 and vCenter Server 6.5)

vCenter Server Appliance Configuration (vCenter Server 6.5)

Enterprise Microsoft PKI with Server 2012 R2 AD Certificates Services

Install & Configure Certificate Authority in Windows Server 2016

 

Remember….BE SOCIABLE, SHARE! 🙂