In my previous blog article I focused primarily on upgrading a vCenter Server Appliance (VCSA) version 5.5 to 6.5. I am going to continue my 5.5 upgrade and leverage VCSA 6.5 to perform an upgrade of my 5.5 hosts and vSAN cluster. This is essentially Part 2 of that article. I am going to demonstrate how to upgrade a simple (3) node vSAN cluster running vSAN 5.5. This cluster is running in a nested lab environment but the procedure I am going to show you is really no different than what you would do for a physical environment. If you do not use vSAN and want to upgrade your hosts from vSphere 5.5 to the latest version of vSphere then take a look at this great article from Melissa Palmer (VCDX 236) where she demonstrates how to upgrade ESXi hosts using vSphere Update Manager (VUM).
The cluster is obviously a hybrid vSAN deployment (SSD + HDD) and not an All-Flash deployment; All-Flash was introduced in vSAN 6.0. To upgrade vSAN 5.5 to 6.5 we must first upgrade the ESXi hosts from 5.5 to 6.5. You can accomplish the upgrade in a few ways.
- vSphere Update Manager (VUM) – reference Melissa’s article.
- Update Manually from console; mount the 6.5 ISO image and upgrade existing ESXi installation.
- Upload the ESXi offline bundle (zip file) and use ESXCLI to upgrade the hosts.
Just about everyone has seen the ESXi upgrade procedure using VUM or performed a host upgrade from the console. I’m going to change it up a bit and demonstrate the upgrade using the offline bundle. I prefer VUM because of its automation and orchestration benefits but I will perform this manual procedure for simple purpose of illustrating what that procedure looks like.
Before we get started you should be well prepared for the vSphere and vSAN upgrad. Start by reviewing the vSAN 6.6.1 Upgrade Considerations document (webpage). This is easily the most important reference to review. Immediately download this PDF (export to PDF option available on webpage) and review it cover-to-cover before you begin working on vSAN.
Here is a HIGH LEVEL overview of the upgrade process for vSAN. You can find the graphic below on Page 5 of the PDF.
VMware also has an Upgrade Checklist available for your convenience. Review this thoroughly and read the KB Articles available in the checklist. They are very helpful!
Upgrading ESXi and vSAN
As I stated just a little bit ago, I am going to perform my upgrade using the Offline Bundle. Why? Just to be a little different. A lot of resources out there demonstrate how to do this using VUM or via console. The offline bundle is going to involve the command-line a bit, specifically ‘esxcli’ on my hosts.
VUM is probably the preferred choice thanks to its wonderful orchestration and automation benefits. I could also upgrade each host manually via console (iLO, iDRAC, etc.) but that would require a bit more overhead. There’s also some elevated risk because I could easily fat finger or overlook something. Chances of user error are much higher using a manual procedure. Whatever the case may be, choose the upgrade procedure/method that suits you and your environment and those options are…
- vSphere Update Manager (VUM).
- Manually via ESXi console.
- Manually using Offline Bundle and command-line.
In my other blog article regarding the upgrade of vCenter 5.5 to 6.5, I stressed the importance of verifying the VMware HCL for your ESXi hosts. If you are running vSAN then not only do you need to reference the ‘Systems/Servers’ HCL but you must reference the VMware vSAN HCL as well. They are two separate HCL lists! And don’t forget to reference the drivers/firmware listed on the vSAN HCL. If you have something and it’s not listed on there then do not use it. Don’t assume it’ll work later just because it works now. Find the correct hardware and/or recommended storage driver, network adapter, etc. that is required for your specific environment. Overlooking any of this will ultimately increase your chances of experiencing chaos and put your data at risk.
HCL Verification is Extremely Important!
- Verify all vSAN Storage I/O components are valid on the VMware vSAN HCL.
- Verify all driver and firmware for both the storage controllers & disks are compliant with the VMware vSAN HCL.
- Update the ‘vSAN HCL Database’ file from the vSphere Web Client. Here is a manual procedure from VMware KB 2145116.
Another very important TIP for you…do not ignore the vSAN Health Check tool or any other error messages you may be receiving in vSphere. If you have an error message…Google it, submit a support ticket to VMware or server hardware vendor, etc. Whatever you do, don’t ignore it and move on. Get the issue resolved before proceeding.
So to this point my VCSA 6.5 is up and running and my next goal is to Patch & Upgrade my ESXi hosts. The three (3) hosts are running vSphere 5.5 and vSAN 5.5. Here is a summary of the versions and specific build numbers that I am working with.
- vCenter Server 6.5 (VCSA) – Build 8024368
- vSphere ESXi 5.5 – Build 7967571 – each host configured with a single SSD (cache) drive and single HDD (capacity) drive.
- vSphere ESXi 6.5 U1 Offline Bundle (update-from-esxi6.5-6.5_update01.zip)
RECOMMENDATION: Plan to perform your upgrade procedure during off-hours.
Before uploading the ESXi 6.5 Offline Bundle I applied critical patches to my hosts via VUM. So make sure patch your systems first. As you can see below my three hosts are compliant with their VUM host baseline and ready to go.
The ESXi Offline Bundle is 460.72 MB in size. To accommodate this file, as I do not have space on the local ESXi boot volume, I temporarily create a local VMFS volume on each host and upload the ZIP file there. As you can see below from my ‘df -h’ (or vdf -h) command on my ESXi host I do not have space for this ZIP file locally on my hosts. That is why my temporary workaround is to use a local VMFS datastore to store my ZIP file. I’ll remove the VMFS volume later when I’m finished.
Once the ESXi offline bundle is uploaded to the local VMFS volume the next step is to place the host into Maintenance Mode; you can choose either the ‘Evacuate all data to other hosts’ or the ‘Ensure data accessibility from other hosts’ from the available options (below). In this example I am going to use the Ensure data accessibility option.
Next open a Putty (SSH) session with the host and use the ‘esxcli software vib install’ command to apply my ESXi 6.5 U1 update. Here is the command format I used followed by screenshot of the specific command I used to update my hosts.
esxcli software vib install -d "/vmfs/volumes/<local-vmfs-datastore>/update-esx65.zip
I press ENTER and let the command run. It only takes a few minutes to apply the update. Once the update installation is confirmed you must then reboot the host from the vSphere Web Client (or the command-line).
After the host reboots, exit Maintenance Mode and then continue updating the remaining hosts in the vSAN cluster one at a time. You may have to manually reconnect the hosts to the cluster after they reboot. I had to do this on a couple occasions so just be aware of that moving forward.
Another common error message you may see is…
“Host cannot communicate with one or more other nodes in the vSAN enabled cluster.”
The workaround for this message can be found in VMware KB Article 2143214. It has happened to me a few times when upgrading. There are two workarounds available in this article including restarting the ‘vpxa’ agent on the local host.
[root@esx-01a:~] /etc/init.d/vpxa restart
Either one of those workarounds summarized in the KB article should resolve this brief minor issue. If it does not then open a support ticket with VMware.
Upgrade vSAN On-disk Format
So we have successfully upgraded the hosts from ESXi 5.5 to 6.5 U1. Next we are going to upgrade the On-disk Format Version for vSAN (version 1.0 -> 5.0). This procedure is accomplished at the vSAN cluster-level in the vSphere Web Client (or HTML5 client). I recommend doing this off-hours and schedule it as planned downtime.
Navigate to ‘Configure > vSAN > General‘ and here you will see the current On-disk Format Version of 1.0; you will also see the number of disks in your configuration. Each of my ESXi hosts contains a single SSD (caching) and two HDD (capacity). Click on Pre-check Upgrade. The process should take just a few moments and then provide you with a Ready to upgrade status.
Click on Upgrade to begin the On-disk Format upgrade process.
The procedure to upgrade the disk format can take some time. A warning message will appear and ask if you want to ‘Allow Reduced Redundancy’ during the procedure. The option is ‘only usable if your setup is at the limit of the protection level’ which basically means based on the protection level in the storage policy assigned to a VM. The disk upgrade happens one disk group at a time where data is completely evacuated. The amount of data on vSAN will impact how much time the process will take to complete. So be very patient and yes do it this during off hours as well.
Now that my ESXi hosts and vSAN disk-format is completely upgraded I can now turn on the ‘Performance Service’ for vSAN. This is accomplished at the cluster-level from Configure > vSAN > Health and Performance as seen below. If you tried enabling this prior to upgrading you most likely received an error message. That is because you must be running vSphere ESXi 6.5 express patch 2 or later in order to use it.
Conclusion
So that was my quick step-by-step for upgrading my ESXi 5.5 hosts running vSAN 5.5 to vSphere 6.5. Not very in-depth but certainly conceptual enough to help guide you down the right path. If you are running vSAN 5.5 and need to upgrade before the September deadline make sure you do your homework. Don’t rush your upgrade, there’s still time to do a lot of planning that will ensure the integrity of your vSAN before, during and after the upgrade is complete.
Let’s reiterate some very important planning steps.
- Use the VMware vSAN Upgrade Checklist. Use this checklist as a guide for your planning.
- Check both HCLs! You will need to validate your environment on the ‘Systems/Servers‘ HCL as well as the ‘vSAN‘ HCL.
- Use the vSphere Upgrade Troubleshooting guide if you run into any issues. Don’t wait for something to happen before reading this. Read through it to see some of the possible issues you could encounter.
- Check out VMware vSAN upgrade best practices (KB 2146381).
- Under any circumstances do not ignore the vSAN Health Check tool before and after the upgrade. Use the tool to your advantage and pay close attention to what it is telling you.
I started this blog article before vSphere 6.7 and vSAN 6.7 was released. So take a look at the data sheets, release notes and the What’s New articles available for both.
vSphere and vSphere with Operations Management
The Wifi-Cable 