HPE Firmware Management for VMware ESXi – Introduction

In most of the VMware vSphere Environments (and even more with vSAN) the firmware and driver management is a critical and at the same level annoying task. That’s one of the reasons why All-In-One solutions like Dell EMC VxRail for VMware vSAN are very popular in large enterprises. The struggle of manually matching and updating the firmware and drivers for VMware ESXi made me write the blog post series HPE Firmware Management for VMware ESXi.

Just to make sure, HPE, Fujitsu, DELL, and other server vendors do have similar challenges (even if there are differences in detail) but I have the most experience with the HPE Proliant series.

Why are firmware updates an important task?

Just like other software updates, this prevents stability problems and security issues. But also performance optimizations and new features are not uncommon.

Why is the driver, firmware, and ESXi build mapping important?

Hypervisors like VMware ESXi operate very close to hardware functions and depend on optimal setup and perfect interoperability.

HPE Driver, Firmware and ESXi Build Mapping

HPE maintains a great document called “Current SPP HPE Custom Image vibsdepot SPP Mapping for Gen9 and later“. This document includes a matrix for the recommended combinations of Driver Addon and Firmware Package, called SPP or VUP for the vSphere Lifecycle Manager (vLCM) Firmware Addon.

HPE Firmware Management for VMware ESXi - HPE Driver, Firmware and ESXi Build Mapping

So, in the case, you have installed your VMware ESXi with the HPE Custom Image for VMware vSphere 7.0 U3 (Oct 26 2021) and applied the SPP 2021.10.0 to your server hardware you are fine. However, the big challenge is to upgrade from an older version to this combination, in terms of drivers and firmware.

Large Scale challenges

Let’s have a look at the possible upgrade processes for the HPE firmware. HPE servers do not have an Out-Of-Band upgrade interface for firmware like the current DELL servers. So, there are mainly two options:

  • Offline Update (OS is shut down and SPP ISO is mounted)
  • OS integrated via Smart Update Tools (available for most common OSes)
    • VMware vSphere Lifecycle Manager (vLCM) Firmware & Driver Addon

In the case, you use the Offline Update your process will look like this:

  1. (Optional) Apply new SPP to the ServerProfile (e.g. per vSphere Cluster) – Typically from ServerProfile Template
  2. Enter ESXi Maintenance Mode
  3. ESXi Shutdown
  4. Offline update of the Firmware (though HPE OneView or manually )
  5. (optional) Additional restart to apply some kinds of Fimware
  6. ESXi Start
  7. ESXi Update (including HPE Drivers)
  8. ESXi Reboot
  9. Exit ESXi Maintenance Mode

In general, this means a downtime per server for at least 45 minutes. In the case of a 32-Mode Cluster, this will take about 24 hours (one server at a time)!

In the case, you use the Smart Update Tools your process will look like this:

  1. Disable ESXi Lockdown Mode if enabled (this should be the case)
  2. Apply new SPP to the ServerProfiles (e.g. per vSphere Cluster) – Typically from ServerProfile Template
  3. Wait untill Fimware is Installed Online
  4. Enable Lockdown Mode is required (this should be the case)
  5. Enter ESXi Maintenance Mode
  6. ESXi Update (including HPE Drivers)
  7. ESXi Reboot
  8. (optional) Additional restart to apply some kinds of Fimware
  9. Exit ESXi Maintenance Mode

The firmware update and the ESXi update can be decoupled in two separate steps, but only one reboot (and one VM evacuation – which is also a significant impact on your environment) of the server is required.

With the firmware offline update, you are not able to get a whole vSphere Cluster update on the scale of 32 or even 64 nodes done in a single day. The solution is the Smart Update Tool, which enables you to do online firmware updates in the first step and the ESXi update, including the reboot to apply the firmware, in a second step.

The firmware offline update can be done with the standard license of HPE OneView but the advanced license allows you to centrally manage the ServerProfile and review the compliance. The Smart Update Tool always requires ServerProfiles and the advanced license for HPE OneView.

vSphere Lifecycle Manager Firmware Addon

The VMware vSphere Lifecycle Manager (vLCM) Firmware Addon is a more advanced integration of the Smart Update Tools into the vLCM update process. But in general, this is based on the same technology (HPE Smart Update Tools + HPE OneView + HPE OneView for VMware vCenter).

Smart Update Tools disatantages

You may have noticed that the Smart Update Tool cannot do its job when ESXi Lockdown mode is enabled. This is one of the main drawbacks of this method. But another ESXi security configuration that is becoming more and more popular also interferes with Smart Update Tools Online Firmware Update: Prohibit the execution of custom code inside ESXi (VMkernel.Boot.execInstalledOnly)

A minor problem might be the additional port requirements between ESXi Management Kernel and HPE OneView.

Explanation of terms

To make sure that all explanations in this and the following part are clear, I would like to explain some important terms.

Firmware Baseline

Typically this is an HPE Service Pack for ProLiant (SPP) ISO in the HPE OneView inventory. In some corner cases, the SPP can be modified by adding custom Firmware Packages.

HPE OneView ServerProfile Template

This is a Template to define server configurations like BIOS and iLO settings. But the definitions we are interested in are Firmware settings:

  • Firmware Baseline (which SPP)
  • Installation Method (Smart Update Tools or not)

HPE OneView ServerProfile

This is the individual configuration profile of one server. The Profile is connected to the Profile Template but can be modified manually. You are able to override the manual changes or Inherit new settings by Updating from Profile Template.

HPE Smart Update Tool (SUT)

The SUT is the OS component that connects to the HPE OneView Appliance.

Note:

Only if the SUT Install Mode is set to at least “AutoDeploy” and the ServerProfile Firmware Installation Method is set to at least “Firmware Only using Smart Update Tools”, an online Firmware Update is performed.

HPE Firmware Management for VMware ESXi – Summary

HPE offers with HPE OneView and the HPE Smart Update Tools a great stack to manage and monitor firmware compliance. The concept of ServerProfile Templates and ServerProfiles is the gold standard for managing server configurations and firmware versions. The only challenge is to efficiently manage a large number of clustered servers with minimal downtime per server. The following parts of the HPE Firmware Management for VMware ESXi blog post series will show some automation examples to close this gap.

External Ressources

Other Parts of this series

Leave a Reply