Are USB-based boot devices being deprecated?

vSphere customers have encountered many issues related to device reliability when SD cards are used as the boot device. Endurance ratings can no longer guarantee the viability of SD cards as a boot option

Let deep dive in to those details for more information

• Reads/Writes to System Storage continue to grow, mainly to the ESX-OSDATA partition. The OSDATA partition is a critical component that is required to be always available and also be persistent across reboots. OSDATA will be used in a variety of different ways as services and applications such as vSAN and NSX grow. OSDATA is used to store some of the runtime state, and for probes, backups of items such as configuration, timestamps etc. Access to other areas on OSDATA, such as VMTools and scratch portions will also grow. With each new vSphere release, more features will start relying on the OSDATA partition.
  

• Performance demands and IO loads cannot be continued to be met by SD cards. For example, the demands on writes to the scratch area have been growing. Another concern is the high frequency of IO with bursts such as logs or traces.
  

• There is no way to reliably check and monitor for the SD card endurance, as these devices usually do not provide support for diagnostics data. Wear-out issues and remaining endurance or lifetime can go undetected. In addition, not just writes but reads are also known to cause wear issues.
  

• SD/USB devices are prone to wear over time and are thus subject to failures, they are not designed for enterprise class use-cases.

Lets see the current partition layout

Partition References

•Partition 1: systemPartition -> Bootloader Partition (4MB )

•Partition 5: linuxNative -> /bootbank (250MB)

•Partition 6: linuxNative -> /altbootbank ( 250MB)

•Partition 7: vmkDiagnostic -> First Diagnostic Partition (110MB)

•Partition 8: linuxNative -> /store (286MB)

•Partition 9: vmkDiagnostic -> Second Diagnostic Partition ( 2.5 GB)

•Partition 2: linuxNative -> /scratch (4GB)

•Partition 3: VMFS datastore

​Partition 1: Bootloader Partition (4MB)

​This Partition is needed for Booting. Which is 4 MB in size (System Partition)

​Partition 5: Bookbank (250MB)

​The hypervisor image is located on this 250 MB partition. It is formatted with old FAT. The image is called s.v00 (124MB compressed) and is decompressed during boot. It will be extracted during the boot process and loaded into the system memory.

​Partition 6: /altbootbank (250 MB)

​/altbootbank partition is empty after a fresh installation of ESXi server. Below is the Example of contents of /altbootbank partition on the fresh install ESXi Server. Once you perform an upgrade of ESXi host , the current image is copied from the bootbank partition. This makes the possibility to return to the previous version of ESXi (last known good configuration) by typing “Shift + R” while booting, if there any issues occurred during the ESXi host upgrade. Below is the Example of contents of /altbootbank partition on the Upgraded ESXi Server. It contains around 138 files in the directory.

​Partition 7: vmkDiagnostic (110 MB) – First Diagnostic Partition

​In case of ESXi Crash or PSOD, host dump file written on this partition

​Partition 8: /store (286 MB)

​This /store partition contains various ISO files for the VMware Tools for all supported operating systems. When you install the VMware Tools on the Virtual machine, VMware tools ISO will get mounted to the CD/DVD drive of virtual machine from this location.

​Partition 9: Second Diagnostic Partition (2.5 GB)

​Second Diagnostic partition is 2.5 GB as addition to the vmkDiagnostic partition (110 MB in size) . Using ESXCLI command, you can check the current status of active diagnostic partition, as we have two diagnostic partitions.

​Partition 2: /scratch (4 GB)

​The scratch partition is a 4 GB VFAT partition that is created to store vm-support output, which is needed to analyse the cause of system failures and which is needed by VMware Support to identify and troubleshoot the issues occurred on ESXi host . If the scratch partition is not present, vm-support output is stored in a ramdisk. It will be lost after the system reboot. This partition will be created only when you have free space of more than 5 GB on the ESXi local disk Partition 3: VMFS datastore

​This partition is only created, if the installation medium is not a flash memory. VMFS partition extends over the total available unallocated space of the disk and is formatted it with VMFS 5/VMFS 6 based on the ESXi version and selection. It is also referred as “Datastores“.

USB-based boot interfaces are not deprecated however VMware encourages use of M.2 and other SSDs for boot because of reliability concerns. OEMs should ensure that ESXi only see a unified "single" boot device on using dual M.2 capability

What is supported for upgrades?

• For all upgrades ESXi 7.0 onwards, we continue to support existing boot devices.

• We still allow upgrading on USB, however an extra disk with the USB device is recommended.

• Having a low endurance device or a USB device might increase the risk of wear-out and might have unreliable outcomes.

• In addition, having a low performance device will affect some features in future ESXi versions that may depend on performance.

• We strongly recommend upgrading to 32GB ( 32 Min – 128 Recommended) or higher high-endurance and high-performance devices for upgrades and new installations.

• We recommend to install larger boot media.

• Customers on previous versions (for example, 6.x) of vSphere should feel confident to move to 7.0U2c as several critical issues have been fixed with this release, and some workarounds have been provided as default mainly for the VMTools & Scratch areas. ( vSphere 6.7 U3p too have latest updates)

What is the plan to continue support for USB/SD boot for vSphere 7.0

• Apart from boot banks being larger, 7.x behavior is still the same as 6.x where things go to RAM disk if there's no high-performance storage available

• The scratch partition can be configured elsewhere (eg. NFS) for logging, but isn't an ideal scenario

• As for supporting USB/SD devices, they can continue to use those devices, but they need to supply a secondary high-quality device. You can do this from the ESXi UI which allows them to delete datastores that are not in use or partedUtil to manually do it.

*Important: If you install ESXi on M.2 or other non-USB low-end flash media, delete the VMFS datastore on the device immediately after installation. For more information on removing VMFS datastores, see the vSphere Storage documentation

What is supported for new installation?

• Strongly recommend using high performance and high endurance devices like M.2, SSDs or NVME for ESXi 7.0 which requires a boot disk of at least 32 GB of persistent. Use USB, SD and non-USB flash media devices only for ESXi boot bank partitions. A boot device must not be shared between ESXi hosts.

• ·For all new installations, you must configure scratch to some persistent location (NFS or VMFS) for logs to be saved.

vSphere 7 System Storage Layout

• Before vSphere 7, partition management was limited in that partition sizes were fixed, and the partition numbers were static. There were constraints on using multiple solutions with the 6.x partition sizes, for example, if you started to combine NSX-T, vSAN, Tanzu, vGPU etc. This restricted the support for installing large modules, debugging functionality, and possible third-party components. The need for ESXi hosts to support other VMware or 3rd party solutions is ever-increasing. Therefore, the need for a more reliable, flexible, and high-performing storage device for ESXi 7.x system storage is a necessity.
  

• Reduce IOs to SD Card by reducing access to VMTools and scratch areas of OSDAT. An ESXi boot device consists of several partitions. One of the boot partitions, the ESX-OSDATA partition is critical to ESX operation. Starting from the next major vSphere release, SD cards/USB media as a standalone boot device will not be supported. SD cards cannot be used to store the OSDATA partition, and an additional persistent device will be required. * A "standalone boot device" refers to disks or devices where all of the partitions created during the ESXi installation, including the ESX-OSDATA partition, are all on that 1 device
  

• Another significant change in the context of SD cards and USB devices is the ESX-OSData partition. All the non-boot partitions such as core dump, locker, and scratch partitions are now consolidated under the new partition called ESX-OSData (VMFS-L) partition.
  

• ESX-OSData partition must be created on a high endurance persistent storage device as there is an increase in IO requests sent to the ESX-OSData partition. The increased IO request is a result of multiple factors that have been introduced with ESXi 7.x
  

• In ESXi 7.0 Update 3 and later, the locker packages are loaded into RAM disk by default if they are stored on a USB or SD card device.

With the new partition schema in vSphere 7.x, only the system boot partition is fixed at 100 MB. The rest of the partitions are dynamic, meaning partition size will be determined based on the boot media size. When a secondary high-endurance storage device is not available, VMFS-L Locker partition is created on USB or SD devices, but this partition is used only to store ROM-data. RAM-data is stored on a RAM disk

Future versions of ESXi (8.0 and beyond)

• Plan to use a local persistent disk as the boot device for all partitions

• If a local persistent device is not already available, add another persistent device. This persistent local boot device can be industrial grade M.2 flash (SLC and MLC), SAS, SATA (including SATADOM), or PCIe NVMe disk (32GB minimum, 128GB recommended)

• If a persistent local device is not available as a boot device, SD cards can be used for boot bank partitions However a separate persistent local device to store the OSDATA partition (32GB minimum, 128GB recommended) must be provided

• Provide SAN/NAS Boot as an option for the entire System Storage (Boot banks + OSDATA partitions), when ESXi is installed on a SAN/NAS LUN

• Existing customers planning to upgrade to future revisions of ESXi and who cannot use any of the above options will not be in a supported configuration.

*Depending the boot media used and if its a fresh installation or upgrade, the capacity used for each partition varies. The only constant here is the system boot partition. If the boot media is larger than 128GB, a VMFS datastore is created automatically to use for storing virtual machine data.