Why vSAN ESA Requires NVMe: Device Rules, Health Checks and Cost Planning in VCF 9.1
Flash prices are up again. One question keeps coming up in design reviews: can a vSAN Express Storage Architecture (ESA) cluster use cheaper SAS SSDs, or even spinning disks, for workloads that "don't need much performance"? The answer is no. ESA is an NVMe-only architecture. That is not a marketing preference: the design of ESA assumes the queueing model, latency and direct PCIe attachment that only NVMe provides. This post explains why, shows how to check that your hosts are compliant, and covers how to keep ESA storage costs down without falling back to legacy media.
What ESA accepts, and what it does not
These are the rules from Broadcom's documentation and KB articles that decide whether a device can be claimed by an ESA storage pool:
NVMe only. ESA has a strict NVMe-only requirement. SAS SSDs, SATA SSDs and HDDs are not supported, including devices wrongly reported as flash (KB 435891). Broadcom states it has no current plan to add SAS or SATA support to ESA (KB 326717).
Direct PCIe attachment. NVMe devices must connect directly to a PCIe slot. PCIe switches are acceptable where more lanes are needed. NVMe devices behind a Tri-Mode controller are not supported in any mode (RAID, HBA, pass-through or JBOD), for both OSA and ESA (KB 314305).
Certified devices. Drives must be a make and model listed in the vSAN ESA section of the Broadcom Compatibility Guide or in the ReadyNode bill of materials. You can swap a drive model only within the same server OEM.
Device count and size. An ESA host needs at least 1 storage device and supports up to 24. A vSAN storage cluster host needs at least 2. Broadcom's stated minimum device capacity is 1.6 TB (KB 326717).
Endurance. Read-intensive (1 DWPD) devices are supported in the ESA-AF-0, AF-2, AF-4, AF-6 and AF-HighDensity profiles, but not in AF-8. You can replace a 1 DWPD device with a 3 DWPD device, but not the other way round, and mixing endurance levels is not recommended (KB 326717).
Network. All-flash configurations, including ESA, need at least 10 GbE. Check the per-profile requirement in the ESA ReadyNode hardware guidance, because larger profiles call for more bandwidth.
Common trap: a "flash" device in the vSphere Client is not necessarily NVMe. A SAS SSD, or an HDD that has been tagged as flash, still shows as flash. What matters is the transport and the device's listing in the compatibility guide.
Why the bus and protocol matter more than the media
Three generations of storage interface are involved here. SATA and SAS were designed when the slowest part of the I/O path was a rotating platter. NVMe was designed for solid-state memory sitting directly on the PCIe bus. The differences below come from the protocol specifications, not from any vSAN tuning:
Characteristic | SATA (AHCI) | SAS | NVMe |
Attachment | Through a shared controller | Through a shared HBA or RAID controller | Directly on the PCIe bus, with its own controller per device |
Command queues | 1 queue | 1 queue per device | Up to about 64K I/O queues |
Commands per queue | 32 | Deeper than SATA, but limited by the device and controller | Up to about 64K |
Duplex | Half duplex | Full duplex | Full duplex, parallel queues per CPU core |
Typical bandwidth | 6 Gb/s link | 12 or 24 Gb/s link | About 7.9 GB/s (PCIe Gen4 x4) or about 15.8 GB/s (PCIe Gen5 x4) |
Two of these matter most for vSAN. The first is parallelism. NVMe lets every CPU core keep its own submission and completion queues, so a busy host does not serialise I/O behind a single queue. The second is the removal of the shared controller. In a SAS design, every device in a host funnels through one controller, and that controller's queue depth, firmware and cache behaviour become part of your storage performance and failure domain. ESA was built to remove that layer completely.
Why ESA in particular cannot live with slow devices
The original storage architecture (OSA) used disk groups: a cache device in front of capacity devices. That design could hide slow capacity media behind a fast write buffer. ESA works differently:
Single tier. Every device in the storage pool provides capacity and performance. There is no cache tier to absorb writes, so a slow device slows the objects stored on it.
Log-structured writes. ESA writes incoming data quickly to a performance leg and later rewrites it as full stripes with erasure coding. That second step relies on high device bandwidth and deep queues.
Data services in the write path. Compression, encryption and, in 9.1, global deduplication are processed at the top of the stack, which assumes a fast back end.
Self-healing speed. When a host fails or enters maintenance with full data migration, vSAN rebuilds or moves its data to the rest of the cluster. Rebuild time is governed by device throughput. On NVMe this is measured in minutes to hours. On spinning media it would stretch long enough to leave data exposed with reduced redundancy for a dangerous length of time.
Broadcom's KB 435891 describes what happens when non-NVMe devices end up in an ESA pool: severe I/O latency, "Device performance has deteriorated" warnings in /var/run/log/vmkwarning.log, and risk of metadata inconsistency.
Check your hosts in five minutes
Run these on each ESX host, from the ESX Shell or SSH, to confirm that only certified NVMe devices are claimed by ESA:
# Devices claimed by the ESA storage pool
esxcli vsan storagepool list
# Vendor, model and transport of each device
esxcli storage core device list | grep -E "Display Name|Vendor|Model|Is SSD|Device Type"
# vSAN disk eligibility and tier (look at the SingleTierDisks group)
vdq -i -H
# Adapters: NVMe devices should sit behind an NVMe/PCIe adapter, not a SAS controller
esxcli storage core adapter listThen compare each model with the vSAN ESA list in the Broadcom Compatibility Guide. In the vSphere Client, the same information is under Cluster > Configure > vSAN > Disk Management, and the vSAN health hardware compatibility checks flag devices that are not certified.
If you find a non-NVMe device in an ESA pool, KB 435891 gives the fix: put the host into maintenance mode with Full data migration, remove the non-compliant device from the pool, install a certified NVMe device, claim it, and bring drivers and firmware to the versions listed in the compatibility guide.
The cost conversation: compare effective capacity, not raw $/TB
Raw price per terabyte is the wrong metric. What you pay for is usable, protected capacity after data reduction. A simple formula keeps the discussion honest:
Effective cost per TB = raw cost per TB x protection overhead / data reduction ratioProtection overhead comes straight from the storage policy:
Policy | Raw capacity per 1 TB of data |
RAID-1, FTT=1 | 2x |
RAID-1, FTT=2 | 3x |
ESA RAID-5 (2+1, small clusters) | 1.5x |
ESA RAID-5 (4+1) | 1.25x |
ESA RAID-6 (4+2) | 1.5x |
An illustrative example, using made-up round prices: if a hybrid design needs RAID-1 FTT=2 (3x) with no data reduction, 1 TB of protected data costs three times the raw HDD price. An ESA design using RAID-6 (1.5x) with a 2:1 data reduction ratio needs only 0.75x the raw NVMe price. NVMe can therefore cost four times as much per raw terabyte and still break even, before you count the cache devices, controllers, extra hosts, rack space and power that the hybrid design needs to reach the same performance. Use your own quotes and your own measured compression and deduplication ratios. Data reduction depends heavily on the data, and encrypted or already-compressed data reduces very little.
Lowering ESA cost the supported way
Let every device carry data. ESA has no dedicated cache devices, so you are not buying drives that hold no capacity.
Use erasure coding. In VCF 9.1, ESA selects RAID-6 automatically as the default when the cluster has enough hosts, which gives FTT=2 protection at 1.5x instead of the 3x of FTT=2 mirroring.
Turn on space efficiency. VCF 9.1 adds global deduplication for ESA: a cluster-wide, post-process setting that also works with encryption. It comes alongside improved compression with no performance penalty.
Pick the right endurance. Read-intensive (1 DWPD) NVMe is supported in most ESA profiles and is often enough for general-purpose virtual machines.
Use high-density and QLC options where they fit. The AF-HighDensity profile, and the QLC-based Cyber Recovery ReadyNodes in 9.1, are aimed at capacity-heavy uses such as deep snapshot retention.
Right-size the cluster. Higher performance per host often means fewer hosts, switch ports and licences for the same service level.
What about my existing hybrid OSA clusters?
vSAN OSA all-flash remains supported. The VCF 9.0 release notes say that the hybrid configuration in the vSAN Original Storage Architecture will be discontinued in a future VCF release. If you run hybrid OSA today, plan the move during your next hardware refresh. You can either rebuild on ESA-certified NVMe nodes, or move to all-flash OSA if you are not ready for ESA. Avoid buying new spinning-disk capacity for vSAN now, because it locks you into a configuration with a published end of life.
Pre-purchase checklist
Every capacity device is NVMe and listed for vSAN ESA in the Broadcom Compatibility Guide.
NVMe drives connect directly to PCIe or through a PCIe switch, never through a Tri-Mode controller.
The host matches an ESA ReadyNode profile, or the changes stay within KB 326717.
The device endurance class is allowed for the chosen profile, and endurance levels are not mixed.
Network bandwidth meets the profile requirement (10 GbE minimum, more for larger profiles).
Capacity is sized on protected, effective capacity using your real data reduction ratios.
Comments