Memory Tiering in VCF 9.1: What Changed, How It Works and How to Plan for It
DRAM is now one of the most expensive line items in a virtualization host, and in many clusters the CPUs run well below capacity because memory fills up first. Memory Tiering over NVMe, introduced in vSphere 9.0, addresses exactly that imbalance. With VCF 9.1 the feature has matured considerably: it no longer needs a reboot, it can mirror the NVMe tier in software, it accepts every VM type on a tiered host, and it is configured once per cluster. This post covers what changed, how tiering actually decides where a page lives, and the planning rules from the Broadcom documentation that determine whether it will work well in your environment.
A quick refresher
Memory Tiering uses a locally installed NVMe device as a second memory tier. DRAM is Tier 0 and the NVMe device is Tier 1; together they are presented to virtual machines as one contiguous pool of memory. Only VM memory is placed on Tier 1 — ESX kernel memory is never tiered. The feature is off by default and integrates with DRS, which considers tiered hosts when placing and balancing VMs.
How a page ends up on NVMe
Activation threshold. Cold pages begin moving to NVMe when host memory consumption reaches roughly 80% of DRAM. Below that, everything stays in DRAM — so a lightly loaded host showing zero Tier 1 usage is normal, not a fault.
Classification. ESX scans each VM's memory in batches and classifies pages by both recency (how recently a page was touched) and frequency (how often). Hot pages are kept in or promoted to DRAM; cold pages are demoted to NVMe at the end of each scan cycle.
Adaptive rate. The scan rate rises and falls with memory pressure.
CPU cost. The software overhead is small — around an eighth of a core on smaller hosts — but it scales with host memory, so keep some CPU headroom.
Why this is not "swap on a fast disk"
Swap picks pages more or less at random and often lives on shared storage that also serves VMDKs. Memory Tiering uses page-ageing to choose what to move and runs on dedicated, direct-attached NVMe, which is what lets it deliver consistent performance rather than just correctness under pressure.
What is new in VCF 9.1
Area | vSphere 9.0 | vSphere / VCF 9.1 |
Enablement | Separate partition and enablement steps; host reboot required | One cluster-level flow in vSphere Configuration Profiles, partition created automatically, maintenance mode only — no reboot |
NVMe redundancy | Hardware RAID (Tri-Mode controller or Intel VROC) | Built-in software mirroring (RAID 1) across two NVMe devices; hardware RAID still supported |
VM types on a tiered host | Several VM types could not power on | All VM types power on; some simply stay in DRAM (see below) |
Visibility | Limited | Memory card on host and cluster Summary with Tier 0 / Tier 1 stats, Memory Tiering column in the Hosts view, device health and wear visibility, notifications for eligible clusters and workloads |
Encryption | Host level | Host level or per VM |
Software mirroring
This is the change most teams will care about. You assign a primary device and a mirror_device per host. Data is written to the same partition and block number on both disks, so if one device fails, the other continues serving the tier and VMs keep running. Two rules from the docs: the mirror disk must be the same size as, or larger than, the primary; and you must pick two different devices. Because it is software, you can now add tiering redundancy to existing servers that have no hardware RAID controller.
VM types that run on tiered hosts but stay in DRAM
In 9.1 these VMs power on normally on a tiered host, but ESX automatically keeps all their memory in Tier 0:
1 GB page VMs, preallocated (prealloc) VMs and Large VMs
Confidential VMs and Fault Tolerant VMs
Low Latency VMs, PCI passthrough VMs and vQAT VMs
Nested virtualization VMs — with one exception: Virtualization-Based Security (VBS) VMs can participate if you set the VMX option sched.mem.enableNestedTiering = TRUE
Very large VMs: Broadcom notes that VMs with more than 512 GB of memory and 64 vCPUs may see performance degradation on tiered hosts. Disable tiering for those VMs and give them a full memory reservation.
Hardware requirements
Software: vCenter 9.1 or later and ESX 9.1 or later for the 9.1 workflow.
Drive class: Mixed-use enterprise NVMe, similar to a vSAN cache device. In the Broadcom Compatibility Guide (vSAN SSD program) filter for Device Type NVMe, Endurance Class D (≥ 7300 TBW), Performance Class F (100,000–349,999 writes/s) or G (350,000+), and 3 or more DWPD.
Placement: Installed locally in the host. NVMe over Fabrics or Ethernet is not supported. Check your server manual for the slot with the best bandwidth.
Dedicated: Do not share the device with vSAN or any datastore. A disk already claimed as vSAN cache or capacity cannot be used until it is removed from vSAN. Delete all existing partitions first (Broadcom KB 323144).
No hot-plug: Hot-plug and "prepare to remove" are not supported for a tiering device; deactivate tiering on the host before replacing it.
Host: No Intel Optane Persistent Memory and no NVDIMM-N installed.
Operational limits to know about: suspend-to-memory is not supported on a tiered host (suspend-to-disk and snapshots are), and Quick Boot is not supported once tiering is active.
When ordering from an OEM, choose the mixed-use NVMe option explicitly. In a vSAN ReadyNode this often means a different drive model for tiering than the one used for vSAN.
Sizing: the rules that decide success
1. Active memory must fit in DRAM
The single most important metric is active memory, not consumed memory. Broadcom says most workloads sit at 10–30% active, and the best candidates are hosts or clusters whose active memory is between 0% and 50% of current DRAM. After enabling tiering and adding VMs, aim to keep active memory at or below about 50% of DRAM.
2. Start with the 1:1 ratio
The ratio is set with tier_size_pct (1–400, a percentage of DRAM). The default of 100 means NVMe tier = DRAM size, doubling host memory, and is the recommended value. By default the tier is capped at the DRAM size up to a maximum of 4 TB; if the NVMe device is smaller than DRAM, its full capacity is used.
tier_size_pct | DRAM : NVMe | Host with 1 TB DRAM |
25 | 4 : 1 | 1 TB + 256 GB = 1.25 TB |
50 | 2 : 1 | 1 TB + 512 GB = 1.5 TB |
100 (default) | 1 : 1 | 1 TB + 1 TB = 2 TB |
Note that the physical drive can be larger than the tier — for example, 1 TB of tier carved from a 1.6 TB drive.
3. A worked example
A 1 TB DRAM host is at 80% consumed (800 GB) and 20% active (200 GB). Enable 1:1 tiering and double the VM count: consumed becomes 1.6 TB of a 2 TB pool and active becomes 400 GB. That 400 GB is still under 50% of the 1 TB DRAM, so performance should hold.
4. Account for higher overhead
A tiered host reserves more memory for the hypervisor itself. Broadcom's example: a 1 TB non-tiered host uses about 2% for overhead, while a host with 512 GB DRAM plus 512 GB NVMe uses about 7.96% of total memory. Expect slightly less reservable memory than the raw numbers suggest.
5. Excluding VMs changes the ratio for everyone else
When you take a VM out of tiering you must give it a full reservation, and that DRAM leaves the tiering pool. On a 1 TB + 1 TB host, excluding two 200 GB VMs leaves 600 GB of DRAM for 1 TB of NVMe — an effective 0.6:1 ratio for the remaining tiered VMs. Exclude sparingly.
Operational behaviour
Don't combine with memory overcommit. Broadcom's guidance is explicit: do not use memory overcommit when tiering is active.
vMotion takes longer. Migrations take about 1.5x to 2x as long because pages must be read from NVMe first, with almost no impact on the running VM and no extra DRAM needed on the source.
Reservations still work. Normal (non-pinned) reservations are honoured and can be split across DRAM and NVMe.
Guest OS behaviour matters. Linux guests that use page cache heavily can keep pages "active" longer, so they may tier less than Windows VMs (Broadcom KB 448305).
What happens when an NVMe device fails
Without a mirror: VMs that use only DRAM are unaffected. VMs with pages on the failed device keep running until one of those pages is accessed; if that page fault cannot be serviced, the VM powers off. With vSphere HA enabled and enough spare capacity, HA restart policies can restart those VMs on any other connected host — a dedicated failover host is not required.
With a mirror: the surviving device continues serving the tier and VMs keep running.
Either way, entering maintenance mode after a failure is a manual step in 9.1 (or automated by a hardware management plug-in such as Dell OpenManage or HPE OneView if configured to do so). The device is then replaced and the configuration updated.
Bottom line
VCF 9.1 turns Memory Tiering from a careful, host-by-host project into a cluster setting. The technology is the easy part now; the outcome depends on planning. Use the right drive class, mirror the tier, keep active memory well inside DRAM, start at 1:1, avoid overcommit, and exclude only the VMs that truly need it. In the next post I walk through the actual configuration, both in the vSphere Client and with ESXCLI.
Comments