Step by Step: Configuring NVMe Memory Tiering in VCF 9.1 (vSphere Client and ESXCLI)
In the previous post I covered what changed in Memory Tiering for VCF 9.1 and how to size it. This one is the hands-on part: enabling tiering across a cluster with vSphere Configuration Profiles, adding a software mirror and encryption, verifying the result, and the ESXCLI equivalents for when you need to work on a single host.
Before you start
Versions: vCenter 9.1+ and ESX 9.1+.
Cluster management: the cluster is managed with a single image and vSphere Configuration Profiles.
Drives: one dedicated, locally attached, mixed-use NVMe per host (3+ DWPD, Endurance Class D, Performance Class F or G). Add a second device per host if you want software mirroring — the mirror must be the same size or larger than the primary.
Clean disks: the devices are not part of vSAN or any datastore and have no existing partitions (see Broadcom KB 323144 to remove them).
Capacity to evacuate: each host enters maintenance mode during configuration, so the rest of the cluster must be able to absorb its VMs through vMotion.
Host checks: no Optane PMem or NVDIMM-N installed. Quick Boot and suspend-to-memory will not be available on tiered hosts.
Using old 9.0 runbooks? 9.1 configures tiering through the esxcli memtier namespace and Configuration Profiles. Update any scripts from 9.0 to the 9.1 commands shown below before using them.
Option A: Configure the whole cluster in the vSphere Client
This is the recommended method. Settings are defined once and applied consistently to every host, with optional per-host overrides.
Step 1: Open the draft
In the vSphere Client, select the cluster and open the Configure tab.
Go to Desired State → Configuration.
On the Draft tab, select esx → memtier.
Click the nvme link, then Configure Settings.
Step 2: Turn tiering on and choose the device on each host
Set enable to True.
Next to device, click View host-specific details (or click device in the left panel).
Add a host by BIOS UUID: select Show unconfigured hosts, click Select, pick the host, then choose its NVMe device from the list of available devices.
Repeat for every host in the cluster.
Tip: If you prefer to identify the disk yourself, select the host, go to Configure → Storage → Storage Devices, select the NVMe device and copy the Location path from the Properties tab. Paste that path into the device field.
Step 3 (recommended): Add a software mirror
Next to mirror_device, click View host-specific details.
Add each host the same way and select its second NVMe device. It must be a different device from the primary.
Step 4 (optional): Encrypt the tier
Pages on the NVMe tier are not encrypted by default. Set encryption to True to encrypt them at the host level. This protects against theft of or unauthorised access to the physical drive. It is separate from VM Encryption.
Step 5 (optional): Set the ratio
tier_size_pct accepts 1–400 and is a percentage of DRAM. Leave it at the default 100 (1:1) unless your active-memory analysis tells you otherwise. Example from the docs: 16 GB DRAM with a value of 50 gives an 8 GB tier and 24 GB total.
Step 6: Save and apply
Click Save, then review each host's device and mirror selections in the draft.
Apply the draft. The Configuration Manager places each host in maintenance mode in turn, evacuates its VMs, creates the tiering partition, applies the settings and returns the host to service. No reboot is required.
To change settings later, return to the same place, click nvme → Edit, adjust and save. To turn tiering off for the cluster, select the nvme configuration and click Delete.
Option B: Configure a single host with ESXCLI
Place the host in maintenance mode before any enable, disable or reconfigure command.
# List NVMe devices that can be used for tiering
esxcli memtier device list
# Enable tiering on one device
esxcli memtier enable -d <device_1>
# Enable tiering with a software mirror
esxcli memtier enable -d <device_1,device_2>
# Enable with encryption and a 50% tier size
esxcli memtier enable -d <device_1> -c t -r 50Day-2 commands
Task | Command |
Add a mirror to an existing configuration | esxcli memtier config set -d "old-device,new-mirror-device" |
Replace the primary device | esxcli memtier config set --devices=<new-device>,<old-mirror-device-if-any> |
Replace the mirror device | esxcli memtier config set --devices=<old-device>,<new-mirror-device> |
Remove only the mirror | esxcli memtier config set -d "old-device-1" |
Change the ratio (e.g. 25% = 4:1) | esxcli memtier config set --tier-size-pct 25 |
Turn on encryption for the host | esxcli memtier config set --encryption true |
Disable, keep the partition | esxcli memtier disable |
Disable and delete the partition | esxcli memtier disable -d |
Delete the configuration entirely | esxcli memtier config delete |
Verify it worked
Cluster: the Memory card on the cluster Summary tab shows Tier 0 and Tier 1 capacity and usage, plus how many hosts are configured.
Hosts list: on the cluster's Hosts tab, use Manage Columns to add the Memory Tiering column. Configured hosts show "Software".
Host: the host Summary tab has its own Memory card with DRAM and NVMe side by side.
CLI: run esxcli memtier config get and esxcli memtier status get. If config shows Enable: True but status shows Disabled, activation failed — check the host logs and alarms.
Performance charts: under the host Memory chart options, add Memory type read/write bandwidth and Memory type read/write latency to watch each tier.
Tier 1 shows 0 GB used? If the capacity shows up correctly, that's usually expected. Pages only start moving to NVMe when host consumption reaches about 80% of DRAM. If Tier 1 capacity itself shows 0, the configuration did not apply — recheck the steps above.
Per-VM controls
Keep a VM entirely in DRAM
The VM must be powered off.
Right-click the VM → Edit Settings → Advanced Parameters.
Add sched.mem.enableTiering with value FALSE.
On Virtual Hardware → Memory, tick Reserve all guest memory (All locked).
Power the VM on.
Don't skip the full reservation. Without it, the excluded VM can end up in the host swap file, which is slower than the NVMe tier. Also remember that every reserved VM shrinks the DRAM left for the tiered VMs.
Encrypt the tier for one VM only
Power off the VM, add sched.mem.EncryptTierNVMe = TRUE under Advanced Parameters and power it on.
Allow VBS VMs to tier
Nested virtualization VMs stay in DRAM by default. For Virtualization-Based Security VMs you can opt in with sched.mem.enableNestedTiering = TRUE. Test the workload before you roll it out.
Replacing a failed tiering device
Place the host in maintenance mode, either manually or automatically through a hardware management plug-in if one is configured for that.
Hot-plug is not supported, so physically replace the device only once the host is evacuated.
Point the configuration at the new device with esxcli memtier config set (see the day-2 table) or by updating the device in the cluster draft.
Exit maintenance mode. DRS rebalances according to its automation level.
Post-enablement checklist
Active memory stays at or below about 50% of DRAM on each host.
Memory overcommit is not in use on tiered clusters.
Very large VMs (more than 512 GB and 64 vCPUs) are excluded and fully reserved.
HA is enabled, so restart policies can recover VMs after a device failure.
Maintenance windows allow for vMotion that takes 1.5x–2x as long on tiered hosts.
NVMe device health and wear are reviewed as part of routine operations.
Comments