Memory Tiering for VKS and VMs in VCF 9.1: Running Kubernetes on Half the DRAM

In most VCF environments I work in, memory is the resource that runs out first. CPU sits at 30 to 40 percent while hosts are "full" because every gigabyte of DRAM is already promised to a VM. Kubernetes makes this worse. Platform teams size worker nodes for peak, developers set generous memory requests, and the result is a lot of configured memory that is touched only now and then.

Memory Tiering in VCF 9.1 tackles exactly that gap. This post looks at it from the Kubernetes side: how vSphere Kubernetes Service (VKS) clusters behave on tiered hosts, what Broadcom's performance testing shows for VKS compared with ordinary VMs, and how to size a VKS cluster so you get the extra density without paying for it in latency.

If you are new to the feature, start with my earlier posts on what changed in VCF 9.1 and how to plan for it and the step-by-step configuration guide. I won't repeat the full setup here.

A 60-second refresher

Memory Tiering adds a second memory tier to each ESX host using a local NVMe device:

• Tier 0 is DRAM.

• Tier 1 is NVMe.

ESX watches how often each memory page is accessed. Hot pages stay in DRAM, cold pages move to NVMe, and a page that becomes hot again is promoted back. The guest OS sees one normal block of memory, so nothing is installed inside the VM or inside Kubernetes.

A few rules from the Broadcom documentation shape everything that follows:

• Off by default. You turn it on per cluster or per host. It needs ESX 9.1 and vCenter 9.1.

• Tier 1 size. The default Tier 1 size equals DRAM (a 1:1 ratio, so 2x total memory), capped at 4 TB. The ESXCLI setting accepts 1 to 400 percent of DRAM. The default is the recommended starting point.

• When pages start moving. Pages start moving to NVMe when host DRAM consumption reaches about 80%. Below that, Tier 1 stays empty.

• What can move. Only VM memory pages can move. ESX kernel memory and pinned memory always stay in DRAM.

• No memory overcommit. Broadcom says not to use memory overcommit with tiering turned on. Total configured VM memory should fit inside DRAM plus NVMe.

Figure 1. To the hypervisor, a VKS node is just another VM. Tiering applies to it the same way it applies to a database or app VM.

How a VKS cluster sees tiered memory

A VKS cluster is a set of control plane and worker node VMs that the Supervisor creates on the vSphere cluster. That is the key point for tiering: a node is a VM. When you enable Memory Tiering on the vSphere cluster that backs the Supervisor, every host in that cluster gets a Tier 1, and any node VM placed on those hosts can use it.

Three practical consequences:

1. No Kubernetes-side change. There is no driver, kubelet flag, or node label to set. The node's guest OS sees its full configured memory. Kubernetes keeps scheduling pods against the node's allocatable memory exactly as before.

2. One policy for both worlds. VKS nodes and traditional VMs on the same host draw from the same two tiers. Broadcom's guidance is to use the same sizing rules for both, so you don't need a separate Kubernetes capacity model.

3. Node VMs are not excluded. Broadcom's list of VM types that run on tiered hosts but are kept out of tiering covers profiles like Fault Tolerant, Low Latency, PCI passthrough and Confidential VMs (full list below). Standard VKS node VMs are none of these, so they participate like any other VM.

Figure 2. Illustration of the cluster Summary Memory Tiering card, with example values for four hosts. Use it to check host count, per-tier capacity and active memory.

VM classes and reservations: what actually stays in DRAM

VKS node sizes come from VM classes. Many teams choose between guaranteed classes, which reserve the node's full memory, and best-effort classes, which reserve none. It is easy to assume a guaranteed class keeps a node out of NVMe. It doesn't.

Broadcom's 9.1 documentation is explicit: memory reservations work with Memory Tiering, and reserved memory can be split between DRAM and NVMe. A reservation guarantees that the capacity exists. It does not promise which tier that capacity comes from. So guaranteed and best-effort nodes both tier. The difference is admission control, not placement in DRAM.

What does force memory into DRAM is pinned memory, for example from passthrough devices, Fault Tolerance or latency sensitivity set to High. Those VMs are kept out of tiering automatically. You can also turn tiering off for an individual VM with an advanced setting (covered in the "Per-VM controls" section of my configuration guide).

Tip from the field: Don't assume a guaranteed VM class keeps a node in DRAM. Check it: the memstats command in the monitoring section below shows how much of each node VM sits in Tier 1. If a workload truly can't tolerate NVMe-backed pages, give it a dedicated node pool and validate that pool before putting production pods on it.

VM types that stay in DRAM on tiered hosts

In VCF 9.1 these VM types can power on on a tiered host but run entirely from DRAM:

Excluded from tiering	Typical reason you'd have one
Fault Tolerant VM	FT-protected infrastructure services
Low Latency VM	Latency sensitivity set to High
PCI Passthrough VM / vQAT VM	GPU, NIC or QuickAssist passthrough
Confidential VM	Memory-encrypted guests
1 GB Page VM / Prealloc VM	Large pages or preallocated guest memory
Large VM	Very large memory footprints
Nested Virtualization VM	Off by default. VBS VMs can opt in with sched.mem.enableNestedTiering = TRUE

Keep in mind that every DRAM-only VM on a host takes DRAM away from everything else. If GPU worker nodes with passthrough share hosts with your general-purpose node pools, the general pools see a smaller share of DRAM, so their effective DRAM-to-NVMe ratio gets worse.

What Broadcom's testing shows

Broadcom's performance team ran HammerDB against MySQL in two forms: inside a VKS node and inside a traditional VM. Both compared the same pair of hosts:

• Baseline: 1 TB of DRAM with tiering off.

• Tiered: 512 GB of DRAM plus 512 GB of NVMe Tier 1, a 1:1 ratio, so the same 1 TB total.

The MySQL guest had 6 vCPUs, 120 GB of memory and a 92 GB InnoDB buffer pool. Each run used a 25-minute ramp-up and a 60-minute measurement window at four concurrency levels.

Virtual users	Think + key time	VKS (% of baseline TPM)	VM (% of baseline TPM)
100	40 ms	97%	97%
200	80 ms	95%	97%
300	120 ms	92%	91%
500	400 ms	92%	94%

Figure 3. With half the DRAM, every run stayed within 10% of the all-DRAM baseline. VKS and VM results track each other closely.

How I read these numbers:

• VKS adds no tiering penalty of its own. At every load level, the VKS and VM results are within 3 points of each other. The Kubernetes layer doesn't change how tiering behaves.

• The cost grows with concurrency, then flattens. Light load loses about 3%. Heavier load settles around 6 to 9%. That fits the design: more users touch more pages, so more pages come from NVMe.

• The test was sized to the guidance. Broadcom's sizing rule is to keep active memory at or below 50% of DRAM, and these runs followed it. Exceed that and your results will differ. The next section is about staying inside it.

Sizing a VKS cluster for a tiered host

The one number that decides success is active memory as a share of DRAM. Active memory is the working set the host is touching right now. It is always lower than consumed memory, and Broadcom notes that most workloads keep it between 10% and 30% of configured memory. Here is the method I use.

Step 1: Measure, don't guess

Before you enable tiering, look at the history for the hosts that will run your node pools. In VCF Operations, or in the vSphere Client performance charts, compare active memory with consumed memory over at least a full business cycle, including batch windows and month-end runs.

Broadcom's guidance is to target hosts and clusters whose active memory sits between 0% and 50% of DRAM before tiering. VCF Operations also has a Memory Tiering dashboard and a what-if analysis you can use to model this.

Step 2: Check both limits

Every host has to pass two checks:

1. Capacity: total configured VM memory must fit in DRAM plus Tier 1, with no overcommit, and leave room for VM overhead and ESX itself.

2. Performance: total active memory should stay at or below 50% of DRAM. Between 50% and 75%, test and monitor. Above 75%, expect a real slowdown.

Step 3: Work an example

Take a host with 512 GB of DRAM and the default 1:1 ratio, so 1 TB of total memory. The worker nodes use a VM class with 64 GB of memory, and we measured about 30% active memory per node.

Figure 4. Configured memory can go well past DRAM. Active memory is what has to stay under the 50% line.

Check	Without tiering	With tiering (1:1)
Usable memory	512 GB DRAM	512 GB DRAM + 512 GB NVMe
64 GB worker nodes	7 (448 GB)	13 (832 GB)
Active memory at 30%	134.4 GB (26% of DRAM)	249.6 GB (48.8% of DRAM)
Left for overhead and headroom	64 GB	192 GB

Thirteen nodes is where it stops. A fourteenth node brings active memory to 268.8 GB, or 52.5% of DRAM, which is just past the recommended line.

The assumption matters as much as the arithmetic. If the same nodes ran at 40% active, 13 nodes would put the host at 65% of DRAM, in the "test and monitor" zone. That's why Step 1 comes first.

Two more things to account for:

• Plan HA at the cluster level, as usual. After a host failure, the surviving hosts must absorb the restarted node VMs and still pass both checks.

• Keep CPU headroom. Tiering has a small CPU cost, which Broadcom puts at about one-eighth of a core on smaller hosts. Packing more nodes per host also raises CPU demand. Keep enough CPU headroom for both.

Avoid monster nodes

Broadcom notes that VMs with more than 512 GB of memory and 64 vCPUs can lose performance on tiered hosts. For VMs that large, the recommendation is to turn off tiering for the VM and use a full memory reservation. For Kubernetes, the better answer is usually more, smaller worker nodes. They tier well, spread across hosts more evenly, and limit the blast radius of a node failure.

Day-2 operations: what changes for platform teams

• vMotion is slower. Live migration of a tiered VM takes about 1.5x to 2x longer, because pages on NVMe have to be read before they are sent. Host maintenance, vLCM remediation and Supervisor upgrades that move node VMs around will take longer, so plan your maintenance windows for it. vMotion doesn't increase DRAM needs on the destination host.

• DRS understands tiers. DRS has placement and load-balancing logic for Memory Tiering and supports clusters that mix tiered and non-tiered hosts. That is handy during a phased rollout.

• Rolling updates create new nodes. A VKS version upgrade or VM class change replaces nodes with freshly booted VMs. Power-on is when DRAM consumption climbs fastest, and tiering only starts moving pages at about 80% DRAM consumption. Upgrading one node pool at a time keeps hosts from getting tight.

• Changing tiering needs maintenance mode, not a reboot. In 9.1, enabling, disabling or reconfiguring tiering needs maintenance mode but no reboot, and the cluster profile remediation evacuates hosts for you.

• Quick Boot is not supported on tiered hosts. Hot-plug of the tiering NVMe device is not supported either. To replace a device, turn tiering off on that host first.

• Software mirroring is worth it. With a second NVMe device per host, 9.1 mirrors Tier 1 in software (RAID1, no RAID controller needed), so tiering keeps running if one device fails. For hosts that run many node VMs, I treat the mirror as mandatory.

Monitoring tiered VKS hosts

After go-live, two numbers tell you whether the sizing holds:

Figure 5. Thresholds from Broadcom's Memory Tiering performance guidance.

In the vSphere Client. The host and cluster Summary tabs show Tier 0 and Tier 1 capacity and consumption. For trends:

1. Select a host or a node VM.

2. Go to Monitor → Performance → Advanced.

3. Open Chart Options and choose the Memory view.

4. Plot the memory type read and write bandwidth and latency counters.

The per-tier read latency is the counter to alert on.

Figure 6. Illustration of where to chart Memory Tiering counters for a host or node VM. Alert on Tier 1 read latency.

On the host. For a per-VM view of how much memory sits in Tier 1, run this in an SSH session on the ESX host:

memstats -r vmtier-stats -u mb -s name:memSize:active:tier1Target:tier1Consumed

To confirm the host's tiering configuration and devices with the 9.1 ESXCLI namespace:

esxcli memtier status get
esxcli memtier config get
esxcli memtier device list -c

Figure 7. Illustration of the read-only esxcli memtier commands for checking a host. Changing settings needs maintenance mode.

Reading the data: Low Tier 1 consumption is not a problem. It usually means DRAM consumption is still under the roughly 80% threshold where tiering starts. Rising Tier 1 read latency together with active memory above 50% of DRAM is the signal to rebalance node pools or add hosts.

Rollout checklist for VKS clusters

• All hosts behind the Supervisor run ESX 9.1, managed by vCenter 9.1.

• Each host has a dedicated, local, mixed-use NVMe device: endurance class D (3 DWPD or more), performance class F or G, not shared with vSAN. Add a second device for mirroring.

• Historical active memory per host is at or below 50% of DRAM, measured across a full business cycle.

• Node pool sizes are recalculated against both checks: configured memory within DRAM plus NVMe, and active memory at or below 50% of DRAM, including the HA failover case.

• Latency-critical or passthrough workloads have their own node pools, and the DRAM they take is in the math.

• No worker node class is above 512 GB of memory and 64 vCPUs.

• Maintenance windows allow for vMotion taking 1.5x to 2x longer.

• Alerts are set for Tier 1 read latency above 200 µs and host active memory above 50% of DRAM.

• Screenshots, tickets and runbooks are scrubbed of IP addresses and FQDNs before sharing.

Bottom line

For Kubernetes on VCF 9.1, Memory Tiering is a density lever, not a workaround. Size by active memory, keep it under half of DRAM, and VKS nodes behave just like the VMs next to them.

The VKS-versus-VM results show that the Kubernetes layer doesn't change the trade-off. The rules are the ones you would apply to any VM: measure active memory, start at 1:1, mirror the NVMe, and watch Tier 1 latency. Follow them and the same hosts can run noticeably more worker nodes. In the example above that was 13 instead of 7, without buying more DRAM.

Broadcom references

• Memory Tiering over NVMe (vSphere 9.1)

• Memory Tiering Considerations and Best Practices

• Virtual Machine Profiles and Performance with Memory Tiering

• VCF 9.1 release notes: What's new in vSphere

• ESXCLI 9.1 reference: esxcli memtier

• KB 449016: Memory Tiering activation thresholds and monitoring

• Memory Tiering performance white paper