Configuring Memory Tiering in VMware Cloud Foundation 9.1 – Step-by-Step Guide

VMware Cloud Foundation 9.1 significantly simplifies the deployment and operational management of Memory Tiering. With software-based NVMe mirroring, enhanced observability, and improved configuration workflows, administrators can now enable Memory Tiering faster and with less operational overhead.

This guide walks through the end-to-end configuration procedure for enabling Memory Tiering in VCF 9.1.

Architecture Overview

Before configuration, it is important to understand the Memory Tiering workflow:

• DRAM acts as the primary memory tier

• NVMe devices act as the secondary memory tier

• ESXi intelligently migrates cold pages to NVMe

• Applications and VMs consume a unified logical memory pool transparently

Prerequisites

Before enabling Memory Tiering, verify the following:

Hardware Requirements:

• VMware Cloud Foundation 9.1

• Supported ESXi hosts

• Enterprise-grade NVMe SSDs

• Sufficient DRAM capacity

• vCenter Server access

• Supported firmware and drivers

Recommended NVMe Guidelines

• High endurance NVMe drives

• Low-latency enterprise SSDs

• Separate devices from vSAN when possible

• Proper sizing ratios between DRAM and NVMe

According to VMware guidance, Memory Tiering can deliver:

• Up to 4x memory expansion

• Up to 40% TCO reduction

• 2x VM consolidation improvements

Step 1 – Validate NVMe Device Visibility

First, confirm that ESXi detects the NVMe devices correctly.

Using ESXCLI

SSH into the ESXi host and run:

esxcli storage core device list | grep nvme

This will display all available NVMe devices.

You must identify the device UID for partition creation.

Example Output

naa.5cd2e414a1234567

Validate all devices before configuration to avoid deployment issues.

Step 2 – Create Memory Tiering Partitions

Once the NVMe devices are identified, create the partition used for Memory Tiering.

Create Partition Using ESXCLI

Example command:

esxcli system tieredmemory add -d <device-UID>

Example:

esxcli system tieredmemory add -d naa.5cd2e414a1234567

This prepares the NVMe device for Memory Tiering usage.

Important Notes

• One partition is required per NVMe device

• In VCF 9.1, software-based NVMe mirroring is now supported

• Hardware RAID controllers are no longer mandatory

Step 3 – Enable Memory Tiering

After partition creation, enable Memory Tiering on the ESXi host.

This can be configured using:

• vCenter UI

• ESXCLI

• PowerCLI

• vSphere Configuration Profiles

The most efficient method to configure Memory Tiering is to use vSphere Configuration Profiles. This allows you to enable the feature on all your hosts at once, while leveraging host overrides for any hosts where you do not wish to enable it.

Activate the Memory Tiering feature at a host level.

1. In the vSphere Client, select a host and click Configure.

2. Navigate to System > Advanced System Settings.

3. From the Key column, click the filter icon and enter tiering.

   The VMKernel.Boot.memoryTiering setting displays.

4. Click Edit, change the value to true, and click OK.

5. Reboot the host.

Perform the following steps to check whether a host is configured for Memory Tiering.

1. In the vSphere Client, navigate to a host and click Configure.

2. Navigate to Hardware and select Overview.

   If the host is configured for Memory Tiering, the Memory table displays Software in the Memory Tiering column.

Monitoring and Observability

VCF 9.1 introduces enhanced observability dashboards.

Administrators can monitor:

• Tier bandwidth

• DRAM vs NVMe usage

• Tier latency

• VM-level tier activity

• Consumed vs active memory

Advanced Configuration Options

VCF 9.1 also supports advanced tuning and encryption controls.

Host-Level NVMe Encryption

Parameter

Mem.EncryptTierNvme

Values

Value	Meaning
1	Enabled
0	Disabled

Per-VM Encryption

Parameter

sched.mem.EncryptTierNVMe

Value

TRUE

Useful for:

• Domain Controllers

• Financial workloads

• Sensitive applications

Disable Tiering for Specific VMs

Some latency-sensitive workloads may require DRAM-only behavior.

Examples:

• SAP HANA

• High-frequency trading

• Real-time workloads

VM Advanced Parameter

sched.mem.tiering.enable = FALSE

Final Thoughts

Memory Tiering in VMware Cloud Foundation 9.1 represents a major advancement in modern infrastructure optimization.

The new release delivers:

• Better performance

• Easier deployment

• Native software mirroring

• Improved observability

• Expanded VM compatibility

For organizations struggling with rising DRAM costs and increasing workload density requirements, Memory Tiering provides a highly practical solution to improve infrastructure efficiency while lowering the total cost of ownership.

Useful References:

• Advanced Memory Tiering Enhancements in VCF 9.1

• Memory Tiering End-to-End Configuration Guide

• Advanced Memory Tiering Configuration Options