VM Snapshots in VCF 9.1: Safe Rollback, Forensics and Immutable Recovery

Mohammed Bilal

3 hours ago

6 min read

VM snapshots are one of the most useful tools in a VCF administrator's kit, and one of the most misused. Used well, they give you a fast way back from a failed change, a way to capture memory evidence during an incident, and, with vSAN data protection, recovery points that ransomware cannot delete. Used badly, they become forgotten delta files that fill datastores, slow VMs and break backups. This post covers how snapshots work in VCF 9.1, the rules Broadcom publishes for them, which management components you must never snapshot, and how to use immutable vSAN snapshots as a last line of defence.

What a snapshot actually contains

A vSphere snapshot preserves the state and data of a VM at a point in time. Broadcom KB 342618 lists the files involved:

File	What it holds
.vmsd	The snapshot database: the tree of snapshots and their child disks. Snapshot Manager reads it.
.vmsn	The VM configuration at snapshot time and, if memory was included, the VM's active state.
-delta.vmdk / -sesparse.vmdk	Child disks that receive every write after the snapshot. On VMFS6 all snapshots use SEsparse. Elsewhere, SEsparse is used for disks larger than 2 TB.
.vmem	The VM's memory contents, present when memory was captured.

From the moment you take a snapshot, the base disk is frozen and every write goes to the child disk. Reverting discards the child. Deleting the snapshot merges (consolidates) the child back into its parent. On vSAN ESA, VM snapshots use ESA's native log-structured snapshot engine instead of delta-disk chains, so creating and deleting them is much lighter. The operational rules below still apply to how long you keep them.

The two options that change everything

Snapshot the virtual machine's memory. This is selected by default. The snapshot stores the VM's running state, so reverting returns a powered-on VM exactly where it was. It takes longer, and the VM is stunned while memory is written. Without memory, a revert returns the VM powered off.
Quiesce guest file system. This needs VMware Tools. Tools flushes guest file system buffers (on Windows through VSS) so on-disk data is consistent. Pre-freeze and post-thaw scripts can add application-specific hooks. Memory and quiesce are mutually exclusive.

With neither option selected, you get a crash-consistent snapshot, which is equivalent to pulling the power cord. That is fine for most rollback scenarios, and it is what several Broadcom management-appliance procedures ask for.

The rules Broadcom publishes

KB 318825 applies to vCenter and ESX 8.x and 9.x:

Snapshots are not backups. If the base disk is lost, the snapshot files alone cannot restore the VM.
Up to 32 snapshots in a chain are supported, but use only 2 to 3 for good performance.
Do not keep a single snapshot for more than 72 hours. It keeps growing and can exhaust the datastore.
Never increase a virtual disk (or a virtual RDM) while it has snapshots. Remove all snapshots first, or you risk corruption and data loss.
Never cancel a running snapshot delete or consolidation task.
Backup software creates snapshots through the API, and those may not appear in Snapshot Manager. Check regularly for leftovers.

VCF management components: what is allowed

This is where most VCF incidents start. Several management appliances have specific rules, and one class must never be snapshotted at all:

Component	Guidance	Source
NSX Manager and NSX Edge nodes	Not supported. The distributed database and real-time state cannot be safely captured or restored with VM snapshots, and doing so can cause cluster instability or database corruption. Use NSX file-based backup to an SFTP server instead.	KB 433822
vCenter	A standalone vCenter can be snapshotted online without memory. With Enhanced Linked Mode, power off every vCenter in the SSO domain, snapshot all of them, and revert them together if needed. Avoid snapshots of vCenters in a vCenter HA configuration.	KB 318333
SDDC Manager	Use a snapshot only as a temporary rollback measure. It is not a backup, so take a file-based backup before upgrades.	KB 432552
VCF Operations	Take the whole cluster offline from the admin UI first. Then snapshot every node, including cloud proxies, with memory and quiesce cleared, and bring the cluster back online.	KB 342576
vSAN witness appliance	Avoid taking snapshots of it.	KB 387069
All management VMs	Delete snapshots within 24 to 48 hours and never keep them past 72. Never revert one management VM on its own, because it can break integration with vCenter and NSX and corrupt internal databases.	KB 452757

Snapshots for incident response

When you suspect a compromise, the most valuable evidence is often in memory: running processes, open network connections, injected code and encryption keys. Logging in to the guest to collect it changes that memory and can alert an attacker. A snapshot that includes memory captures it from outside the guest.

Take a snapshot with Snapshot the virtual machine's memory selected.
Copy the .vmsn and .vmem files (plus the disks if needed) to an isolated forensic location, and record hashes.
Only then contain the VM: disconnect its network adapters, isolate it with distributed firewall rules, or power it off.
Convert the memory checkpoint with Broadcom's vmss2core tool (KB 323788). Use -W8 for Windows 8 to 11 and Server 2012 to 2025, -W for older Windows, and -N for Linux.

vmss2core -W8 suspect-vm-Snapshot1.vmsn suspect-vm-Snapshot1.vmem

Two caveats from the KB. For encrypted VMs, the files must be decrypted before conversion, so agree the key-access procedure with your security team before an incident. For Windows guests, Virtualization-Based Security and Hyper-V can prevent a usable dump.

Immutable snapshots with vSAN data protection

Attackers who reach vCenter commonly delete snapshots and backups before they encrypt. vSAN data protection, included with VCF on vSAN ESA clusters, gives you scheduled snapshots that ordinary administration cannot remove.

Requirements. A vSAN ESA cluster, or a compute cluster mounting a vSAN storage cluster, and the VMware Snapshot Service appliance. Encrypted VMs, linked clones, and VMs that already have vSphere snapshots cannot join a protection group.
Protection groups. Select VMs individually, by name patterns using * and ?, or (new in 9.1) by vSphere tags, so new workloads are protected automatically.
Schedules. Up to 10 local snapshot schedules per group, each with its own frequency and retention. 9.1 adds daily, weekly and monthly retention schedules for long-term operational and cyber recovery.
Consistency. vSAN data protection snapshots are not quiesced. They are crash-consistent.
Immutability mode. Once enabled on a protection group, the group cannot be edited or deleted, its membership cannot change, and its snapshots cannot be edited or deleted, even by an administrator. Snapshots expire only through retention. Turning immutability off requires a support case (KB 411539), so size retention carefully: immutable snapshots consume capacity until they expire.
Replication. Replication to a peer cluster (with a VMware Live Recovery subscription) supports an RPO from 1 minute to 24 hours (default 5 minutes) and up to 200 retained snapshots on the peer site. Replication-type protection groups do not support immutability mode, so keep a separate local group with immutability for the same VMs.
Health check. 9.1 adds the vSAN Snapshots Immutability Check. It turns red when any protection group on a cyber recovery cluster has immutability disabled (KB 424480).

Design point: local snapshots live on the same datastore as the VMs. They protect you against deletion and encryption, not against losing the cluster. Pair them with replication to a cluster managed by a separate vCenter and SSO domain, and with off-site backups.

Keep snapshots short-lived and locked down

Split privileges. vCenter has separate privileges under Virtual machine > Snapshot management: Create snapshot, Remove snapshot, Rename snapshot and Revert to snapshot. Give operators only what they need. Very few people need Revert.
Protect the datastore. A memory snapshot contains whatever was in RAM, including credentials and keys. Treat datastore browse and download rights as access to that data.
Find old snapshots. The vCenter alarm Virtual machine Consolidation Needed status flags failed consolidations. A quick PowerCLI report finds anything older than three days:

Get-VM | Get-Snapshot |
  Where-Object { $_.Created -lt (Get-Date).AddDays(-3) } |
  Select-Object VM, Name, Created, @{N='SizeGB';E={[math]::Round($_.SizeGB,1)}} |
  Sort-Object Created

Automate the clean-up. Schedule snapshot creation before a maintenance window from VM > Configure > Scheduled Tasks. Make deletion part of the change's closure criteria, and review the snapshot list in VCF Operations regularly.

Quick decision guide

Situation	Use
Patching a guest OS or application	vSphere snapshot without memory (quiesced if the app supports it). Delete it within 72 hours.
Upgrading VCF management components	Follow the component rules above. File-based backups first, and never snapshot NSX.
Suspected compromise	Memory snapshot, copy the files out, then contain the VM and analyse with vmss2core.
Ransomware resilience	vSAN data protection with immutability, plus replication and off-site backup.
Long-term retention	A backup product, or vSAN data protection schedules. Never long-lived vSphere snapshots.

VM Snapshots in VCF 9.1: Safe Rollback, Forensics and Immutable Recovery

What a snapshot actually contains

The two options that change everything

The rules Broadcom publishes

VCF management components: what is allowed

Snapshots for incident response

Immutable snapshots with vSAN data protection

Keep snapshots short-lived and locked down

Quick decision guide

Broadcom references

Recent Posts

Comments

Subscribe Form