Upgrading to VCF 9.1, Part 2: Management Components, NSX, vCenter, ESX and Finishing Tasks
In Part 1, we got the control plane onto 9.1:
VCF Operations and its cloud proxies
SDDC Manager
The new VCF management services and the license server
Licensing
The fleet software depot
This part finishes the job. We upgrade the remaining management components, plan and precheck the management domain, and then work through NSX, vCenter, ESX, NSX Edge, and the finishing tasks, in the order Broadcom requires.
Everything here runs from Build → Lifecycle in VCF Operations unless I say otherwise. After each upgrade stage completes, wait about 15 minutes before scheduling the next one. VCF runs an automatic reconciliation in the background, and overlapping tasks can cancel the upgrade or make reconciliation fail.
Step 6: Upgrade the management components
Do these in this order, and only the ones you run:
VCF Identity Broker
VCF Automation
VCF Operations orchestrator
VCF Operations for networks
Log Management
VCF Operations HCX
Data Services Manager
Until each one reaches 9.1, you can't run lifecycle, certificate or password operations on it from VCF Operations.
VCF Identity Broker
An embedded Identity Broker 9.0.x is not upgraded here. It upgrades as part of its vCenter upgrade.
An appliance-mode Identity Broker must sit on the management network and be able to reach your identity provider. To upgrade it:
Go to Build → Lifecycle → VCF Management and open the Upgrade tab.
Next to Identity Broker, click Run Prechecks and fix anything that fails.
Click Upgrade and follow the progress in Upgrade details.
Remove the old Identity Broker appliance only after all management components are on 9.1.
If the Upgrade tab is missing under VCF Management, see Broadcom KB 444151.
VCF Automation
The 9.1 upgrade is a blue-green deployment. A new VCF Automation instance is built next to the running one, which stays active until the switch. Prepare:
IP addresses on the same network as the current nodes. Up to 9.1.0.300, you need a /29, which is 8 sequential IPs. From 9.1.0.400, you need 5 unique IPs: three for the new nodes and two as a buffer. The old node IPs are freed after the upgrade.
One new internal FQDN with a unique IP outside that range. Your existing user-facing FQDN is kept.
One password for the new vmware-system-user and admin accounts.
From 5.2.x, Aria Automation 8.x is first imported into VCF Operations and then upgraded to VCF Automation 9.1. Authentication later moves from VMware Identity Manager to VCF Identity Broker.
The rest
Upgrade the remaining components through the VCF Operations lifecycle, using each one's guide:
VCF Operations orchestrator
VCF Operations for networks, which can only be upgraded through VCF Operations fleet lifecycle, not its own UI
Log Management 9.1
HCX. Once HCX is on 9.1, every paired HCX appliance must run 9.0 or later, or the site pairing breaks.
Data Services Manager
From 5.2.x, Aria Operations for Logs has no upgrade path. You deploy Log Management 9.1 as a day-N task.
Step 7: Plan the domain upgrade
The core components are planned per domain, starting with the management domain.
Go to Build → Lifecycle, expand VCF Instances, and select the management domain.
On the Upgrades tab, click Plan Domain Upgrade.
On Target Version, pick the 9.1.x target.
On Customize Target Version, optionally click Customize Upgrade to set a different target for individual components.
On vCenter and NSX Upgrade Options, choose optimized maintenance window or sequential. Step 9 explains the difference.
Review the Plan Overview and click Submit Plan.
Step 8: Run the upgrade prechecks
On the domain's Upgrades tab, under Prechecks, click Run Prechecks.
Choose a target version and a scope: all components or specific ones. The Upgrade readiness option is for planning a future release.
When the run finishes, click View Details. Use the Errors, Warnings, and Silenced tabs, and expand a result to read its Result, Remediation, and Impact.
Fix the issues and click Retry on a task, or Retry All Failed Resources.
To keep the full list, use Export Full Report to download it as CSV. The view shows only errors, warnings, and silenced items.
A failed precheck doesn't technically block the upgrade, but treat every error as a blocker unless you understand it. A precheck can't be cancelled once started, and only the latest results are kept.
Step 9: Upgrade NSX Manager and vCenter
Before you start:
Assign a valid vDefend license if you use NSX security features. NSX security licenses from before version 9 are no longer supported.
If the domain runs vSphere Supervisor, check KB 92227 and the VKS Upgrade Planner first.
If you use NSX Federation, upgrade the NSX Global Managers before the local NSX Managers.
In 9.1, the NSX Manager cluster and the NSX Edge clusters are upgraded separately. The Edges come later, in Step 11. For NSX Manager and vCenter, you pick one of the two scenarios you chose in the plan.
Option A: Optimized maintenance window
This scenario builds the new vCenter while the old one keeps running, upgrades NSX, and then switches over. vCenter is down only for the switchover, which Broadcom puts at about 5 minutes under ideal conditions. It requires the vCenter target version to match the current SDDC Manager version.
Under VMware vCenter Preparation Stage, click Configure. Choose a backup option (a file-based vCenter backup is recommended), then the temporary network: Automatic generates a link-local address, and Static uses an IP, mask and gateway on the management subnet. Choose the deployment and storage size, then click Finish.
Click Run Prechecks, then Schedule. Choose when preparation starts and whether switchover is Automatic or Manual.
Under VMware NSX Manager Nodes Upgrade, click Configure, run the prechecks, and click Upgrade Now. Wait for the confirmation that all NSX Manager nodes are upgraded.
If the NSX instance is shared by several domains, make sure every one of those domains has completed its vCenter switchover.
Under VMware vCenter Switchover Stage, click Switchover Now.
If something fails before the switchover completes, the vCenter reduced downtime upgrade (RDU) rolls back to the source vCenter automatically.
Option B: Sequential
NSX Manager first, then vCenter.
Under VMware NSX Manager Nodes Upgrade, click Run Prechecks, then Upgrade Now or Schedule. The precheck covers every domain that shares the NSX Manager cluster.
Under VMware vCenter Upgrade, click Configure and choose the mechanism. vCenter Reduced Downtime Upgrade uses the same backup, temporary network, and size choices as above, with an automatic, manual or scheduled switchover. vCenter Regular Upgrade means vCenter is down for the whole upgrade. It is only offered when the source is 9.0 or later. From 5.2.x or vSphere 8, RDU is the only choice.
Optionally run the prechecks, then click Upgrade Now or Schedule.
After either option:
Restore each cluster's vSphere DRS automation level to the value it had before the upgrade. Record the values before you start. KB 87631 has a PowerCLI method.
Transition any remaining vLCM baseline clusters to images before the ESX upgrade. In VCF, this is done with Broadcom's PowerShell script or the SDDC Manager API, not by hand in the vSphere Client.
Next, upgrade VKS and then the Supervisor, if you run them. Use the VKS Upgrade Planner, because these can need intermediate versions.
Step 10: Upgrade ESX
VCF 9.x supports only vLCM image-based clusters. NSX Manager and vCenter must already be upgraded.
Physical vSAN witness host: upgrade it manually before the stretched cluster.
Virtual witness: upgraded automatically.
External storage: check and update its drivers yourself. VCF doesn't manage them.
Then run the upgrade:
Build the 9.1 image in the vSphere Client. Use a composite image if you have mixed hardware. Confirm that the ESX version matches the VCF bill of materials.
In VCF Operations, select the VCF instance and go to Image Management → Import Image. Import from the vCenter or from an exported file.
Select the domain. On the Upgrades tab, under VMware ESX Upgrade, click Configure.
Select the clusters (all by default, or Custom Selection), assign the image, and set the options described below.
Click Finish. Hosts you left out can go in another batch with Configure New Batch.
Click Run Prechecks, then Upgrade Now or Schedule for each batch.
Option | What it does | My take |
Enable sequential cluster upgrade | Off means clusters upgrade in parallel. With more than ten clusters, the first ten run in parallel and the rest run in sequence. | Turn it on for the management domain. |
Enable Quick Boot | Skips the full hardware reboot. | Only on hardware that supports it. Leave it off on hosts with Memory Tiering, which doesn't support Quick Boot. |
VM power state / Migrate powered-off and suspended VMs | Controls what happens to VMs when a host enters maintenance mode. | Migrating powered-off VMs avoids surprises later. |
Retry policy | Sets the retry delay and the number of attempts. | The defaults are fine. |
Enable Live Patch | Applies a Live Patch without rebooting. | Doesn't apply to a major upgrade such as 9.0 to 9.1. Choose Auto or clear it. |
Step 11: Upgrade the NSX Edges and finalize NSX
On the domain's Upgrades tab, under VMware NSX Edge Clusters Upgrade and NSX Upgrade Finalize, click Configure.
Select all Edge clusters (the default) or specific ones. Choose sequential or parallel, then click Finish. If there are no Edge clusters, the wizard skips straight to finalize.
Click Run Prechecks, then Upgrade Now or Schedule.
NSX 9.x features stay unavailable until every ESX host in every cluster that shares the NSX Manager runs 9.x and the finalize step completes.
Step 12: Finishing tasks
vSphere Distributed Switch. Once ESX and vCenter are done, in the vSphere Client go to Networking, right-click the switch and choose Upgrade → Upgrade Distributed Switch. Pick the version, check host compatibility, and click Finish.
Data Services Manager databases, if you use DSM.
VMware Tools 13.1 and VM compatibility. Plan these with the application owners. Both need guest reboots.
vSAN on-disk format, then vSAN File Service. The on-disk format upgrade is one-way: once done, the cluster can't go back to the earlier format. Do it only when every host in the cluster is on 9.1 and you are sure you won't roll back.
Configuration upgrades. On the domain's Upgrades tab, run the prechecks, then under Configuration Upgrades click Apply All. Do this in a maintenance window. Pending configuration upgrades don't block future upgrades, but don't leave them for long.
Workload domains and other instances
Once the management domain is done, you're on VCF 9.1. Workload domains can follow on your own schedule, using the same plan, precheck and upgrade flow. With several VCF instances, finish the management domains of the remaining instances before any workload domain.
Monitoring, history and logs
Progress: each stage on the Upgrades tab shows its status. Running and finished tasks are also in the Tasks panel.
History: the upgrade history shows what ran, when, and with what result.
Logs: when a stage fails, collect the upgrade logs before you retry, and open a support case if you need one.
Post-upgrade checklist
Confirm every domain component shows the 9.1.x target version, and that the latest prechecks are clean.
Delete the pre-upgrade snapshots of VCF Operations and SDDC Manager.
Delete the powered-off 9.0 fleet management appliance and, once all management components are on 9.1, the old Identity Broker appliance.
Confirm licensing: all vCenters and hosts licensed, with no license sync errors.
Restore the DRS automation levels you recorded.
Re-check backups. Point SDDC Manager, vCenter and NSX backups at your backup target and run one successful backup of each.
Import the customized VCF Operations content you exported in Part 1.
From 5.2.x: keep Aria Suite Lifecycle and VMware Identity Manager until you have moved authentication to VCF Identity Broker, then decommission them.
Bottom line
The 9.1 upgrade rewards patience. Upgrade VCF Operations by hand, let it lead everything else, wait out the reconciliation between stages, and make the vSAN format change last.
Comments