Upgrading to VCF 9.1, Part 1: Planning, VCF Operations, SDDC Manager and Management Services
Upgrading to VCF 9.1 is not just a version bump. The lifecycle engine moves into VCF Operations, the 9.0 fleet management appliance is retired, and a new set of VCF management services plus a central license server appears in your management domain. If you start the upgrade the way you upgraded 5.2 or 9.0, the 9.1 binaries simply won't show up.
This is Part 1 of a two-part guide for upgrading an existing VCF fleet to 9.1.
It covers planning, the prerequisites, and the first four steps:
VCF Operations,
SDDC Manager,
VCF management services, and
licensing.
Part 2 covers the remaining management components and the core infrastructure:
NSX,
vCenter,
ESX,
NSX
Edge and
the clean-up steps.
The steps follow a 9.0.x source. Where a 5.2.x upgrade differs, I call it out.
What changes in the 9.1 upgrade
VCF Operations drives the upgrade. You plan, precheck, and run upgrades from Build → Lifecycle in VCF Operations. SDDC Manager is still upgraded and still does work behind the scenes, but its UI is being deprecated and will be removed in a future release.
The fleet management appliance goes away. Its job moves to the Fleet Lifecycle and SDDC Lifecycle services that run inside the new VCF management services. During the VCF Operations upgrade, the 9.0 appliance is powered off and left for you to delete.
VCF management services are new and mandatory for VCF. They run on a VCF services runtime in the management domain and need their own IP block and FQDNs.
A central license server is mandatory. It is deployed together with the management services, and your licenses are registered or transferred to it.
Workload domains can wait. To reach 9.1, you must upgrade the fleet-level components and the management domain. Workload domains are upgraded later as a separate day-N task.
Supported upgrade paths
You are running | Path to 9.1 |
VCF 9.0.x | In-place upgrade, following the sequence in this guide. |
VCF 5.2.x | Direct upgrade to 9.1 is supported. The order is slightly different (Aria components instead of their VCF 9 equivalents). |
VCF older than 5.2 | Upgrade the management domain and all workload domains to 5.2.x first. |
vSphere 8 or 9.0 only, with or without Aria or NSX | Separate converge or import paths. Use the VCF Upgrade Planning Tool. |
Two rules apply to every path:
Forward only. The target must be released after the version you run, and this applies to every component in your current bill of materials. You can move to a patch only if you could also move to its parent release.
No need to stop at 9.1.0. You can go straight to 9.1.1. Broadcom's guidance is to always use the latest available binaries for every component.
Before you start, run your exact versions through the VCF Upgrade Planning Tool and the Broadcom Interoperability Matrix. If you run vSphere Supervisor, also use the VKS Upgrade Planner. The vCenter, VKS and Supervisor upgrades can need several hops.
The upgrade order at a glance
This is the order for the first VCF instance, from a 9.0.x source. Stick to it. Most upgrade failures I have seen come from skipping ahead, for example, trying to upgrade SDDC Manager before VCF Operations.
# | Component | What happens | Part |
1 | VCF Identity Broker (appliance mode) | Redeploy first, only if it sits on an NSX overlay segment or on a network or datastore other than the management network | 1 |
2 | VCF Operations and cloud proxy | Upgrade with the 9.1 PAK file. The fleet management appliance is retired here | 1 |
3 | vSphere Replication, Live Site Recovery, vSAN Data Protection, Avi Load Balancer | Only if you use them, each with its own guide | 1 |
4 | SDDC Manager | Upgrade from VCF Operations | 1 |
5 | VCF management services and license server | New deployment, then license transfer or registration | 1 |
6 | Identity Broker, VCF Automation, Operations orchestrator, Operations for networks, Log Management, HCX, Data Services Manager | Management components, in that order | 2 |
7 | NSX Global Manager, NSX Manager, vCenter | Optimized or sequential scenario | 2 |
8 | VKS, Supervisor, vSAN witness, ESX | vLCM images only | 2 |
9 | NSX Edge and NSX-T, vSphere Distributed Switch, VMware Tools, VM compatibility, vSAN on-disk format, vSAN File Service | Finishing tasks | 2 |
From 5.2.x, the order is the same in principle, with these differences:
Aria Operations is upgraded in place to VCF Operations.
Aria Suite Lifecycle is not upgraded. Keep it for VMware Identity Manager day-N tasks, and don't delete VMware Identity Manager.
Aria Automation is upgraded to VCF Automation.
Aria Operations for Logs has no upgrade path. You deploy Log Management 9.1 as a day-N task.
With several VCF instances, upgrade the instance that hosts VCF Operations first, then the management domains of the other instances, and the workload domains last.
Before you start: the prerequisite checklist
I split these into checks for the whole platform and checks for the new 9.1 pieces.
Platform health
No failed workflows in SDDC Manager, and no resources in an activating or error state. If you have any, talk to Broadcom Support before you upgrade.
No expired or expiring passwords or certificates.
No active alarms on hosts or clusters in the vSphere Client.
ESX hardware supported for 9.1 in the Broadcom Compatibility Guide, and the vSAN HCL database updated manually.
TPM modules on 2.0 firmware, or turned off in the BIOS if you don't use them.
No vCenter joined to an Active Directory domain.
One temporary IP address on the management subnet for the vCenter upgrades. The same address can be reused for each vCenter.
All clusters managed by vSphere Lifecycle Manager images. Baselines are not supported in VCF 9.x. You must transition baseline clusters before the ESX upgrade.
Upgrade-related known issues are reviewed in the 9.1.1 release notes.
Backups and snapshots
A file-based backup of SDDC Manager to an external SFTP server, plus a cold snapshot of the SDDC Manager VM.
Backups of every vCenter and NSX Manager.
A snapshot of every VCF Operations node, and an export of any customized VCF Operations content. The upgrade resets the out-of-the-box alerts, dashboards, and reports to their defaults, so manual edits to them are lost.
New 9.1 requirements
Certificates with complete Subject Alternative Names. VCF Operations, VCF Automation, and VCF Operations for Networks certificates must have complete SANs. For Operations for Networks, that means the FQDNs and IPs of all platform and collector nodes.
VCF Automation location. VCF Automation 9.0.x must run in the management domain, not in a workload domain.
Identity Broker placement. An Identity Broker 9.0.x appliance must run on the VCF management network, on the same network and datastore as the management services. If it sits on an NSX overlay segment or a different network, redeploy it before you upgrade VCF Operations. Otherwise, the later Identity Broker upgrade fails.
Cloud proxy. The first VCF instance needs a cloud proxy. Existing proxies upgrade automatically.
No SSL-terminating proxy. If VCF Operations uses a proxy server with SSL termination, remove that configuration first.
vCenter integration entries. The vCenter field in VCF Operations integrations must hold a plain FQDN or IP address, with no URL, slashes, or sub-paths.
Fleet management appliance root password. The VCF Operations upgrade asks for it to migrate the appliance's data.
IP block and DNS records for the management services. At least 12 addresses (a /28), 30 recommended (a /27). You also need five lowercase FQDNs in DNS: services runtime, fleet components, instance components, identity broker, and license server. The license server needs both an A record and a PTR record. Details are in Step 3.
Required ports are open, as listed in the Broadcom ports reference.
Tip from the field: Run the SDDC Manager upgrade prechecks a week before the window, not on the night. Expired passwords, stale certificates, and alarms are quick to fix with a week of notice and painful at 2 a.m.
Step 1: Upgrade VCF Operations to 9.1
VCF Operations goes first, and it is the one component you upgrade manually. Until VCF Operations runs 9.1, the 9.1 binaries do not appear in the lifecycle screens (Broadcom KB 440282).
Prepare
Download the VCF Operations 9.1 upgrade PAK file from the Broadcom Support Portal. Go to My Downloads → VMware Cloud Foundation 9, choose the 9.1.x release, and open the VCF Operations group.
Take a snapshot of every VCF Operations node. The safest way is to take the cluster offline in the administration interface, snapshot the nodes, and bring the cluster fully online again. The cluster must be online when the installation starts.
Export any customized dashboards, alerts, reports, and other content you changed.
Upgrade
Log in to the administration interface of the primary node at https://<primary-node>/admin as admin.
Click Software Update, then Install a Software Update.
Accept the terms in the Recommended Actions dialog and click Next.
Browse to the PAK file, click Upload, then click Next.
Accept the EULA, review the update information, and click Install.
The interface restarts. Log back in and open Software Update to follow progress.
From 9.0.x only: when the Fleet Management v9.0 Node data Migration to VCF Operations dialog appears, enter the fleet management appliance root password, select the confirmation check box, and click OK.
Wait until all nodes and the cluster show online on the System Status page. If the cluster stays offline, bring it online manually and click Finish Installation.
Confirm the success message under Software Update.
What to expect afterwards
Cloud proxies. The cloud proxies upgrade automatically, up to 10 at a time, rebooting as they go. If one fails, upgrade it with the cloud proxy CLI.
Fleet management appliance. The 9.0 appliance is powered off. Delete it once you're happy.
License sync warning. "Licenses could not be synchronized with vCenter instances" is expected until the license server is registered in Step 4.
Certificates and passwords. Certificate and password management for VCF Automation, Identity Broker, and the logs and networks components is unavailable until those components reach 9.1.
Snapshots. Delete the VCF Operations snapshots once the upgrade is confirmed good. Keeping several snapshots degrades performance.
From 5.2.x, you upload the same PAK file in the Aria Operations administration interface. Afterwards, remove VCF Operations from the Aria Suite Lifecycle environment, without deleting its VMs from vCenter.
Step 2: Upgrade SDDC Manager to 9.1
Prepare
A recent successful SDDC Manager backup to SFTP, plus a snapshot of the SDDC Manager VM.
Recent backups of the components SDDC Manager manages.
If you run Avi Load Balancer, check that its version is supported with SDDC Manager 9.1.
The SDDC Manager 9.1 binary is downloaded. In VCF Operations, go to Build → Lifecycle, select the VCF instance, and open Binary Management. Choose the 9.1 version, open the Upgrade Binaries tab, select SDDC Manager, and click Download.
The SDDC Manager prechecks run, with any failures fixed.
If your management domain uses external storage instead of vSAN, the vSAN-related precheck results are expected. Override them only after you've validated the storage yourself.
Upgrade (from 9.0.x)
In VCF Operations, go to Build → Lifecycle.
Expand VCF Instances and select the instance that hosts SDDC Manager.
On the SDDC Manager Updates tab, open Available updates and click Update Now, then Finish.
When SDDC Manager reports 9.1, log in to SDDC Manager and open Administration → Depot Settings. Reconfigure the online depot to use an activation code from the VCF Business Services console instead of a download token.
From 5.2.x, the SDDC Manager upgrade still starts in the SDDC Manager UI:
Open the management domain's Updates tab and click Plan Upgrade.
Pick the target version and confirm the change summary.
Click Update Now or Schedule Update.
Step 3: Deploy VCF management services and the license server
This step is the biggest change in 9.1. VCF Operations deploys a VCF services runtime, which hosts the fleet and instance services, plus the identity broker (if you don't already have an appliance-mode one) and the license server.
Binaries the depot must hold first
For the first VCF instance, download all nine before you start:
Fleet Lifecycle
SDDC Lifecycle
VCF Services Runtime
Salt RaaS
Salt Master
VCF Identity Broker
Software Depot
Telemetry
License Server
Additional VCF instances need only SDDC Lifecycle, VCF Services Runtime, Salt Master, and Telemetry (Broadcom KB 425089). For an offline or restricted SDDC Manager, use the VCF Download Tool.
Plan the network and DNS
Item | Requirement |
VCF services runtime IP pool | Minimum 12 addresses (/28), 30 recommended (/27). From 9.1.0.400, you can exclude addresses or enter a list of non-contiguous addresses. |
Deployment network | 9.1.0.x deploys to the VCF management network (or the VM management network if there is none). From 9.1.1, you can choose the network in the UI. |
Internal cluster CIDR | Default 198.18.0.0/15. It must not overlap with any routes you configure. Alternatives are 240.0.0.0/15 or 250.0.0.0/15, selectable in the UI from 9.1.1. |
FQDNs | Services runtime, fleet components, instance components, identity broker, and license server. All lowercase, resolving to unique unused IPs outside the pool. The .local domain is not supported. |
License server | IPv4 only at deployment. Needs an A record and a PTR record. |
Deploy
Confirm that VCF Operations and SDDC Manager both run 9.1.x.
In VCF Operations, go to Build → Lifecycle, expand VCF Instances, and select the instance that hosts VCF Operations.
Open the SDDC Manager Upgrades tab, then Install Management Services. Under Next steps, click Install.
In the Components Installation dialog, enter the VCF Operations load balancer or primary node FQDN and the administrator credentials, then click Connect.
Fill in the runtime IP pool, the FQDNs, the runtime size, and the runtime password. Add the deployment network and internal CIDR if you are on 9.1.1 or later. Click Install.
Wait for the deployment to complete. The new management services VMs and the license server then appear in the management vCenter inventory.
The same deployment is available through the SDDC Manager API: POST /v1/vcf-management-components/validations, then POST /v1/vcf-management-components, from the API Explorer under Build → Developer Center. Use the API if you need a VLAN-backed network through the xRegionNetwork parameter.
Leave these VMs where they are. Don't move the management services VMs to another resource pool or VM folder, and don't Storage vMotion them.
Step 4: License the upgraded environment
From 9.0.x, how your licenses move depends on how VCF Operations connects to Broadcom:
Connected mode: the licenses transfer to the new license server automatically.
Disconnected mode: you register VCF Operations and move the licenses with files, as follows.
For disconnected mode:
In VCF Operations, go to Manage → Licensing → Licenses & Registration and click Continue in the Register & License VCF Operations pane.
Download the registration file.
On an internet-connected machine, sign in to the VCF Business Services console and select your Site ID. Under Licensing → VCF Operations Registrations, create a new registration, import the file, and give the instance a unique name.
Exchange the verification and confirmation files between the console and VCF Operations as the cards prompt.
Download the license file from the console and import it into VCF Operations.
Licenses return to the assets they were assigned to before the upgrade. Duplicate entries can show during the transfer and clear when it finishes. Don't assign licenses while the transfer is running.
From 5.2.x, register VCF Operations and license the environment as a new 9.x registration instead.
Step 5: Configure the fleet software depot
Before you touch any other component, configure the fleet-level software depot (online or offline). In Build → Lifecycle → Binary Management, download the binaries for the next components: VCF Automation, NSX, vCenter, and the ESX components. For ESX, click Synchronize Now under ESX Components so the base images, add-ons, and VMware Tools are available for building your vLCM image.
Known issues worth reading before you start
KB | Issue |
440282 | 9.1 binaries don't show in VCF Operations 9.0. Upgrade VCF Operations with the PAK file first. |
425089 | Lists the binaries required for the management services deployment. |
444734 | Identity Broker 9.1 upgrade fails when the 9.0 appliance is on an overlay segment or another network or datastore. |
440471, 424533 | vCenter licensing errors after the 9.1 upgrade. |
440407 | Sync error during the ESX host upgrade after moving from VCF Operations 9.0.2. |
440459 | "Import VCF Operations in Fleet Lifecycle" fails because of an Operations for Networks certificate SAN problem. |
441526 | License server black screen at boot because of a Docker network conflict with 172.17.0.0/16. Fixed in license server 9.1.0.0200 and later. |
Where you are now
At this point you have:
VCF Operations and its cloud proxies on 9.1
SDDC Manager on 9.1
The VCF management services and license server deployed
Licenses registered
The fleet depot is ready
Your NSX, vCenter, and ESX are still on 9.0.x, and they keep running normally. In Part 2, we upgrade the remaining management components, then the management domain core: NSX, vCenter, ESX, NSX Edge, and the finishing tasks.
Comments