Applying vCenter Server 7.0 Update 2d patch on VMware Cloud Foundation on 4.1.x, 4.2.x, 4.3

Products

VMware Cloud Foundation

Issue/Introduction

The recommendation to customers on these prior VCF 4.1, 4.1.0.1, 4.2. 4.2.1, 4.3 versions, is to upgrade to the latest VCF 4.3.1 release.
If customers are unable to do so, the purpose of this article is to provide guidance for such customers to upgrade just vCenter Server appliance
The information contained in this article also applies to VCF on VXRail environments, and also to vSAN Ready Nodes

The JAR file attached to this article can be used to automate the steps required.
See details on how to use this utility below

Symptoms:

As documented in VMSA-2021-0020 , all versions of the vCenter Server 7.0 appliance are prior to 7.0U2d are affected by the vulnerabilities listed in the advisory
Since the VMware Cloud Foundation(VCF) 4.x versions, prior to VCF 4.3.1, bundle impacted releases of vCenter Server, the VCF versions 4.1, 4.1.0.1, 4.2. 4.2.1, 4.3, are similarly impacted by the vulnerabilities listed in the advisory

Environment

VMware Cloud Foundation 4.2.x
VMware Cloud Foundation 4.1
VMware Cloud Foundation 4.3.x

Resolution

VMware Cloud Foundation Version	If you are unable to upgrade to VCF4.3.1 at this time
4.1 or earlier	First upgrade to VCF version 4.1 or later and follow the respective recommended approach
4.1.x	Apply the steps in the Workaround section of this article
4.2.X	Apply the steps in the Workaround section of this article
4.3	Apply the steps in the Workaround section of this article

NOTE: You can also choose to upgrade to VCF 4.3.1 to consume VC security patch and other product fixes and enhancements. Please refer to VCF 4.3.1 upgrade guide and release notes for more details.

Workaround:

To apply the vCenter Server 7.0 U2d patch on VCF 4.1.x, VCF 4.2.x and VCF 4.3 follow these steps in the exact sequence

STEP 1: Perform below steps on each vCenter Server VM deployed in your Cloud Foundation environment

1) Take a snapshot of the vCenter Server before applying the patch
2) Apply the VMware vCenter server 7.0 Update 2d patch available at the Product Patch page to all vCenter Servers (Management & VI Workload Domain) in the environment. Refer to vCenter server 7.0 Update 2d release notes section "Download and Installation" on steps to upgrade. NOTE: This will be out of band upgrade, outside of SDDC Manager using vCenter product upgrade procedure.

STEP 2: Perform below steps on each SDDC Manager VM deployed in your Cloud Foundation environment

1) Login to SDDC manager VM via SSH and sudo to root account

2) Get upgraded VMware vCenter Server ID from VCF inventory:
To get VMware vCenter Server details from VCF inventory run following command/Curl/API:
Command:

curl localhost/inventory/vcenters | json_pp
% Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
100   359    0   359    0     0  19944      0 --:--:-- --:--:-- --:--:-- 19944
[
   {
      "status" : "ACTIVE",c
      "version" : "7.0.1.00000-16860138",
      "bundleRepoDatastore" : "lcm-bundle-repo",
      "datastoreForVmDeploymentName" : "sfo01-m01-vsan",
      "domainType" : "MANAGEMENT",
      "vmName" : "vcenter-1",
      "domainId" : "d8864d48-96c5-4407-8665-d5988c52c05b",
      "hostName" : "<vcenter_FQDN>",
      "managementIpAddress" : "10.0.0.6",
      "id" : "<VMware vCenter Server ID>"
   }
]

The field "id" in response, corresponds to VMware vCenter Server id.
The "version" field for each of the VMware vCenter Server provides the current version of the VMware vCenter Server.

3) Update VCF inventory for VMware vCenter Servers
Note: Repeat below commands for all the VMware vCenter Severs with their corresponding vcenter-id that were upgraded.

<SDDC_Manager_FQDN > = Fully qualified domain name of SDDC manager.
<VMware vCenter Server_Id> = Id of VMware vCenter Server for which version is to be updated in VCF inventory
7.0.2.00500-18455184 = Version of VMware vCenter Server patch that was applied on hosts.

curl -X PATCH '<SDDC_Manager_FQDN >/inventory/entities/<VMware vCenter Server ID>' -d '{"version":"7.0.2.00500-18455184", "type":"VCENTER"}' -H 'Content-Type:application/json'

4) Verify VMware vCenter Server versions

curl localhost/inventory/vcenters | json_pp
Sample Output:
  % Total    % Received % Xferd Average Speed   Time    Time     Time Current
                                 Dload Upload   Total   Spent    Left Speed
100  1069    0  1069    0     0   173k      0 --:--:-- --:--:-- --:--:-- 173k
[
   {
      "bundleRepoDatastore" : "lcm-bundle-repo",
      "managementIpAddress" : "10.0.0.6",
      "version" : "7.0.2.00500-18455184",
      "vmName" : "vcenter-1",
      "domainType" : "MANAGEMENT",
      "domainId" : "d8864d48-96c5-4407-8665-d5988c52c05b",
      "datastoreForVmDeploymentName" : "sfo01-m01-vsan",
      "status" : "ACTIVE",
      "id" : "<VMware vCenter Server ID>",
      "hostName" : "<vcenter_FQDN>"
   }
]

5) Import host keys for the upgraded VMware vCenter server into SDDC manager known hosts file

NOTE: If you are running VMware Cloud Foundation versions 4.3, this step is not needed

a. Take backup of existing keys of SDDC manager known hosts file

cp /etc/vmware/vcf/commonsvcs/known_hosts /etc/vmware/vcf/commonsvcs/known_hosts.bak

b. Import ecdsa and ed25519 keys of <vcenter_FQDN> into SDDC manager known hosts fileInfo:
<vcenter_FQDN> = Fully qualified domain name of upgraded vCenter Server

ssh-keyscan -t ecdsa-sha2-nistp256 -p 22 <vcenter_FQDN> 2>/dev/null >> /etc/vmware/vcf/commonsvcs/known_hosts
ssh-keyscan -t ssh-ed25519 -p 22 <vcenter_FQDN> 2>/dev/null >> /etc/vmware/vcf/commonsvcs/known_hosts

c. Login to SDDCManager UI and navigate to specific workload domain Inventory → Workload Domains → Select Workload Domain → Updates/Patches → Current Versions to verify the VC version after few minutes.

STEP 3: Perform below add-on steps on the upgraded vCenter Server VM in your Cloud Foundation environment

Known Issue: After upgrade of vCenter Server to 7.0 Update 2d in Cloud Foundation environment, if SDDC Manager VM is rebooted, SDDC Manger UI cannot communicate to newly upgraded vCenter Server VM. vCenter server 7.0 Update2 onwards SHA1 is removed from SSH. SDDC Manager 4.1.x, 4.2.x uses SHA1 based SSH to communicate with vCenter server. This issue is fixed in SDDC Manager 4.3 and above versions

NOTE: If you are running VMware Cloud Foundation versions 4.3, this step is not needed

Enable SHA1 (ssh-rsa) host key algorithms on VC

a. Login to upgraded vCenter Server via SSH

b. Take backup of sshd_config file

cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak

c. Open the sshd_config file in editor and locate "HostKeyAlgorithms"

vi /etc/ssh/sshd_config

d. Change the HostKeyAlgorithms entry by appending "ssh-rsa" towards the end of the line

From

HostKeyAlgorithms [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256

To

HostKeyAlgorithms [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa

e. Save and exit the file /etc/ssh/sshd_config

f. Restart SSH daemon in vcenter

systemctl restart sshd

Note:
Repeat STEP1, STEP2 and STEP3 for all workload domain vCenters in your Cloud Foundation environment.
Every time a new VI workload domain is created, these steps need to be performed.

Alternatively: STEP2 and STEP3 can be executed via automated script as follows

(1) Download and scp the VCF4x-KB_85718.jar file to SDDC manager in /tmp location
(2) Login to SDDC manager VM via SSH and sudo to root account and provide executable permission to the binary:

chmod ugo+x /tmp/VCF4x-KB_85718.jar

STEP 2: Perform below steps on each SDDC Manager VM deployed in your Cloud Foundation environment

(1) Update VCF inventory following the steps below:

<vcenter_FQDN> = Fully qualified domain name of upgraded vCenter Server

java -jar /tmp/VCF4x-KB_85718.jar --vcsaip <vcenter_FQDN> --username root --updateVCFInventory

NOTE: If you are running VMware Cloud Foundation versions 4.3, this step is not needed
(2) Import host keys for the upgraded VMware vCenters

java -jar /tmp/VCF4x-KB_85718.jar --vcsaip <vcenter_FQDN> --username root --importSSHKeys

STEP 3: Perform below add-on steps on the upgraded vCenter Server VM in your Cloud Foundation environment

NOTE: If you are running VMware Cloud Foundation versions 4.3, this step is not needed
(1) Enable SHA1 host key algorithm

java -jar /tmp/VCF4x-KB_85718.jar --vcsaip <vcenter_FQDN> --username root --enableSHA1

Sample Output
NOTE: We will be required to perform aliasing steps when further upgrades are required after we have completed the steps from the KB.

Additional Information

Impact/Risks:

After applying the vCenter Server 7.0 Update 2d on your VCF 4.1, 4.1.0.1, 4.2, 4.2.1, or 4.3 environment using the procedure below, the supported upgrade path is to VCF 4.3.1 using Skip Upgrade from SDDCManager UI. For more details reference the VMware Cloud Foundation Upgrade Documentation .

Attachments

VCF4x-KB_85718 get_app