HCX Software Updates are published to connected HCX Cloud Managers & HCX Connectors that have a valid connectionto the mothership at connect.hcx.vmware.com. At times the state of a download may come into question.

In this post we explore the relevant product areas to get more information during the download activity..

Software Update is Available

A banner shows updates are available:

A manual check for updates operation is available to have an HCX Manager proactively query the connect.hcx.vmware.com for updates (click it once)

Download Status in the HCX System Updates UI

The Download (only) and the Download & Upgrade operations are displayed in this interface.

Verifying the Download Endpoint

pinging hybridity-depot.vmware.com resolves a specific akamaiedge address, in this case 23.192.208.27

(The HCX Manage’s depot endpoint will be provided by the CDN service, it may vary from this example)

[root@w3-hs2-grosas-hcx-connector /home/admin]# ping hybridity-depot.vmware.com
PING e751.dscd.akamaiedge.net (23.192.208.27) 56(84) bytes of data.

64 bytes from a23-192-208-27.deploy.static.akamaitechnologies.com (23.192.208.27): icmp_seq=1 ttl=46 time=44.7 ms
64 bytes from a23-192-208-27.deploy.static.akamaitechnologies.com (23.192.208.27): icmp_seq=2 ttl=46 time=44.6 ms

Verifying Network Connection for Active Download

tcpdump -p 443 reflects a connection to the hybridity-depot endpoint

[root@hcx-connector /home/admin]# tcpdump port 443

22:53:41.836665 IP a23-192-208-27.deploy.static.akamaitechnologies.com.https > 10.x.y.2.57632: Flags [P.], seq 2274235:2275595, ack 1, win 501, length 1360

22:53:41.837129 IP 10.x.y.2.57632 > a23-192-208-27.deploy.static.akamaitechnologies.com.https: Flags [.], ack 2275595, win 326, length 0

Use a filtered tcpdump to validate ongoing connections during a download.

[root@hcx-connector /home/admin]# tcpdump host 23.192.208.28 

Bytes Downloaded

The temporary location can be used to verify current download progress in bytes /common/tmp/upgrade/download/

[root@hcx-connector ~]# ls -ls  /common/tmp/upgrade/download/

                          bytes downloaded 
-rw-r--r-- 1 admin secureall 3040480517 Nov  9 23:11 VMware-HCX-Connector-upgrade-bundle-4.2.2.0-18867885-signed.tar.gz  

Log Entries for the Download Process

Search app.log for VrealmUpgradeService entries.

[admin@hcx-connector ~]$ grep -i vrealm  /common/logs/admin/app.log

Log entries when the download is triggered:

2021-11-09 22:50:07.065 UTC [VrealmUpgradeService_SvcThread-3, Enterprise: HybridityAdmin, Username: HybridityAdmin, TxId: ] INFO  c.v.v.h.s.upgrade.VrealmUpgradeUtils- Checking required disk space 5898 MB upgrade bundle in /tmp/upgrade/download
2021-11-09 22:50:07.246 UTC [VrealmUpgradeService_SvcThread-3, Enterprise: HybridityAdmin, Username: HybridityAdmin, TxId: ] INFO  c.v.v.h.depot.adapter.AkamaiAdapter- starting bytes 0

The required diskspace value referenced is for the uncompressed file, it won’t match what is downloaded to the tmp/ location.

Log entry when the download completes:

2021-11-09 23:12:06.158 UTC [VrealmUpgradeService_SvcThread-3, Enterprise: HybridityAdmin, Username: HybridityAdmin, TxId: ] INFO  c.v.v.h.s.upgrade.DownloadFileJob- Downloaded upgrade bundle /VMware-HCX-Connector-upgrade-bundle-4.2.2.0-18867885-signed.tar.gz to /tmp/upgrade/download/VMware-HCX-Connector-upgrade-bundle-4.2.2.0-18867885-signed.tar.gz

Completed Download

The Upgrade operation becomes available

Logs Entries When

2021-11-10 00:14:00.592 UTC [VrealmUpgradeService_SvcThread-4, Enterprise: HybridityAdmin, Username: HybridityAdmin, TxId: ] INFO  c.v.v.h.s.upgrade.DownloadFileJob- Already downloaded upgrade bundle /VMware-HCX-Connector-upgrade-bundle-4.2.2.0-18867885-signed.tar.gz to /tmp/upgrade/download/VMware-HCX-Connector-upgrade-bundle-4.2.2.0-18867885-signed.tar.gz

2021-11-10 00:14:35.569 UTC [VrealmUpgradeService_SvcThread-4, Enterprise: HybridityAdmin, Username: HybridityAdmin, TxId: ] INFO  c.v.v.h.s.upgrade.VrealmUpgradeUtils- Checking required disk space 2999 MB upgrade bundle in /tmp/upgrade/download

2021-11-10 00:14:35.569 UTC [VrealmUpgradeService_SvcThread-4, Enterprise: HybridityAdmin, Username: HybridityAdmin, TxId: ] INFO  c.v.v.h.s.upgrade.StartUpgradeJob- Extracting upgrade bundle /tmp/upgrade/download/VMware-HCX-Connector-upgrade-bundle-4.2.2.0-18867885-signed.tar.gz

2021-11-10 00:14:55.300 UTC [VrealmUpgradeService_SvcThread-4, Enterprise: HybridityAdmin, Username: HybridityAdmin, TxId: ] INFO  c.v.v.h.s.upgrade.StartUpgradeJob- Extracted upgrade bundle /tmp/upgrade/download/VMware-HCX-Connector-upgrade-bundle-4.2.2.0-18867885-signed.tar.gz


2021-11-10 00:15:59.412 UTC [VrealmUpgradeService_SvcThread-4, Enterprise: HybridityAdmin, Username: HybridityAdmin, TxId: ] INFO  c.v.v.h.m.k.KafkaProducerDelegate- Initializing a new Kafka producer with retries 

2021-11-10 00:16:04.667 UTC [VrealmUpgradeService_SvcThread-4, Enterprise: HybridityAdmin, Username: HybridityAdmin, TxId: ] INFO  c.v.v.h.s.upgrade.StartUpgradeJob- Posted event to kick off upgrade

2021-11-10 00:16:04.674 UTC [VrealmUpgradeService_SvcThread-4, Enterprise: HybridityAdmin, Username: HybridityAdmin, TxId: ] INFO  c.v.v.h.s.upgrade.StartUpgradeJob- Pausing StartUpgradeJob till the upgrade completes

At this point ServiceManager will shut down all services. The process will never reach this stage if the download is not complete.