VMs Hanging on One ESXi Host — Suspected Hardware Issue (3PAR SAN Storage)

[Greatson99]

Hi everyone,

I’m facing a recurring issue in our VMware environment and would appreciate some insights.

We have a cluster of 6x HPE ProLiant DL380 Gen10 servers running VMware ESXi 7.0 Update 3, all connected to HPE 3PAR SAN for shared storage. The ESXi hosts are only used for compute resources (CPU/RAM) — all VM storage resides on 3PAR.

Here’s the issue:

• One specific host has had 3 incidents where multiple VMs hang/freeze and become completely unresponsive.

• The affected VMs cannot be shut down or restarted via vCenter or CLI (vim-cmd and esxcli vm process kill fail).

• The host itself remains up and connected in vCenter, but the stuck VMs essentially lock up the host.

• I cannot reboot the host easily due to it running 60+ critical production VMs.

• Other hosts in the cluster using the same 3PAR datastore are not affected.

What I’ve done so far:

• Logged a ticket with Broadcom (VMware), who reviewed the issue remotely and concluded it's likely a hardware issue.

• Logged a case with HPE, but the server is out of warranty, so no direct support there.

• Checked vSphere logs, and I’m starting to suspect either an HBA issue, a faulty FC port or cable, or possibly driver/firmware mismatch.

• Planning to test FC path stability and possibly rotate cables/ports if needed.

Has anyone encountered a similar situation where only one ESXi host in a cluster behaves like this with shared SAN storage?

Would really appreciate any suggestions for:

• What specific logs or metrics to check?

• Common signs of a bad HBA or FC path?

• Any non-disruptive tests I can run on this live host?

• Tools for validating hardware (HBA, RAM, etc.) without taking it down?

Thanks in advance for any help!

[support_s]

System recommended content:

1. HPE OneView for VMware vCenter Release Notes (9.1)

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[shiva_jr]

Hi @Greatson99 ,

     As per my understanding,  I am surely suspecting on FC HBA, Storage connectivity and likely ESXI server hardware performance. 
      By the way, please check Server resouces are suffice to run the 60 virtual machines. It might be loaded.

• What specific logs or metrics to check?
  Use the 'esxtop' command to view the vmware resource monitor.
  Use the multiple logs such as vmkernel, hostd etc from /var/log/.
  You can also get the VM logs in vSphere Web GUI.

• Common signs of a bad HBA or FC path?
   You can verify the bad HBA, its status by following commands: (Sometimes, the issue occurs if the ESXi showing duplicate HBA for single physical HBA .)
   'esxcli storage core adapter list'
   'esxcli storage core adapter rescan'
   'esxcli storage san fc list'  
  
  

• Any non-disruptive tests I can run on this live host?
   I have no idea about live test tools.

• Tools for validating hardware (HBA, RAM, etc.) without taking it down?
  esxtop command would help you here.

• Other suggestions from me :
  a.  Please check the FC ports connected to the server are online. Try with  alternative cable.
  b. Check the Esxi host and their presented volumes status (online, disk space, bandwidth) from the 3par Storage.
  c. Firmware and driver of a particular host may be lower version.
  d. Check the hardware version of VMs may be lower.
  
  I hope, my post will help you, if yes, please give KUDOS and mark my post as an Accepted Solution, let me know if issue still persists.
  Regards,
  Shiva_jr

[support_s]

Hello,

Let us know if you were able to resolve the issue.

If you are satisfied with the answers then kindly click the "Accept As Solution" button for the most helpful response so that it is beneficial to all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".