Downlink Ports Unlinked

[Chris86]

Hi Community,

i have:

2 Compute Nodes (480 Gen 10 and 660 Gen 10) in the same frame (F2) with ESXi 7.0U3 showing:

vmhba offline

The Compute Nodes have problems with:

Downlink Ports on different interconnects (SAN, not LAN/ethernet)

The server profile connection status is green for mezzanine 5 and mezzanine 2.

The vmhba1 (mezz 2:1) goes online after 20 min. The vmhba67 (mezz 5:2) goes online after hours.

After going online, the vmhbas are showing Dropped Frames Event on the ESXi.

We checked the firmware/drivers with VMware support. The fw of san storage, san switch and synergy are latest.

The HPE support let me set back one of the compute nodes with efuse command. But this dont solve our problems.

efuse of the interconnect doesent help too.

What can I do? Please help me.

Greetings

Chris

[F_Silva1]

Hi Chris86.

Do you have 1 or 2 frames?

Do you tried to move this server for any bays empty or one bay that doens't have a problem. Why, this tests can isolate if the problem it's in CNA card or in the internal lines. 

If there has 2 frames,  what is the model of the modules that communications between Frames (Virtual Connect SE 40gb or 100Gb) connected on Interconnect Link Module for example from 20gb or 50Gb?.

[MV3]

Hello 

Is this a recent implementation or an established configuration? Do you happen to know when the problem first emerged, or were there any modifications made to the environment around the time the issue was reported?

Cheers...

[Chris86]

Hi,

I moved 6 working Compute Nodes ( 4x 480 Gen 10 and 2 x 660 Gen10) from an existing frame into 2 new frames.
After relocation, I updated the OVA from 8 to 8.60.1 and the SSPs from 2022.11.01 to 2023.10.01. The configuration was already checked by HPE support.

480 GEN 10:
On one of the 480 Gen 10 Compute Nodes the FC links going up within seconds, but showed "Dropped Frames" on the ESXi level vmhba1 = Mezzanie 2:1
I changed the location Frame 2 bay 5 to Frame 2 bay 6 and now the Links are up an running without dropped frames.

660 GEN 10:
I added two Connections: FC Ports Mezz 5:1 and 5:2 to one of the 660 Profiles.
The 660 Gen 10 showed on vmhba 67 = Mezz 5:2 "Cable not plugged in" on the ESXi level. After 20 min the vmhba1 = Mezz 2:1 goes online but with "Dropped Frames".
After 5 hours vmbha67 = Mezz 5:2 goes online, but showing Dropped Frames.
So I changed the mezz 2 and mezz 5, still the same error on the same vmhbas.
After that, I changed the Compute Node bay and all Links going up within seconds, but showing dropped frames on vmhba67 = mezz 5:2.
The vmhba1 = mezz 2:1 is now online and showed no errors after changing the bay.

I already efused the 660 gen 10 compute node and the interconnect for mezz 5:2

Switch (Brocade G630) and SAN (Primera A650) is up to date, showing no errors.

Uplinks: ok
Interconnects: ok
Connections: ok

Synergy 5830C 32Gb FC HBA
Driver Name: qlnativefc
DriverVersion: 5.3.81.2
Invalid Tx Word Count: 73975524

The things that changed after relocation of the compute nodes: OVA and SSP version, adding 2 new connections to 660 Gen 10 Compute Node.

Greetings

Chris

[Chris86]

Hi,

its an Virtual Connect SE 32Gb FC Module for Synergy

3 Uplink Ports (Trunk), 2 Fabrics. 

Its not an CNA, the Mezzanine 2 and Mezzanine 5 are Synergy 5830C 32Gb FC HBA

[MV3]

Hello 

A comprehensive log analysis is required for this issue. Kindly open a support case with HPE.

Cheers....

[Chris86]

Hi,

yeah I know, already did that. The HPE support is still working on that case, I think since a month or so. VMware support was super fast, analyzing the logs of 12 esxi on one day and found no issues on the esxi level. HPE L1 support found some problems with ethernet, sending a link cable because of port flapping, but didnt recognize our understand my SAN problem. After discoussions with L1 and some HPE managers, L2 and L3 is on the case now...